From makaplan at us.ibm.com  Fri Dec  1 04:25:30 2017
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Thu, 30 Nov 2017 23:25:30 -0500
Subject: [gpfsug-discuss] FIle system vs Database
In-Reply-To: <OFDAB5F371.63BBCEDD-ON002581E8.00776AE7@LocalDomain>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com><OF1A4C52AF.364AA2DD-ON002581E8.00592509@LocalDomain><OF0CB3DA65.211AD6C3-ON852581E8.006590C3-852581E8.0065FFAE@notes.na.collabserv.com><OF777157A3.FE35AFDD-ON002581E8.0066BB88@LocalDomain><OFC357735B.03D413FB-ON852581E8.0072148B-852581E8.0072A3C9@notes.na.collabserv.com>
	<OFDAB5F371.63BBCEDD-ON002581E8.00776AE7@LocalDomain>
Message-ID: <OF4ADF66F6.43003461-ON852581E9.0017BDD1-852581E9.00184F30@notes.na.collabserv.com>

Indeed, for a very large directory you might get some speedup using

samples/ilm/mmfind directory -ls -maxdepth 1

There are some caveats, the same as those for the command upon which 
mmfind rests, mmapplypolicy.




From:   Skylar Thompson <skylar2 at u.washington.edu>
To:     gpfsug-discuss at spectrumscale.org
Date:   11/30/2017 04:44 PM
Subject:        Re: [gpfsug-discuss] FIle system vs Database
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Interesting, thanks for the information Marc. Could there be an 
improvement
for something like "ls -l some-dir" using the API, though? Instead of 
getdents +
stat for every file (entering and leaving kernel mode many times), could 
it be done
in one operation with one context switch?




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171130/4e738bf8/attachment.htm>

From chris.schlipalius at pawsey.org.au  Fri Dec  1 05:32:28 2017
From: chris.schlipalius at pawsey.org.au (Chris Schlipalius)
Date: Fri, 01 Dec 2017 13:32:28 +0800
Subject: [gpfsug-discuss] 2018 March 26th Singapore Spectrum Scale User
 Group event announced - Call for user stories
In-Reply-To: <A675FAE6-BAE2-4E17-AC2B-7EF1607CC7C9@pawsey.org.au>
References: <A675FAE6-BAE2-4E17-AC2B-7EF1607CC7C9@pawsey.org.au>
Message-ID: <391567D9-DC51-46B9-8E0E-D8EE13F6B814@pawsey.org.au>

Hello,
This is an announcement for the event held just before SCA18 and  general call out for user use cases or presentations that you would like to present, at the inaugural Spectrum Scale Usergroup Singapore on the Monday 26th March 2018, Sentosa, Singapore.
This is being held in conjunction with SCA18  https://sc-asia.org/
     
All current Singapore Spectrum Scale User Group event details can be found here: http://goo.gl/dXtqvS
Feel free to circulate this event link to all that may need it.
Please reserve your tickets now, and if you would like to speak, please email me the title and duration of the talk and speakers name details so I can add this to the agenda on Eventbrite and promote this in this discussion list and spectrumscale.org website.
     
Accommodation.
Once numbers of attendees are known I can obtain delegate accommodation custom rate for guests staying at the resort for this event. So please reserve a ticket in the Eventbrite link above ASAP.
We are looking forwards to a great new Usergroup next year in a fabulous venue.
Thanks again to NSCC and IBM for helping to arrange the venue and event booking.
     
Regards,
Chris Schlipalius
     
Team Lead, Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO)
12 Burvill Court
Kensington  WA  6151
Australia
     
Tel  +61 8 6436 8815  
Email  chris.schlipalius at pawsey.org.au

Regards,
Chris Schlipalius
 
Team Lead, Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO)
12 Burvill Court
Kensington  WA  6151
Australia
 
Tel  +61 8 6436 8815  
Email  chris.schlipalius at pawsey.org.au
Web  www.pawsey.org.au <http://www.pawsey.org.au/>
 
 
 

 
 
Web  www.pawsey.org.au <http://www.pawsey.org.au/>
     
     
    




From bhartner at us.ibm.com  Fri Dec  1 12:44:38 2017
From: bhartner at us.ibm.com (Bill Hartner)
Date: Fri, 1 Dec 2017 06:44:38 -0600
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
Message-ID: <OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>

> "It has a significant performance penalty for small files in large
> block size filesystems"

Aaron,

Below are mdtest results for a test we ran for CORAL - file size was 32k.

We have not gone back and ran the test on a file system formatted without >
32 subblocks.  We'll do that at some point...

-Bill

-- started at 10/28/2017 17:51:38 --

mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
Command line used: /tmp/mdtest-binary-dir/mdtest
-d /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360 -u
-y
Path: /ibm/fs2-16m-10
FS: 128.1 TiB   Used FS: 0.3%   Inodes: 476.8 Mi   Used Inodes: 0.0%

228 tasks, 67239936 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean
Std Dev
   ---------                      ---            ---           ----
-------
   File creation     :      51953.498      50558.517      51423.221
616.643
   File stat         :          0.000          0.000          0.000
0.000
   File read         :          0.000          0.000          0.000
0.000
   File removal      :      96746.376      92149.535      94658.774
1900.187
   Tree creation     :          1.588          0.070          0.599
0.700
   Tree removal      :          0.213          0.034          0.097
0.082

-- finished at 10/28/2017 19:51:54 --

Bill Hartner
IBM Systems
Scalable I/O Development
Austin, Texas
bhartner at us.ibm.com
home office 512-784-0980


gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:

> From: Aaron Knister <aaron.knister at gmail.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 11/29/2017 04:42 PM
> Subject: Re: [gpfsug-discuss] Online data migration tool
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
> Thanks, Nikhil. Most of that was consistent with my understnading,
> however I was under the impression that the >32 subblocks code is
> required to achieve the touted 50k file creates/second that Sven has
> talked about a bunch of times:
>
>
http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf

> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
> _Sven_Oehme_-_News_from_Research.pdf
> http://files.gpfsug.org/presentations/2016/SC16/12_-
> _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf

> from those presentations regarding 32 subblocks:
>
> "It has a significant performance penalty for small files in large
> block size filesystems"

> although I'm not clear on the specific definition of "large". Many
> filesystems I encounter only have a 1M block size so it may not
> matter there, although that same presentation clearly shows the
> benefit of larger block sizes which is yet *another* thing for which
> a migration tool would be helpful.

> -Aaron
>
> On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com>
wrote:
> Hi,
>
> I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
> For all Spectrum Scale clusters that are currently at 4.X.X, it is
> possible to migrate to 5.0.0 with no offline data migration and no
> need to move data. Once these clusters are at 5.0.0, they will
> benefit from the performance improvements, new features (such as
> file audit logging), and various enhancements that are included in 5.0.0.
>
> That being said, there is one enhancement that will not be applied
> to these clusters, and that is the increased number of sub-blocks
> per block for small file allocation. This means that for file
> systems with a large block size and a lot of small files, the
> overall space utilization will be the same it currently is in 4.X.X.
> Since file systems created at 4.X.X and earlier used a block size
> that kept this allocation in mind, there should be very little
> impact on existing file systems.
>
> Outside of that one particular function, the remainder of the
> performance improvements, metadata improvements, updated
> compatibility, new functionality, and all of the other enhancements
> will be immediately available to you once you complete the upgrade
> to 5.0.0 -- with no need to reformat, move data, or take your data
offline.
>
> I hope that clarifies things a little and makes the upgrade path
> more accessible.
>
> Please let me know if there are any other questions or concerns.
>
> Thank you,
> Nikhil Khandelwal
> Spectrum Scale Development
> Client Adoption
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-

>
siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/18d266d7/attachment.htm>

From janfrode at tanso.net  Fri Dec  1 12:53:21 2017
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 01 Dec 2017 12:53:21 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
Message-ID: <CAHwPatj2=BUa1zHO453BBtAWvmwhxXoBqSByoEDYKg_PvZEa-A@mail.gmail.com>

Bill, could you say something about what the metadata-storage here was?
ESS/NL-SAS/3way replication?

I just asked about this in the internal slack channel #scale-help today..



-jf

fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <bhartner at us.ibm.com>:

> > "It has a significant performance penalty for small files in large
> > block size filesystems"
>
> Aaron,
>
> Below are mdtest results for a test we ran for CORAL - file size was 32k.
>
> We have not gone back and ran the test on a file system formatted without
> > 32 subblocks. We'll do that at some point...
>
> -Bill
>
> -- started at 10/28/2017 17:51:38 --
>
> mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> Command line used: /tmp/mdtest-binary-dir/mdtest -d
> /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360 -u -y
> Path: /ibm/fs2-16m-10
> FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>
> 228 tasks, 67239936 files
>
> SUMMARY: (of 3 iterations)
> Operation Max Min Mean Std Dev
> --------- --- --- ---- -------
> File creation : 51953.498 50558.517 51423.221 616.643
> File stat : 0.000 0.000 0.000 0.000
> File read : 0.000 0.000 0.000 0.000
> File removal : 96746.376 92149.535 94658.774 1900.187
> Tree creation : 1.588 0.070 0.599 0.700
> Tree removal : 0.213 0.034 0.097 0.082
>
> -- finished at 10/28/2017 19:51:54 --
>
> Bill Hartner
> IBM Systems
> Scalable I/O Development
> Austin, Texas
> bhartner at us.ibm.com
> home office 512-784-0980
>
>
> gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:
>
> > From: Aaron Knister <aaron.knister at gmail.com>
>
>
> > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>
> > Date: 11/29/2017 04:42 PM
>
>
> > Subject: Re: [gpfsug-discuss] Online data migration tool
> > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
> >
>
> > Thanks, Nikhil. Most of that was consistent with my understnading,
> > however I was under the impression that the >32 subblocks code is
> > required to achieve the touted 50k file creates/second that Sven has
> > talked about a bunch of times:
> >
> >
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf
> > http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
> > _Sven_Oehme_-_News_from_Research.pdf
> > http://files.gpfsug.org/presentations/2016/SC16/12_-
> > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>
>
> > from those presentations regarding 32 subblocks:
> >
> > "It has a significant performance penalty for small files in large
> > block size filesystems"
>
> > although I'm not clear on the specific definition of "large". Many
> > filesystems I encounter only have a 1M block size so it may not
> > matter there, although that same presentation clearly shows the
> > benefit of larger block sizes which is yet *another* thing for which
> > a migration tool would be helpful.
>
> > -Aaron
> >
> > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com>
> wrote:
>
> > Hi,
> >
> > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
> > For all Spectrum Scale clusters that are currently at 4.X.X, it is
> > possible to migrate to 5.0.0 with no offline data migration and no
> > need to move data. Once these clusters are at 5.0.0, they will
> > benefit from the performance improvements, new features (such as
> > file audit logging), and various enhancements that are included in 5.0.0.
> >
> > That being said, there is one enhancement that will not be applied
> > to these clusters, and that is the increased number of sub-blocks
> > per block for small file allocation. This means that for file
> > systems with a large block size and a lot of small files, the
> > overall space utilization will be the same it currently is in 4.X.X.
> > Since file systems created at 4.X.X and earlier used a block size
> > that kept this allocation in mind, there should be very little
> > impact on existing file systems.
> >
> > Outside of that one particular function, the remainder of the
> > performance improvements, metadata improvements, updated
> > compatibility, new functionality, and all of the other enhancements
> > will be immediately available to you once you complete the upgrade
> > to 5.0.0 -- with no need to reformat, move data, or take your data
> offline.
> >
> > I hope that clarifies things a little and makes the upgrade path
> > more accessible.
> >
> > Please let me know if there are any other questions or concerns.
> >
> > Thank you,
> > Nikhil Khandelwal
> > Spectrum Scale Development
> > Client Adoption
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
>
> > https://urldefense.proofpoint.com/v2/url?
> >
> u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
> >
> siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/a6a697b3/attachment.htm>

From luis.bolinches at fi.ibm.com  Fri Dec  1 14:00:19 2017
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Fri, 1 Dec 2017 14:00:19 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF9BFF17D3.E70240F6-ON002581E9.0046DD04@LocalDomain>
Message-ID: <OFECC4DC82.CE3786C2-ON002581E9.004CEEB3-1512136819057@notes.na.collabserv.com>

For Hana 2.0 only SP1 and 2 are supported

--
Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations
Luis Bolinches
Consultant IT Specialist
Mobile Phone: +358503112585
https://www.youracclaim.com/user/luis-bolinches

"If you always give you will always have" --  Anonymous

> On 1 Dec 2017, at 14.53, Jan-Frode Myklebust <janfrode at tanso.net> wrote:
> 
> Bill, could you say something about what the metadata-storage here was? ESS/NL-SAS/3way replication?
> 
> I just asked about this in the internal slack channel #scale-help today..
> 
> 
> 
> -jf
> 
>> fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <bhartner at us.ibm.com>:
>> > "It has a significant performance penalty for small files in large 
>> > block size filesystems"
>> 
>> 
>> Aaron,
>> 
>> Below are mdtest results for a test we ran for CORAL - file size was 32k.
>> 
>> We have not gone back and ran the test on a file system formatted without > 32 subblocks. We'll do that at some point...
>> 
>> -Bill
>> 
>> -- started at 10/28/2017 17:51:38 --
>> 
>> mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>> Command line used: /tmp/mdtest-binary-dir/mdtest -d /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360 -u -y
>> Path: /ibm/fs2-16m-10
>> FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>> 
>> 228 tasks, 67239936 files
>> 
>> SUMMARY: (of 3 iterations)
>> Operation Max Min Mean Std Dev
>> --------- --- --- ---- -------
>> File creation : 51953.498 50558.517 51423.221 616.643
>> File stat : 0.000 0.000 0.000 0.000
>> File read : 0.000 0.000 0.000 0.000
>> File removal : 96746.376 92149.535 94658.774 1900.187
>> Tree creation : 1.588 0.070 0.599 0.700
>> Tree removal : 0.213 0.034 0.097 0.082
>> 
>> -- finished at 10/28/2017 19:51:54 --
>> 
>> Bill Hartner
>> IBM Systems
>> Scalable I/O Development
>> Austin, Texas
>> bhartner at us.ibm.com
>> home office 512-784-0980
>> 
>> 
>> gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:
>> 
>> > From: Aaron Knister <aaron.knister at gmail.com>
>> 
>> 
>> > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> 
>> > Date: 11/29/2017 04:42 PM
>> 
>> 
>> > Subject: Re: [gpfsug-discuss] Online data migration tool
>> > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> 
>> > 
>> 
>> > Thanks, Nikhil. Most of that was consistent with my understnading, 
>> > however I was under the impression that the >32 subblocks code is 
>> > required to achieve the touted 50k file creates/second that Sven has
>> > talked about a bunch of times:
>> > 
>> > http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf
>> > http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
>> > _Sven_Oehme_-_News_from_Research.pdf
>> > http://files.gpfsug.org/presentations/2016/SC16/12_-
>> > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>> 
>> 
>> > from those presentations regarding 32 subblocks:
>> > 
>> > "It has a significant performance penalty for small files in large 
>> > block size filesystems"
>> 
>> > although I'm not clear on the specific definition of "large". Many 
>> > filesystems I encounter only have a 1M block size so it may not 
>> > matter there, although that same presentation clearly shows the 
>> > benefit of larger block sizes which is yet *another* thing for which
>> > a migration tool would be helpful.
>> 
>> > -Aaron
>> > 
>> > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com> wrote:
>> 
>> > Hi,
>> > 
>> > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
>> > For all Spectrum Scale clusters that are currently at 4.X.X, it is 
>> > possible to migrate to 5.0.0 with no offline data migration and no 
>> > need to move data. Once these clusters are at 5.0.0, they will 
>> > benefit from the performance improvements, new features (such as 
>> > file audit logging), and various enhancements that are included in 5.0.0.
>> > 
>> > That being said, there is one enhancement that will not be applied 
>> > to these clusters, and that is the increased number of sub-blocks 
>> > per block for small file allocation. This means that for file 
>> > systems with a large block size and a lot of small files, the 
>> > overall space utilization will be the same it currently is in 4.X.X.
>> > Since file systems created at 4.X.X and earlier used a block size 
>> > that kept this allocation in mind, there should be very little 
>> > impact on existing file systems.
>> > 
>> > Outside of that one particular function, the remainder of the 
>> > performance improvements, metadata improvements, updated 
>> > compatibility, new functionality, and all of the other enhancements 
>> > will be immediately available to you once you complete the upgrade 
>> > to 5.0.0 -- with no need to reformat, move data, or take your data offline.
>> > 
>> > I hope that clarifies things a little and makes the upgrade path 
>> > more accessible.
>> > 
>> > Please let me know if there are any other questions or concerns.
>> > 
>> > Thank you,
>> > Nikhil Khandelwal
>> > Spectrum Scale Development
>> > Client Adoption
>> > 
>> > _______________________________________________
>> > gpfsug-discuss mailing list
>> > gpfsug-discuss at spectrumscale.org
>> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> 
>> > _______________________________________________
>> > gpfsug-discuss mailing list
>> > gpfsug-discuss at spectrumscale.org
>> 
>> > https://urldefense.proofpoint.com/v2/url?
>> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>> > siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3 
Registered in Finland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/e09df482/attachment.htm>

From bhartner at us.ibm.com  Fri Dec  1 16:44:23 2017
From: bhartner at us.ibm.com (Bill Hartner)
Date: Fri, 1 Dec 2017 10:44:23 -0600
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com><OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain><OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
Message-ID: <OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>


ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the results @
user group meeting.

-Bill

Bill Hartner
IBM Systems
Scalable I/O Development
Austin, Texas
bhartner at us.ibm.com
home office 512-784-0980




From:	Jan-Frode Myklebust <janfrode at tanso.net>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	12/01/2017 06:53 AM
Subject:	Re: [gpfsug-discuss] Online data migration tool
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Bill, could you say something about what the metadata-storage here was?
ESS/NL-SAS/3way replication?

I just asked about this in the internal slack channel #scale-help today..



-jf

fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <bhartner at us.ibm.com>:
  > "It has a significant performance penalty for small files in large
  > block size filesystems"



  Aaron,

  Below are mdtest results for a test we ran for CORAL - file size was 32k.

  We have not gone back and ran the test on a file system formatted without
  > 32 subblocks. We'll do that at some point...

  -Bill

  -- started at 10/28/2017 17:51:38 --

  mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
  Command line used: /tmp/mdtest-binary-dir/mdtest
  -d /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360
  -u -y
  Path: /ibm/fs2-16m-10
  FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%

  228 tasks, 67239936 files

  SUMMARY: (of 3 iterations)
  Operation Max Min Mean Std Dev
  --------- --- --- ---- -------
  File creation : 51953.498 50558.517 51423.221 616.643
  File stat : 0.000 0.000 0.000 0.000
  File read : 0.000 0.000 0.000 0.000
  File removal : 96746.376 92149.535 94658.774 1900.187
  Tree creation : 1.588 0.070 0.599 0.700
  Tree removal : 0.213 0.034 0.097 0.082

  -- finished at 10/28/2017 19:51:54 --

  Bill Hartner
  IBM Systems
  Scalable I/O Development
  Austin, Texas
  bhartner at us.ibm.com
  home office 512-784-0980


  gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:

  > From: Aaron Knister <aaron.knister at gmail.com>



  > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>


  > Date: 11/29/2017 04:42 PM



  > Subject: Re: [gpfsug-discuss] Online data migration tool
  > Sent by: gpfsug-discuss-bounces at spectrumscale.org


  >


  > Thanks, Nikhil. Most of that was consistent with my understnading,
  > however I was under the impression that the >32 subblocks code is
  > required to achieve the touted 50k file creates/second that Sven has
  > talked about a bunch of times:
  >
  >
  http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf

  > http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
  > _Sven_Oehme_-_News_from_Research.pdf
  > http://files.gpfsug.org/presentations/2016/SC16/12_-
  > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf



  > from those presentations regarding 32 subblocks:
  >
  > "It has a significant performance penalty for small files in large
  > block size filesystems"

  > although I'm not clear on the specific definition of "large". Many
  > filesystems I encounter only have a 1M block size so it may not
  > matter there, although that same presentation clearly shows the
  > benefit of larger block sizes which is yet *another* thing for which
  > a migration tool would be helpful.

  > -Aaron
  >
  > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com>
  wrote:


  > Hi,
  >
  > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
  > For all Spectrum Scale clusters that are currently at 4.X.X, it is
  > possible to migrate to 5.0.0 with no offline data migration and no
  > need to move data. Once these clusters are at 5.0.0, they will
  > benefit from the performance improvements, new features (such as
  > file audit logging), and various enhancements that are included in
  5.0.0.
  >
  > That being said, there is one enhancement that will not be applied
  > to these clusters, and that is the increased number of sub-blocks
  > per block for small file allocation. This means that for file
  > systems with a large block size and a lot of small files, the
  > overall space utilization will be the same it currently is in 4.X.X.
  > Since file systems created at 4.X.X and earlier used a block size
  > that kept this allocation in mind, there should be very little
  > impact on existing file systems.
  >
  > Outside of that one particular function, the remainder of the
  > performance improvements, metadata improvements, updated
  > compatibility, new functionality, and all of the other enhancements
  > will be immediately available to you once you complete the upgrade
  > to 5.0.0 -- with no need to reformat, move data, or take your data
  offline.
  >
  > I hope that clarifies things a little and makes the upgrade path
  > more accessible.
  >
  > Please let me know if there are any other questions or concerns.
  >
  > Thank you,
  > Nikhil Khandelwal
  > Spectrum Scale Development
  > Client Adoption
  >
  > _______________________________________________
  > gpfsug-discuss mailing list
  > gpfsug-discuss at spectrumscale.org
  > http://gpfsug.org/mailman/listinfo/gpfsug-discuss



  > _______________________________________________
  > gpfsug-discuss mailing list
  > gpfsug-discuss at spectrumscale.org


  > https://urldefense.proofpoint.com/v2/url?
  >
  u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-

  >
  siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=


  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at spectrumscale.org
  http://gpfsug.org/mailman/listinfo/gpfsug-discuss
  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at spectrumscale.org
  https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/60b7fce0/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/60b7fce0/attachment.gif>

From r.sobey at imperial.ac.uk  Fri Dec  1 16:57:46 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Fri, 1 Dec 2017 16:57:46 +0000
Subject: [gpfsug-discuss] Policy rule to list files based on access time
Message-ID: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Hi all,

I'm getting some strange results using a policy to list all the files (well, I'm only interested in the total size that GPFS chooses) that have not been accessed for more than 1/2/3 etc years.

The policy rule I'm using is:

RULE 'list_old' LIST 'atime_gt_730'
WHERE DAYS(CURRENT_TIMESTAMP) -DAYS(ACCESS_TIME) > 730

..for anything over 2 years. My filesystem is 3.5 years old and we first started migrating to it about three years ago, yet although GPFS is choosing ~400000 files that match, the total size is just north of 11000KB:

[I] Summary of Rule Applicability and File Choices:
Rule#      Hit_Cnt          KB_Hit          Chosen       KB_Chosen          KB_Ill     Rule
     0       453765           11776          453765           11776               0     RULE 'list_old' LIST 'atime_gt_730' WHERE(.)

I'm having a hard time believing half a million files total 11MB especially given the very random spread of usage the system sees.

The mmapplypolicy command I'm running is:

/usr/lpp/mmfs/bin/mmapplypolicy /gpfs/path -f /gpfs/policies/ -I defer --scope filesystem -P ./list_old_allfs.txt

The directory 'path' above is the root of the main CES share which encompasses all my filesets.

This is my first real foray into using the policy engine so I might be doing something silly, or misinterpreting the results, or there is simply a far easier way to achieve my goal.

Can anyone help?

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/69ada640/attachment.htm>

From alex at calicolabs.com  Fri Dec  1 17:18:57 2017
From: alex at calicolabs.com (Alex Chekholko)
Date: Fri, 1 Dec 2017 09:18:57 -0800
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <CANcy_PbUS93Y7TMxL2f+LcH7_4fJk0SkpJED54n3h=2Ju5Lzfg@mail.gmail.com>

Hi Richard,

Either add something like "-L 2" to your mmapplypolicy command or add an
'EXEC ...' to your policy file to have it print out the actual files that
matched. There is an example exec script somewhere in the samples folder

I have found that in the past when I doubted the policy results, there was
something I misunderstood. So just go ahead and examine the metadata on the
actual files that match.

Perhaps at some point in the past, you had some kind of application scan
through the filesystem and thus all the access times got updated to newer
than you expect.

For the file count/size numbers, I don't find that unusual; there are all
kinds of user applications that have pathological patterns like 450k files
totaling 11MB.  But worth double-checking.  You can run some example
policies like the one you ran, but list only >10TB files or list only files
of 0 size; you never know what you might find.

Regards,
Alex


On Fri, Dec 1, 2017 at 8:57 AM, Sobey, Richard A <r.sobey at imperial.ac.uk>
wrote:

> Hi all,
>
>
>
> I?m getting some strange results using a policy to list all the files
> (well, I?m only interested in the total size that GPFS chooses) that have
> not been accessed for more than 1/2/3 etc years.
>
>
>
> The policy rule I?m using is:
>
>
>
> RULE 'list_old' LIST 'atime_gt_730'
>
> WHERE DAYS(CURRENT_TIMESTAMP) -DAYS(ACCESS_TIME) > 730
>
>
>
> ..for anything over 2 years. My filesystem is 3.5 years old and we first
> started migrating to it about three years ago, yet although GPFS is
> choosing ~400000 files that match, the total size is just north of 11000KB:
>
>
>
> [I] Summary of Rule Applicability and File Choices:
>
> Rule#      Hit_Cnt          KB_Hit          Chosen
> KB_Chosen          KB_Ill     Rule
>
>      0       453765           11776          453765
> 11776               0     RULE 'list_old' LIST 'atime_gt_730' WHERE(.)
>
>
>
> I?m having a hard time believing half a million files total 11MB
> especially given the very random spread of usage the system sees.
>
>
>
> The mmapplypolicy command I?m running is:
>
>
>
> /usr/lpp/mmfs/bin/mmapplypolicy /gpfs/path -f /gpfs/policies/ -I defer
> --scope filesystem -P ./list_old_allfs.txt
>
>
>
> The directory ?path? above is the root of the main CES share which
> encompasses all my filesets.
>
>
>
> This is my first real foray into using the policy engine so I might be
> doing something silly, or misinterpreting the results, or there is simply a
> far easier way to achieve my goal.
>
>
>
> Can anyone help?
>
>
>
> Thanks
>
> Richard
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/1395f933/attachment.htm>

From ewahl at osc.edu  Fri Dec  1 17:59:21 2017
From: ewahl at osc.edu (Edward Wahl)
Date: Fri, 1 Dec 2017 12:59:21 -0500
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <20171201125921.507043a3@osc.edu>


You don't specify a choice algorithm?  Might want to change your script with a
quick thing like this to view all the candidates.  I use a similar 120day and
I've looked at atime/ctimes of them before and was happy with the output.
You can then quickly size the output with a quick sed/awk/whatever.

SHOW (varchar(file_size) || '  ' || varchar(USER_ID))
WHERE DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME) >= 180 

and add "--choice-algorithm exact" to the command line to see what files match. 

I can't recall what the default is. Might already be exact?  Oops, man pages
says yes.  Might then run all 3 and see if the output is much different. 

Ed

On Fri, 1 Dec 2017 16:57:46 +0000
"Sobey, Richard A" <r.sobey at imperial.ac.uk> wrote:

> Hi all,
> 
> I'm getting some strange results using a policy to list all the files (well,
> I'm only interested in the total size that GPFS chooses) that have not been
> accessed for more than 1/2/3 etc years.
> 
> The policy rule I'm using is:
> 
> RULE 'list_old' LIST 'atime_gt_730'
> WHERE DAYS(CURRENT_TIMESTAMP) -DAYS(ACCESS_TIME) > 730
> 
> ..for anything over 2 years. My filesystem is 3.5 years old and we first
> started migrating to it about three years ago, yet although GPFS is choosing
> ~400000 files that match, the total size is just north of 11000KB:
> 
> [I] Summary of Rule Applicability and File Choices:
> Rule#      Hit_Cnt          KB_Hit          Chosen       KB_Chosen
> KB_Ill     Rule 0       453765           11776          453765
> 11776               0     RULE 'list_old' LIST 'atime_gt_730' WHERE(.)
> 
> I'm having a hard time believing half a million files total 11MB especially
> given the very random spread of usage the system sees.
> 
> The mmapplypolicy command I'm running is:
> 
> /usr/lpp/mmfs/bin/mmapplypolicy /gpfs/path -f /gpfs/policies/ -I defer
> --scope filesystem -P ./list_old_allfs.txt
> 
> The directory 'path' above is the root of the main CES share which
> encompasses all my filesets.
> 
> This is my first real foray into using the policy engine so I might be doing
> something silly, or misinterpreting the results, or there is simply a far
> easier way to achieve my goal.
> 
> Can anyone help?
> 
> Thanks
> Richard



-- 

Ed Wahl
Ohio Supercomputer Center
614-292-9302


From makaplan at us.ibm.com  Sat Dec  2 18:03:25 2017
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Sat, 2 Dec 2017 13:03:25 -0500
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>
Message-ID: <OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>

You might want to look at FILE_SIZE.  KB_ALLOCATED will be 0 if the file 
data fits into the inode.

You might also want to use SIZE(FILE_SIZE) in the policy LIST rule, this 
will cause the KB_HIT and KB_CHOSEN numbers to be the sum of FILE_SIZEs 
instead of the default SIZE(KB_ALLOCATED).

--marc of GPFS

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171202/1703340b/attachment.htm>

From r.sobey at imperial.ac.uk  Sat Dec  2 18:53:14 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Sat, 2 Dec 2017 18:53:14 +0000
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>,
	<OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>
Message-ID: <DB6PR0601MB210125AFF8783FFFAB2BF3E4DF3E0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Thank you all for your replies. I will take a look at them on Monday. And reply individually where I need to..

Richard

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Marc A Kaplan <makaplan at us.ibm.com>
Sent: Saturday, December 2, 2017 6:03:25 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Policy rule to list files based on access time

You might want to look at FILE_SIZE.  KB_ALLOCATED will be 0 if the file data fits into the inode.

You might also want to use SIZE(FILE_SIZE) in the policy LIST rule, this will cause the KB_HIT and KB_CHOSEN numbers to be the sum of FILE_SIZEs instead of the default SIZE(KB_ALLOCATED).

--marc of GPFS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171202/815f8412/attachment.htm>

From bipcuds at gmail.com  Mon Dec  4 02:49:06 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Sun, 3 Dec 2017 21:49:06 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file quota/limit
	possible to set?
Message-ID: <CAAxuGpEFH7mi80Aa2u_LbaHP+3JU1y4qKQPFOB+HohCD76XZ2Q@mail.gmail.com>

From: Keith Ball <kball at redlineperf.com>
To: gpfsug-discuss at spectrumscale.org
Cc:
Bcc:
Date: Sun, 3 Dec 2017 16:13:13 -0500
Subject: Smallest block quota/limit and file quota/limit possible to set?
HI All,

We have a system where all users have their own private group as well.
However, for a given fileset (we are using --perfileset-quota), we would
like to ONLY allow users who also belong to just a few central groups to be
able to write to the fileset.

That is, user "userA" has its own "groupA", but we only want the user to be
able to write to the fileset if:
 - userA belongs to one of the groups (e.g. group1, group2, group3) that
have explicitly set quotas
 - The group(s) in question are within quota/limits.

In general, we do not want any users that do NOT belong to one of the three
groups with enabled quotas to be able to write anything at all to the
fileset.

Is there a way to set a ZERO quota for block/file in GPFS, that means what
it actually should mean? i.e. "Your limit is 0 file = you cannot create
files in this fileset". Creating some kind of "supergroup" owner of the
fileset (with entitled users as members of the group) could work, but that
will only work for *one* group.

If we cannot set the block and file limits to zero, what *are* the smallest
block and fie limits? In GPFS 3.5, they seem to be 1760MB for block. Is
there a smallest quota for files? (blocksize is 16MB, which will be reduced
to 4MB probably, in a subsequent cluster).

Many Thanks,
  Keith



-- 
Keith D. Ball, PhD
RedLine Performance Solutions, LLC
web:  http://www.redlineperf.com/
email: kball at redlineperf.com <aqualkenbush at redlineperf.com>
cell: 540-557-7851 <(540)%20557-7851>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171203/8c867510/attachment.htm>

From scale at us.ibm.com  Mon Dec  4 03:49:38 2017
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Mon, 4 Dec 2017 09:19:38 +0530
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit	possible to set?
In-Reply-To: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
Message-ID: <OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>

Hi Keith,

You can use ACLs for fine grained permissions. A quota limit of 0 in GPFS 
implies no limits.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   Keith Ball <bipcuds at gmail.com>
To:     gpfsug-discuss at spectrumscale.org
Date:   12/04/2017 08:19 AM
Subject:        [gpfsug-discuss] Smallest block quota/limit and file 
quota/limit     possible to set?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



From: Keith Ball <kball at redlineperf.com>
To: gpfsug-discuss at spectrumscale.org
Cc: 
Bcc: 
Date: Sun, 3 Dec 2017 16:13:13 -0500
Subject: Smallest block quota/limit and file quota/limit possible to set?
HI All,

We have a system where all users have their own private group as well. 
However, for a given fileset (we are using --perfileset-quota), we would 
like to ONLY allow users who also belong to just a few central groups to 
be able to write to the fileset. 

That is, user "userA" has its own "groupA", but we only want the user to 
be able to write to the fileset if:
 - userA belongs to one of the groups (e.g. group1, group2, group3) that 
have explicitly set quotas
 - The group(s) in question are within quota/limits.

In general, we do not want any users that do NOT belong to one of the 
three groups with enabled quotas to be able to write anything at all to 
the fileset.

Is there a way to set a ZERO quota for block/file in GPFS, that means what 
it actually should mean? i.e. "Your limit is 0 file = you cannot create 
files in this fileset". Creating some kind of "supergroup" owner of the 
fileset (with entitled users as members of the group) could work, but that 
will only work for *one* group. 

If we cannot set the block and file limits to zero, what *are* the 
smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB for 
block. Is there a smallest quota for files? (blocksize is 16MB, which will 
be reduced to 4MB probably, in a subsequent cluster).

Many Thanks,
  Keith 



-- 
Keith D. Ball, PhD
RedLine Performance Solutions, LLC
web:  http://www.redlineperf.com/
email: kball at redlineperf.com
cell: 540-557-7851_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=N0b8UJpG1zFuuQjvCE1SaaIG2EUDv4v783wDc3TbnyM&s=vuNQN4VJonV9nmcTEOKKr-hL2LAWSOUV7HKAQBKWcK4&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/0c8ffd87/attachment.htm>

From z.han at imperial.ac.uk  Mon Dec  4 09:52:31 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Mon, 4 Dec 2017 09:52:31 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or above?
Message-ID: <alpine.LFD.2.20.1712040941230.2395@laptop>

Hi All,

Any one is using a Linux kernel 3.12.x or above
to run gpfs 4.2.3-4.2? I mean you've compiled
your own kernel without paying for a professional
service.

We're stuck by CentOS/RHEL's distributed kernel
as the PCI passthrough is required for VMs. Your
comments or suggestions are much appreciated.

Kind regards,

Zong-Pei



From S.J.Thompson at bham.ac.uk  Mon Dec  4 10:29:27 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Mon, 4 Dec 2017 10:29:27 +0000
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712040941230.2395@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
Message-ID: <B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>

The FAQ at:

https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux

Lists support with (e.g. Ubutu 16.04.2) with kernel 4.4.0-62, so likely it would work with a build your own kernel, but that doesn?t mean it is **supported**

Simon

On 04/12/2017, 09:52, "gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk" <gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk> wrote:

    Hi All,
    
    Any one is using a Linux kernel 3.12.x or above
    to run gpfs 4.2.3-4.2? I mean you've compiled
    your own kernel without paying for a professional
    service.
    
    We're stuck by CentOS/RHEL's distributed kernel
    as the PCI passthrough is required for VMs. Your
    comments or suggestions are much appreciated.
    
    Kind regards,
    
    Zong-Pei
    
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
    


From z.han at imperial.ac.uk  Mon Dec  4 10:37:39 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Mon, 4 Dec 2017 10:37:39 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
	<B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>
Message-ID: <alpine.LFD.2.20.1712041036340.2395@laptop>

Thanks, Simon.

Only on clients (oh so-called clients!) ;)


On Mon, 4 Dec 2017, Simon Thompson (IT Research Support) wrote:
> Date: Mon, 4 Dec 2017 10:29:27 +0000
> From: "Simon Thompson (IT Research Support)" <S.J.Thompson at bham.ac.uk>
> Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
>     above?
> 
> The FAQ at:
>
> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux
>
> Lists support with (e.g. Ubutu 16.04.2) with kernel 4.4.0-62, so likely it would work with a build your own kernel, but that doesn?t mean it is **supported**
>
> Simon
>
> On 04/12/2017, 09:52, "gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk" <gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk> wrote:
>
>    Hi All,
>
>    Any one is using a Linux kernel 3.12.x or above
>    to run gpfs 4.2.3-4.2? I mean you've compiled
>    your own kernel without paying for a professional
>    service.
>
>    We're stuck by CentOS/RHEL's distributed kernel
>    as the PCI passthrough is required for VMs. Your
>    comments or suggestions are much appreciated.
>
>    Kind regards,
>
>    Zong-Pei
>
>    _______________________________________________
>    gpfsug-discuss mailing list
>    gpfsug-discuss at spectrumscale.org
>    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From ulmer at ulmer.org  Mon Dec  4 13:46:31 2017
From: ulmer at ulmer.org (Stephen Ulmer)
Date: Mon, 4 Dec 2017 08:46:31 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit	possible to set?
In-Reply-To: <OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
Message-ID: <8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>

I don?t understand why not having permission(s) doesn?t prevent the user from writing into the fileset...

As described, your case is about not wanting userA to be able to write to a fileset if userA isn?t in some groups. Don?t put them in those groups. That?s not even Spectrum Scale specific, it?s about generic *nix permissions.

What am I missing? I don?t understand why you would want to use quota to enforce permissions. (There could be a legitimate reason here, but I don?t understand it.)

Liberty,

-- 
Stephen Ulmer

Sent from a mobile device; please excuse autocorrect silliness.

> On Dec 3, 2017, at 10:49 PM, IBM Spectrum Scale <scale at us.ibm.com> wrote:
> 
> Hi Keith,
> 
> You can use ACLs for fine grained permissions. A quota limit of 0 in GPFS implies no limits.
> 
> Regards, The Spectrum Scale (GPFS) team
> 
> ------------------------------------------------------------------------------------------------------------------
> If you feel that your question can benefit other users of  Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. 
> 
> If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact  1-800-237-5511 in the United States or your local IBM Service Center in other countries. 
> 
> The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.
> 
> 
> 
> From:        Keith Ball <bipcuds at gmail.com>
> To:        gpfsug-discuss at spectrumscale.org
> Date:        12/04/2017 08:19 AM
> Subject:        [gpfsug-discuss] Smallest block quota/limit and file quota/limit        possible to set?
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> 
> 
> 
> From: Keith Ball <kball at redlineperf.com>
> To: gpfsug-discuss at spectrumscale.org
> Cc: 
> Bcc: 
> Date: Sun, 3 Dec 2017 16:13:13 -0500
> Subject: Smallest block quota/limit and file quota/limit possible to set?
> HI All,
> 
> We have a system where all users have their own private group as well. However, for a given fileset (we are using --perfileset-quota), we would like to ONLY allow users who also belong to just a few central groups to be able to write to the fileset. 
> 
> That is, user "userA" has its own "groupA", but we only want the user to be able to write to the fileset if:
>  - userA belongs to one of the groups (e.g. group1, group2, group3) that have explicitly set quotas
>  - The group(s) in question are within quota/limits.
> 
> In general, we do not want any users that do NOT belong to one of the three groups with enabled quotas to be able to write anything at all to the fileset.
> 
> Is there a way to set a ZERO quota for block/file in GPFS, that means what it actually should mean? i.e. "Your limit is 0 file = you cannot create files in this fileset". Creating some kind of "supergroup" owner of the fileset (with entitled users as members of the group) could work, but that will only work for *one* group. 
> 
> If we cannot set the block and file limits to zero, what *are* the smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB for block. Is there a smallest quota for files? (blocksize is 16MB, which will be reduced to 4MB probably, in a subsequent cluster).
> 
> Many Thanks,
>   Keith 
> 
> 
> 
> -- 
> Keith D. Ball, PhD
> RedLine Performance Solutions, LLC
> web:  http://www.redlineperf.com/
> email: kball at redlineperf.com
> cell: 540-557-7851_______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=N0b8UJpG1zFuuQjvCE1SaaIG2EUDv4v783wDc3TbnyM&s=vuNQN4VJonV9nmcTEOKKr-hL2LAWSOUV7HKAQBKWcK4&e=
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/9b42e1fc/attachment.htm>

From valdis.kletnieks at vt.edu  Mon Dec  4 16:27:38 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Mon, 04 Dec 2017 11:27:38 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit possible to set?
In-Reply-To: <8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
Message-ID: <19755.1512404858@turing-police.cc.vt.edu>

On Mon, 04 Dec 2017 08:46:31 -0500, Stephen Ulmer said:

> As described, your case is about not wanting userA to be able to write to a
> fileset if userA isn???t in some groups. Don???t put them in those groups. That???s
> not even Spectrum Scale specific, it???s about generic *nix permissions.

The problem is that doesn't stop userA from scribbling into a world-writable directory,
and they may have some of those because more than one group needs to write in
a specific directory and ACLs are problematic for other reasons....
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/18b3756a/attachment.sig>

From stuartb at 4gh.net  Mon Dec  4 16:33:17 2017
From: stuartb at 4gh.net (Stuart Barkley)
Date: Mon, 4 Dec 2017 11:33:17 -0500 (EST)
Subject: [gpfsug-discuss] Smallest block quota/limit and file
 quota/limit possible to set?
In-Reply-To: <8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
Message-ID: <alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>

We have a /projects filesystem where individual projects can "buy" a
specific amount of disk space.  We enforce this purchase limit by
creating a specific group for the allocation, adding designated users
to the group and setting a group quota.

This works fine as long as the users properly use setgid directories
and keep proper group ownership of the files and directories.

However, for various reasons our users keep creating files and
directories with incorrect group ownership.  In most cases this is
accidental and eventually causes problems when other group members
need to access the files.  In abusive cases (not yet seen) people
could use this to exceed project disk space allocations.

To address this problem we have default quotas set to about 2GB (the
smallest we seem to be able to set).  This prevents users for
consuming too much unpurchased disk space.  However, this continues to
allow users to create files and directories with incorrect group
ownership and it takes users a while to discover their error.  User
education and cleanup becomes a problem long after the user thinks
things are working.

We would like to have groups without quota definitions to not be able
to create any files.  This would prevent accidental file creation at
the first attempt.

Stuart Barkley

On Mon, 4 Dec 2017 at 08:46 -0000, Stephen Ulmer wrote:

> I don?t understand why not having permission(s) doesn?t prevent the
> user from writing into the fileset...
>
> As described, your case is about not wanting userA to be able to
> write to a fileset if userA isn?t in some groups. Don?t put them in
> those groups. That?s not even Spectrum Scale specific, it?s about
> generic *nix permissions.
>
> What am I missing? I don?t understand why you would want to use
> quota to enforce permissions. (There could be a legitimate reason
> here, but I don?t understand it.)
>
> Liberty,
>
> --
> Stephen Ulmer
>
> Sent from a mobile device; please excuse autocorrect silliness.
>
> > On Dec 3, 2017, at 10:49 PM, IBM Spectrum Scale <scale at us.ibm.com> wrote:
> >
> > Hi Keith,
> >
> > You can use ACLs for fine grained permissions. A quota limit of 0
> > in GPFS implies no limits.
> >
> > Regards, The Spectrum Scale (GPFS) team
> >
> > From:        Keith Ball <bipcuds at gmail.com>
> > To:        gpfsug-discuss at spectrumscale.org
> > Date:        12/04/2017 08:19 AM
> > Subject:        [gpfsug-discuss] Smallest block quota/limit and file quota/limit        possible to set?
> > Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> >
> > HI All,
> >
> > We have a system where all users have their own private group as
> > well. However, for a given fileset (we are using
> > --perfileset-quota), we would like to ONLY allow users who also
> > belong to just a few central groups to be able to write to the
> > fileset.
> >
> > That is, user "userA" has its own "groupA", but we only want the
> > user to be able to write to the fileset if:
> >  - userA belongs to one of the groups (e.g. group1, group2,
> >  group3) that have explicitly set quotas
> >  - The group(s) in question are within quota/limits.
> >
> > In general, we do not want any users that do NOT belong to one of
> > the three groups with enabled quotas to be able to write anything
> > at all to the fileset.
> >
> > Is there a way to set a ZERO quota for block/file in GPFS, that
> > means what it actually should mean? i.e. "Your limit is 0 file =
> > you cannot create files in this fileset". Creating some kind of
> > "supergroup" owner of the fileset (with entitled users as members
> > of the group) could work, but that will only work for *one* group.
> >
> > If we cannot set the block and file limits to zero, what *are* the
> > smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB
> > for block. Is there a smallest quota for files? (blocksize is
> > 16MB, which will be reduced to 4MB probably, in a subsequent
> > cluster).
> >
> > Many Thanks,
> >   Keith


From S.J.Thompson at bham.ac.uk  Mon Dec  4 17:08:19 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Mon, 4 Dec 2017 17:08:19 +0000
Subject: [gpfsug-discuss] Smallest block quota/limit and file
 quota/limit possible to set?
In-Reply-To: <alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>,
	<alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>

Stuart,

Have you looked at using filesets instead an using fileset quotas to achieve this?

This is what we do and the max number of filesets (currently) isn't an issue for us.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of stuartb at 4gh.net [stuartb at 4gh.net]
Sent: 04 December 2017 16:33
To: gpfsug main discussion list
Cc: gpfsug-discuss-bounces at spectrumscale.org
Subject: Re: [gpfsug-discuss] Smallest block quota/limit and file quota/limit possible to set?

We have a /projects filesystem where individual projects can "buy" a
specific amount of disk space.  We enforce this purchase limit by
creating a specific group for the allocation, adding designated users
to the group and setting a group quota.

This works fine as long as the users properly use setgid directories
and keep proper group ownership of the files and directories.

However, for various reasons our users keep creating files and
directories with incorrect group ownership.  In most cases this is
accidental and eventually causes problems when other group members
need to access the files.  In abusive cases (not yet seen) people
could use this to exceed project disk space allocations.

To address this problem we have default quotas set to about 2GB (the
smallest we seem to be able to set).  This prevents users for
consuming too much unpurchased disk space.  However, this continues to
allow users to create files and directories with incorrect group
ownership and it takes users a while to discover their error.  User
education and cleanup becomes a problem long after the user thinks
things are working.

We would like to have groups without quota definitions to not be able
to create any files.  This would prevent accidental file creation at
the first attempt.

Stuart Barkley

On Mon, 4 Dec 2017 at 08:46 -0000, Stephen Ulmer wrote:

> I don?t understand why not having permission(s) doesn?t prevent the
> user from writing into the fileset...
>
> As described, your case is about not wanting userA to be able to
> write to a fileset if userA isn?t in some groups. Don?t put them in
> those groups. That?s not even Spectrum Scale specific, it?s about
> generic *nix permissions.
>
> What am I missing? I don?t understand why you would want to use
> quota to enforce permissions. (There could be a legitimate reason
> here, but I don?t understand it.)
>
> Liberty,
>
> --
> Stephen Ulmer
>
> Sent from a mobile device; please excuse autocorrect silliness.
>
> > On Dec 3, 2017, at 10:49 PM, IBM Spectrum Scale <scale at us.ibm.com> wrote:
> >
> > Hi Keith,
> >
> > You can use ACLs for fine grained permissions. A quota limit of 0
> > in GPFS implies no limits.
> >
> > Regards, The Spectrum Scale (GPFS) team
> >
> > From:        Keith Ball <bipcuds at gmail.com>
> > To:        gpfsug-discuss at spectrumscale.org
> > Date:        12/04/2017 08:19 AM
> > Subject:        [gpfsug-discuss] Smallest block quota/limit and file quota/limit        possible to set?
> > Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> >
> > HI All,
> >
> > We have a system where all users have their own private group as
> > well. However, for a given fileset (we are using
> > --perfileset-quota), we would like to ONLY allow users who also
> > belong to just a few central groups to be able to write to the
> > fileset.
> >
> > That is, user "userA" has its own "groupA", but we only want the
> > user to be able to write to the fileset if:
> >  - userA belongs to one of the groups (e.g. group1, group2,
> >  group3) that have explicitly set quotas
> >  - The group(s) in question are within quota/limits.
> >
> > In general, we do not want any users that do NOT belong to one of
> > the three groups with enabled quotas to be able to write anything
> > at all to the fileset.
> >
> > Is there a way to set a ZERO quota for block/file in GPFS, that
> > means what it actually should mean? i.e. "Your limit is 0 file =
> > you cannot create files in this fileset". Creating some kind of
> > "supergroup" owner of the fileset (with entitled users as members
> > of the group) could work, but that will only work for *one* group.
> >
> > If we cannot set the block and file limits to zero, what *are* the
> > smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB
> > for block. Is there a smallest quota for files? (blocksize is
> > 16MB, which will be reduced to 4MB probably, in a subsequent
> > cluster).
> >
> > Many Thanks,
> >   Keith
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From stuartb at 4gh.net  Mon Dec  4 20:24:50 2017
From: stuartb at 4gh.net (Stuart Barkley)
Date: Mon, 4 Dec 2017 15:24:50 -0500 (EST)
Subject: [gpfsug-discuss] Smallest block quota/limit and file
 quota/limit possible to set?
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>,
	<alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
Message-ID: <alpine.BSF.2.21.1712041521500.34093@freeman.4gh.net>

On Mon, 4 Dec 2017 at 12:08 -0000, Simon Thompson (IT Research Support) wrote:

> Have you looked at using filesets instead an using fileset quotas to
> achieve this?
>
> This is what we do and the max number of filesets (currently) isn't
> an issue for us.

This is a possibility.  We will be moving to a new infrastructure in a
couple of months and I'll have our administrator and contractor
discuss this possibility.

Thanks,
Stuart
-- 
I've never been lost; I was once bewildered for three days, but never lost!
                                        --  Daniel Boone


From valdis.kletnieks at vt.edu  Mon Dec  4 20:44:34 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Mon, 04 Dec 2017 15:44:34 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit possible to set?
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>,
	<alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
Message-ID: <50130.1512420274@turing-police.cc.vt.edu>

On Mon, 04 Dec 2017 17:08:19 +0000, "Simon Thompson (IT Research Support)" said:
> Have you looked at using filesets instead an using fileset quotas to achieve this?

Note that fileset quotas aren't able to represent "No Storage Allowed" either....
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/c92bcc95/attachment.sig>

From Greg.Lehmann at csiro.au  Mon Dec  4 23:28:05 2017
From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au)
Date: Mon, 4 Dec 2017 23:28:05 +0000
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712041036340.2395@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
	<B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>
	<alpine.LFD.2.20.1712041036340.2395@laptop>
Message-ID: <e65b855db0924e62abed07a24be62251@exch1-cdc.nexus.csiro.au>

We run GPFS client SW on SLES 12 SP2 which has a 4.4 kernel. It is only at 4.2.3-1 at present.

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of z.han at imperial.ac.uk
Sent: Monday, 4 December 2017 8:38 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or above?

Thanks, Simon.

Only on clients (oh so-called clients!) ;)


On Mon, 4 Dec 2017, Simon Thompson (IT Research Support) wrote:
> Date: Mon, 4 Dec 2017 10:29:27 +0000
> From: "Simon Thompson (IT Research Support)" <S.J.Thompson at bham.ac.uk>
> Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
>     above?
> 
> The FAQ at:
>
> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux
>
> Lists support with (e.g. Ubutu 16.04.2) with kernel 4.4.0-62, so likely it would work with a build your own kernel, but that doesn?t mean it is **supported**
>
> Simon
>
> On 04/12/2017, 09:52, "gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk" <gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk> wrote:
>
>    Hi All,
>
>    Any one is using a Linux kernel 3.12.x or above
>    to run gpfs 4.2.3-4.2? I mean you've compiled
>    your own kernel without paying for a professional
>    service.
>
>    We're stuck by CentOS/RHEL's distributed kernel
>    as the PCI passthrough is required for VMs. Your
>    comments or suggestions are much appreciated.
>
>    Kind regards,
>
>    Zong-Pei
>
>    _______________________________________________
>    gpfsug-discuss mailing list
>    gpfsug-discuss at spectrumscale.org
>    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From r.sobey at imperial.ac.uk  Tue Dec  5 10:38:37 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 5 Dec 2017 10:38:37 +0000
Subject: [gpfsug-discuss] Hidden fileset
Message-ID: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Quick question: is there a way to make a fileset hidden so it doesn't appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

I want to migrate a load of files in but I don't want the customer to access them prematurely.

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/ab0283f4/attachment.htm>

From robert.horton at icr.ac.uk  Tue Dec  5 10:47:51 2017
From: robert.horton at icr.ac.uk (Robert Horton)
Date: Tue, 5 Dec 2017 10:47:51 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <1512470871.4350.27.camel@icr.ac.uk>

On Tue, 2017-12-05 at 10:38 +0000, Sobey, Richard A wrote:
Quick question: is there a way to make a fileset hidden so it doesn?t appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

Can you link it somewhere obscure, put the data in, then re-link where you actually want it?

Rob

--

Robert Horton | Research Data Storage Lead
The Institute of Cancer Research | 237 Fulham Road | London | SW3 6JB
T +44 (0)20 7153 5350 | E robert.horton at icr.ac.uk<mailto:dugan.witherick at icr.ac.uk> | W www.icr.ac.uk | Twitter @ICR_London
Facebook: www.facebook.com/theinstituteofcancerresearch

Making the discoveries that defeat cancer

The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.

This e-mail message is confidential and for use by the addressee only.  If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/d2edfd94/attachment.htm>

From jonathan.buzzard at strath.ac.uk  Tue Dec  5 10:56:19 2017
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Tue, 05 Dec 2017 10:56:19 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <1512470871.4350.27.camel@icr.ac.uk>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<1512470871.4350.27.camel@icr.ac.uk>
Message-ID: <1512471379.18554.162.camel@strath.ac.uk>

On Tue, 2017-12-05 at 10:47 +0000, Robert Horton wrote:
> On Tue, 2017-12-05 at 10:38 +0000, Sobey, Richard A wrote:
> > Quick question: is there a way to make a fileset hidden so it
> > doesn?t appear when browsing the filesystem through a CES SMB
> > export? Could I do something clever with the directory mask?
> 
> Can you link it somewhere obscure, put the data in, then re-link
> where you actually want it?
> 

Would not setting the DOS hidden attribute on the directory do the
trick? That's assuming the CES SMB is doing everything correctly, no
experience with that but if it is that will do the trick.

Obviously if a user has Explorer set to view hidden files and folders
they will see it.

The other option would be to do something like

chown root:root /path/to/fileset
chmod 700 /path/to/fileset

Then from memory Samba won't show it because you don't have permissions
to access it.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG



From chair at spectrumscale.org  Tue Dec  5 10:58:33 2017
From: chair at spectrumscale.org (Simon Thompson)
Date: Tue, 05 Dec 2017 10:58:33 +0000
Subject: [gpfsug-discuss] SSUG @ CIUK next week [Tuesday 12th December 2017]
Message-ID: <00CFF55D-4D04-4F8D-90D9-F26AF7DA3B1D@spectrumscale.org>

Hi All,

 

Just a reminder that the last #SSUG of 2017 is happening next week as part of Computing Insights UK (CIUK) in Manchester. The agenda for the session is:

 

2pm Introductions and welcome, Simon Thompson & Claire O?Toole (SSUG Committee) 
2:10pm Spectrum Scale 5.0, Daniel Kidger (IBM) 
2:40pm Deep Dive: Blueprints, Ulf Troppens (IBM) 
3:40pm User Talk: Institute of Cancer Research, Robert Horton (ICR) 
4:00pm User Talk: Red Bull Technology, John Brookes (Red Bull Technology) 
4:20pm Questions and discussion 
4:50pm Close

 

To attend, you must be registered as a delegate of CIUK:

https://eventbooking.stfc.ac.uk/news-events/ciuk-2017

 

Simon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/22096afa/attachment.htm>

From S.J.Thompson at bham.ac.uk  Tue Dec  5 11:04:40 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Tue, 5 Dec 2017 11:04:40 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A3A3@EX13.adf.bham.ac.uk>

Don't give them permissions on the fileset and use the smb option:

"hideunreadable=yes"

https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htmhttps://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htm

This enables access based enumeration.

Simon
________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sobey, Richard A [r.sobey at imperial.ac.uk]
Sent: 05 December 2017 10:38
To: 'gpfsug-discuss at spectrumscale.org'
Subject: ?spam? [gpfsug-discuss] Hidden fileset

Quick question: is there a way to make a fileset hidden so it doesn?t appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

I want to migrate a load of files in but I don?t want the customer to access them prematurely.

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/4ae420b2/attachment.htm>

From r.sobey at imperial.ac.uk  Tue Dec  5 11:14:03 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 5 Dec 2017 11:14:03 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A3A3@EX13.adf.bham.ac.uk>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A3A3@EX13.adf.bham.ac.uk>
Message-ID: <DB6PR0601MB2101298220A37AA14C2DA71EDF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Holy.. why did I not know about that option ?

Will look into that.

And Robert ? yes linking it somewhere else is my current option but it just makes TSM back it all up again when it gets linked to the correct path. Next on the agenda is how to exclude certain directories from the backup.

Thanks
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support)
Sent: 05 December 2017 11:05
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Hidden fileset

Don't give them permissions on the fileset and use the smb option:

"hideunreadable=yes"

https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htmhttps://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htm

This enables access based enumeration.

Simon
________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sobey, Richard A [r.sobey at imperial.ac.uk]
Sent: 05 December 2017 10:38
To: 'gpfsug-discuss at spectrumscale.org'
Subject: ?spam? [gpfsug-discuss] Hidden fileset
Quick question: is there a way to make a fileset hidden so it doesn?t appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

I want to migrate a load of files in but I don?t want the customer to access them prematurely.

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/135ccbd2/attachment.htm>

From andreas.mattsson at maxiv.lu.se  Tue Dec  5 13:15:38 2017
From: andreas.mattsson at maxiv.lu.se (Andreas Mattsson)
Date: Tue, 5 Dec 2017 13:15:38 +0000
Subject: [gpfsug-discuss] Infiniband connection rejected,
	ibv_create_qp err 13
Message-ID: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>

Hi.

Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message "ibv_create_qp err 13"?
I'm having issues with this on a IBM ESS system.

The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to.
The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs.
MPI-jobs on the client nodes can communicate over the infiniband fabric without issues.
Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue.
When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted interfaces.

I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this.

Any help or suggestions is appreciated.
Regards,
Andreas Mattsson
_____________________________________________
[cid:part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer

MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/0e987953/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5610 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/0e987953/attachment.png>

From aaron.s.knister at nasa.gov  Tue Dec  5 13:23:43 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Tue, 5 Dec 2017 13:23:43 +0000
Subject: [gpfsug-discuss] Infiniband connection rejected,
 ibv_create_qp err 13
In-Reply-To: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>
References: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>
Message-ID: <957FBC22-DDAF-4BEF-95DA-1DB7B9AB2CCA@nasa.gov>



Looks like 13 is EPERM which means apparently permissions didn?t exist to create the QP of the desired type which is odd since mmfsd runs as root. Is there any remote chance SELinux is enabled (e.g. sestatus)? Although I?d think mmfsd would run unconfined in the default policy, but maybe it didn?t transition correctly.

On December 5, 2017 at 08:16:49 EST, Andreas Mattsson <andreas.mattsson at maxiv.lu.se> wrote:

Hi.



Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message ?ibv_create_qp err 13??

I?m having issues with this on a IBM ESS system.



The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to.

The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs.

MPI-jobs on the client nodes can communicate over the infiniband fabric without issues.

Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue.

When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted interfaces.



I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this.



Any help or suggestions is appreciated.

Regards,

Andreas Mattsson

_____________________________________________

[mid:d8d07f7e01ec4fcca5ae124f40c2d457 at maxiv.lu.se/part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer



MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/1d5fad85/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5610 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/1d5fad85/attachment.png>

From andreas.mattsson at maxiv.lu.se  Tue Dec  5 14:22:11 2017
From: andreas.mattsson at maxiv.lu.se (Andreas Mattsson)
Date: Tue, 5 Dec 2017 14:22:11 +0000
Subject: [gpfsug-discuss] Infiniband connection rejected,
 ibv_create_qp err 13
In-Reply-To: <957FBC22-DDAF-4BEF-95DA-1DB7B9AB2CCA@nasa.gov>
References: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>
	<957FBC22-DDAF-4BEF-95DA-1DB7B9AB2CCA@nasa.gov>
Message-ID: <869901bdbfd347969d21d619204a9b87@maxiv.lu.se>

Thanks!
I?ve double checked the selinux state, and it is disabled on both all the ESS nodes and on all the client nodes.
mmfsd is running as root on all nodes as well.
It seem a bit strange that this would be an issue of permissions though. I forgot to state this in my original question, but the issue comes and goes, and can affect some clients while not affecting others at the same time, and which clients are affected at any given time changes as well.
Just a thought; How does infiniband queue pairs react to time skew between nodes?

For future reference, where did you find the specification of ibv_create_qp error 13?
I must have been looking in all the wrong places, because I?ve been unable to find the meaning of this error.

Regards,
Andreas
_____________________________________________
[cid:part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer

MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>

Fr?n: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] F?r Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
Skickat: den 5 december 2017 14:24
Till: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
?mne: Re: [gpfsug-discuss] Infiniband connection rejected, ibv_create_qp err 13



Looks like 13 is EPERM which means apparently permissions didn?t exist to create the QP of the desired type which is odd since mmfsd runs as root. Is there any remote chance SELinux is enabled (e.g. sestatus)? Although I?d think mmfsd would run unconfined in the default policy, but maybe it didn?t transition correctly.
On December 5, 2017 at 08:16:49 EST, Andreas Mattsson <andreas.mattsson at maxiv.lu.se<mailto:andreas.mattsson at maxiv.lu.se>> wrote:

Hi.



Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message ?ibv_create_qp err 13??

I?m having issues with this on a IBM ESS system.



The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to.

The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs.

MPI-jobs on the client nodes can communicate over the infiniband fabric without issues.

Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue.

When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted interfaces.



I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this.



Any help or suggestions is appreciated.

Regards,

Andreas Mattsson

_____________________________________________

[mid:d8d07f7e01ec4fcca5ae124f40c2d457 at maxiv.lu.se/part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer



MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/9909e177/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5610 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/9909e177/attachment.png>

From r.sobey at imperial.ac.uk  Tue Dec  5 14:40:33 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 5 Dec 2017 14:40:33 +0000
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <DB6PR0601MB210125AFF8783FFFAB2BF3E4DF3E0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>,
	<OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>
	<DB6PR0601MB210125AFF8783FFFAB2BF3E4DF3E0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <DB6PR0601MB210100429C099C1A2BFBD016DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Thanks all, managed to rerun the policy with the additional list options (kb_allocated etc) and verified that indeed there are at least 60000 files with the same name that are 0 bytes, and more besides.

Now my paranoia is setting in that somehow these files are all corrupted ?

Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 02 December 2017 18:53
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Policy rule to list files based on access time

Thank you all for your replies. I will take a look at them on Monday. And reply individually where I need to..
Richard
Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Marc A Kaplan <makaplan at us.ibm.com>
Sent: Saturday, December 2, 2017 6:03:25 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Policy rule to list files based on access time

You might want to look at FILE_SIZE.  KB_ALLOCATED will be 0 if the file data fits into the inode.

You might also want to use SIZE(FILE_SIZE) in the policy LIST rule, this will cause the KB_HIT and KB_CHOSEN numbers to be the sum of FILE_SIZEs instead of the default SIZE(KB_ALLOCATED).

--marc of GPFS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/9593f4f0/attachment.htm>

From Mark.Bush at siriuscom.com  Tue Dec  5 22:43:35 2017
From: Mark.Bush at siriuscom.com (Mark Bush)
Date: Tue, 5 Dec 2017 22:43:35 +0000
Subject: [gpfsug-discuss] CES and local workstations
Message-ID: <DM5PR05MB35774E1C84500C5EBAA9A5E0E43D0@DM5PR05MB3577.namprd05.prod.outlook.com>

I have a windows 10 machine that is part of my local domain.  I have a separate SpecScale test cluster that has local (not part of my AD domain) ldap and CES (NFS/SMB) running.  I cannot get my local workstation to connect to a SMB share at all.  When I get the logon prompt I'm using IBM NAS\Administrator and the password for that user but it never lets me in.  Is this just defaulting to regular linux SAMBA authentication mechanisms at this point?  Of is the SAMBA code in the gpfs packages unique somehow in this?  I've confirmed that my mmuserauth service appears correct and that ldap is working properly. Any clues?

Frustrated.



Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/72015aba/attachment.htm>

From daniel.kidger at uk.ibm.com  Tue Dec  5 22:44:31 2017
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Tue, 5 Dec 2017 22:44:31 +0000
Subject: [gpfsug-discuss] mmauth/mmremotecluster wonkyness?
In-Reply-To: <OF7BE5A1DA.D530374F-ON002581E8.005B1ECF@LocalDomain>
Message-ID: <OFE0EA91FB.30381E41-ON002581ED.007CED24-1512513871966@notes.na.collabserv.com>

If you think about it, if a CES node opens a file for an NFS client and then a client in a remote MC mounted cluster opens the same file then that MC client node needs to talk to that CES node to negotiate the lock.

Daniel


 

 
 	
Dr Daniel Kidger 
IBM Technical Sales Specialist
Software Defined Solution Sales

+ 44-(0)7818 522 266 
daniel.kidger at uk.ibm.com

> On 30 Nov 2017, at 17:35, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:
> 
> It?s my understanding and experience that all member nodes of two clusters that are multi-clustered must be able to (and will eventually given enough time/activity) make connections to any and all nodes in both clusters. Even if you don?t designate the 2 protocol nodes as contact nodes I would expect to see connections from remote clusters to the protocol nodes just because of the nature of the beast. If you don?t want remote nodes to make connections to the protocol nodes then I believe you would need to put the protocol nodes in their own cluster. CES/CNFS hasn?t always supported this but I think it is now supported, at least with NFS. 
> 
> 
> 
> 
> 
>> On November 30, 2017 at 11:28:03 EST, valdis.kletnieks at vt.edu wrote:
>> We have a 10-node cluster running gpfs 4.2.2.3, where 8 nodes are GPFS contact 
>> nodes for 2 filesystems, and 2 are protocol nodes doingNFS exports of the 
>> filesystems. 
>> 
>> But we see some nodes in remote clusters trying to GPFS connect to 
>> the 2 protocol nodes anyhow. 
>> 
>> My reading of the manpages is that the remote cluster is responsible 
>> for setting '-n contactNodes' when they do the 'mmremotecluster add', 
>> and there's no way to sanity check or enforce that at the local end, and 
>> fail/flag connections to unintended non-contact nodes if the remote 
>> admin forgets/botches the -n. 
>> 
>> Is that actually correct? If so, is it time for an RFE? 
>> _______________________________________________ 
>> gpfsug-discuss mailing list 
>> gpfsug-discuss at spectrumscale.org 
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/da71f602/attachment.htm>

From z.han at imperial.ac.uk  Wed Dec  6 14:04:00 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Wed, 6 Dec 2017 14:04:00 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712040941230.2395@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
Message-ID: <alpine.LFD.2.20.1712061358300.10433@laptop>

I'm pleased to report that kernel 4.4.103 has been
successfully compiled and works with gpfs 4.2.3-4.2,
after a few failed attempts for other 3/4 kernels!

The only minor issue is the iDRAC virtual console
stops working, but fortunately the serial connection
is fine. Any expert advice will be warmly received.

Kind regards,

Zong-Pei


From z.han at imperial.ac.uk  Wed Dec  6 14:05:46 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Wed, 6 Dec 2017 14:05:46 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712061358300.10433@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
	<alpine.LFD.2.20.1712061358300.10433@laptop>
Message-ID: <alpine.LFD.2.20.1712061404430.10433@laptop>

PS

On CentOS Linux release 7.4.1708 (Core)


On Wed, 6 Dec 2017, z.han at imperial.ac.uk wrote:
> Date: Wed, 6 Dec 2017 14:04:00 +0000
> From: z.han at imperial.ac.uk
> Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
>     above?
> 
> I'm pleased to report that kernel 4.4.103 has been
> successfully compiled and works with gpfs 4.2.3-4.2,
> after a few failed attempts for other 3/4 kernels!
>
> The only minor issue is the iDRAC virtual console
> stops working, but fortunately the serial connection
> is fine. Any expert advice will be warmly received.
>
> Kind regards,
>
> Zong-Pei
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>


From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Dec  6 21:56:33 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 6 Dec 2017 21:56:33 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
Message-ID: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>

Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/886bd1a4/attachment.htm>

From jdkwedar at gmail.com  Wed Dec  6 22:10:11 2017
From: jdkwedar at gmail.com (Joshua Kwedar (froz1))
Date: Wed, 6 Dec 2017 17:10:11 -0500
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
Message-ID: <2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)

chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]

Josh K

> On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu> wrote:
> 
> Hi All,
> 
> So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(
> 
> Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.
> 
> Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)
> 
> Thanks?
> 
> ?
> Kevin Buterbaugh - Senior System Administrator
> Vanderbilt University - Advanced Computing Center for Research and Education
> Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/c23c3d52/attachment.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Dec  6 22:13:46 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 6 Dec 2017 22:13:46 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
Message-ID: <26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID  {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]


Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:

Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/28c88bcc/attachment.htm>

From jdkwedar at gmail.com  Wed Dec  6 22:32:18 2017
From: jdkwedar at gmail.com (Joshua Kwedar)
Date: Wed, 06 Dec 2017 17:32:18 -0500
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
Message-ID: <D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>

Hmm.. odd.? 

 

Here?s what the lsuser output should look like.? 

 

# /usr/lpp/mmfs/gui/cli/lsuser

Name? Long name Password status Group names???????????????? Failed login attempts

admin?????????? active????????? Administrator,SecurityAdmin 0

EFSSG1000I The command completed successfully.

 

Can you try something like?

 

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin

 

 

 

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

 

All, 

 

Sorry - should?ve mentioned that:

 

/root

root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231

EFSSG0001C Cannot validate option: login

/root

root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y

lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:

/root

root at testnsd1# 

 

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

 

Kevin



On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com> wrote:

 

The GUI password can be changed via command line using chuser.

 

/usr/lpp/mmfs/gui/cli/chuser



Usage is as follows (where userID = admin)



chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]


Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu> wrote:

Hi All, 

 

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

 

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

 

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

 

Thanks?

 

?

Kevin Buterbaugh - Senior System Administrator

Vanderbilt University - Advanced Computing Center for Research and Education

Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633

 

 

 

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

 

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/aec01adf/attachment.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Dec  6 22:41:12 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 6 Dec 2017 22:41:12 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
Message-ID: <D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin

On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin


On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser


Usage is as follows (where userID = admin)



chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/531a9a8d/attachment.htm>

From r.sobey at imperial.ac.uk  Thu Dec  7 11:57:07 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 7 Dec 2017 11:57:07 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>,
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
Message-ID: <DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin

On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin


On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser


Usage is as follows (where userID = admin)



chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171207/c3bdf293/attachment.htm>

From r.sobey at imperial.ac.uk  Thu Dec  7 12:10:52 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 7 Dec 2017 12:10:52 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>,
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
	<DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Sorry I need to learn to read... didn't see the "object 'Administrator' does not exist" error.

That said, my workaround for the problem of forgetting the password was to create a new "admin2" user and use that to reset the password on admin itself.

[root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin
EFSSG0019I The user admin2 has been successfully created.
EFSSG1000I The command completed successfully.


Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 07 December 2017 11:57
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.
Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I'll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it's subsequently been updated to GPFS 4.2.3.5 ... not sure that's relevant but wanted to mention it just in case.

Thanks!

Kevin


On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here's what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like...

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should've mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it's like the login doesn't exist ... but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I've forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM's, re-installing, and starting over from scratch?  If that's what I have to do, it's no big deal as this is just our little 6-node test cluster, but I thought I'd ask before going down that route.

Oh, and if someone has a way to accomplish this that they'd rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won't tell if you won't tell (and hopefully Michael Flynn won't tell either!).... ;-)

Thanks...

-
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171207/9e93a421/attachment.htm>

From bipcuds at gmail.com  Thu Dec  7 22:48:49 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Thu, 7 Dec 2017 17:48:49 -0500
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <CAAxuGpHj+4z6cLff69Z25CqfUaTWFjoV9vNGWhHSEUqWgWOZ9w@mail.gmail.com>

Hi All,

In working with per-fileset quotas (not user/group/fileset quotas at the
filesystem level), I have the follwing issues/questions.

1.) Setting grace periods. I notice that some of the group quotas in a
specific fileset have a grace period (or remaining grace period) of X days,
while others report "none":

# mmrepquota -g --block-size G hpc-fs:fileset01
                         Block Limits
|                     File Limits
Name       fileset    type             GB      quota      limit
in_doubt    grace |    files   quota    limit in_doubt    grace
groupa      fileset01 GRP            2257          2          2
0   4 days |      143     100      100        0   4 days
root       fileset01 GRP             710          0          0
0     none |    15578       0        0        0     none
groupb   fileset01 GRP            2106        400        400          0   4
days |        1 1048576  1048576        0     none
...

How can I set a grace period of "none" on group quotas? mmsetquota does not
appear (from the man pages) to provide any way to set grace periods for
per-fileset quotas:

mmsetquota Device ??grace {user | group | fileset}
           {[??block GracePeriod] [??files GracePeriod]}

How can I set to "none" or "0days"? (i.e. no grace period given if over
quota). Or, for that matter, set grace periods for any duration at all?


2.) How to remove any explicit quota settings for (not just deactivating
default quota settings) at the per-fileset level. The mmdefquotaoff docs
seem to suggest that the ?-d? option will not remove explicit per-fileset
quota settings if they are non-zero (so really, what use is the -d option
then?)

Many Thanks!
  Keith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171207/b790fd92/attachment.htm>

From Robert.Oesterlin at nuance.com  Fri Dec  8 12:39:58 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Fri, 8 Dec 2017 12:39:58 +0000
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <D7F39BE9-B69E-4AB4-8E6E-7B8FA022EE9D@nuance.com>

1) One way you can set the grace period to none by setting the soft and hard quotas to the same value.

2) By removing any quota setting, do you mean unlimited quota? Which is just setting it (quota) to zero. (meaning unlimited)

Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Keith Ball <bipcuds at gmail.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Thursday, December 7, 2017 at 4:49 PM
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] [gpfsug-discuss] Working with per-fileset quotas

How can I set a grace period of "none" on group quotas? mmsetquota does not appear (from the man pages) to provide any way to set grace periods for per-fileset quotas:

mmsetquota Device ??grace {user | group | fileset}
           {[??block GracePeriod] [??files GracePeriod]}
How can I set to "none" or "0days"? (i.e. no grace period given if over quota). Or, for that matter, set grace periods for any duration at all?


2.) How to remove any explicit quota settings for (not just deactivating default quota settings) at the per-fileset level. The mmdefquotaoff docs seem to suggest that the ?-d? option will not remove explicit per-fileset quota settings if they are non-zero (so really, what use is the -d option then?)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/c006d850/attachment.htm>

From truongv at us.ibm.com  Fri Dec  8 15:41:04 2017
From: truongv at us.ibm.com (Truong Vu)
Date: Fri, 8 Dec 2017 10:41:04 -0500
Subject: [gpfsug-discuss] Working with per-fileset quotas
In-Reply-To: <OF2C4BFBB9.F213F6C2-ON002581F0.0041F153@LocalDomain>
References: <OF2C4BFBB9.F213F6C2-ON002581F0.0041F153@LocalDomain>
Message-ID: <OF95F138D7.BB03BC26-ON852581F0.00560D00-852581F0.00562863@notes.na.collabserv.com>


1) That is correct.  The grace period can't be set for per-fileset
quota.  As you pointed out, you can only change the grace period for
user, group or fileset.

If you want a particular fileset to have no grace period, you can
set the hard limit to be the same as the soft limit.

When the grace column shows "none", this means the soft limit has
not been reached.  Once the soft limit is reached, the grace period
is start counting.

2) To remove explicit quota settings, you need to set the limit to 0.




From:	gpfsug-discuss-request at spectrumscale.org
To:	gpfsug-discuss at spectrumscale.org
Date:	12/08/2017 07:00 AM
Subject:	gpfsug-discuss Digest, Vol 71, Issue 19
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Send gpfsug-discuss mailing list submissions to
		 gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit

https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=5OySIaqfU0j1miWKKp6aydLjiGbE8z5pDz5JGveRRlQ&s=HKXwjIgPAsNTzNSL3-FrvHAXVyvZdzGYugbZgJ3FvMI&e=

or, via email, send a message with subject or body 'help' to
		 gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
		 gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Working with per-fileset quotas (Keith Ball)


----------------------------------------------------------------------

Message: 1
Date: Thu, 7 Dec 2017 17:48:49 -0500
From: Keith Ball <bipcuds at gmail.com>
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID:
		 <CAAxuGpHj
+4z6cLff69Z25CqfUaTWFjoV9vNGWhHSEUqWgWOZ9w at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi All,

In working with per-fileset quotas (not user/group/fileset quotas at the
filesystem level), I have the follwing issues/questions.

1.) Setting grace periods. I notice that some of the group quotas in a
specific fileset have a grace period (or remaining grace period) of X days,
while others report "none":

# mmrepquota -g --block-size G hpc-fs:fileset01
                         Block Limits
|                     File Limits
Name       fileset    type             GB      quota      limit
in_doubt    grace |    files   quota    limit in_doubt    grace
groupa      fileset01 GRP            2257          2          2
0   4 days |      143     100      100        0   4 days
root       fileset01 GRP             710          0          0
0     none |    15578       0        0        0     none
groupb   fileset01 GRP            2106        400        400          0   4
days |        1 1048576  1048576        0     none
...

How can I set a grace period of "none" on group quotas? mmsetquota does not
appear (from the man pages) to provide any way to set grace periods for
per-fileset quotas:

mmsetquota Device ??grace {user | group | fileset}
           {[??block GracePeriod] [??files GracePeriod]}

How can I set to "none" or "0days"? (i.e. no grace period given if over
quota). Or, for that matter, set grace periods for any duration at all?


2.) How to remove any explicit quota settings for (not just deactivating
default quota settings) at the per-fileset level. The mmdefquotaoff docs
seem to suggest that the ?-d? option will not remove explicit per-fileset
quota settings if they are non-zero (so really, what use is the -d option
then?)

Many Thanks!
  Keith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20171207_b790fd92_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=5OySIaqfU0j1miWKKp6aydLjiGbE8z5pDz5JGveRRlQ&s=-DNcYGwFlUrOQZsZ9yQAYBdVv46u1xxeA4wJ2-VxQ_A&e=
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=5OySIaqfU0j1miWKKp6aydLjiGbE8z5pDz5JGveRRlQ&s=HKXwjIgPAsNTzNSL3-FrvHAXVyvZdzGYugbZgJ3FvMI&e=



End of gpfsug-discuss Digest, Vol 71, Issue 19
**********************************************



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/790f55b2/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/790f55b2/attachment.gif>

From eric.wonderley at vt.edu  Fri Dec  8 16:10:16 2017
From: eric.wonderley at vt.edu (J. Eric Wonderley)
Date: Fri, 8 Dec 2017 11:10:16 -0500
Subject: [gpfsug-discuss] mm'add|del'node with ccr enabled
Message-ID: <CABOSGQf2JTQdp3E+8dkFXY07x5u9SnePh0yhAgyD6YMty2WhHA@mail.gmail.com>

Hello:

If I recall correctly this does not work...correct?  I think the last time
I attempted this was gpfs version <=4.1.  I think I attempted to add a
quorum node.

The process was that I remember doing was mmshutdown -a, mmchcluster
--ccr-disable, mmaddnode yadayada, mmchcluster --ccr-enable, mmstartup.

I think with ccr disabled mmaddnode can be run with gpfs up.  We would like
to run with ccr enabled but it does make adding/removing nodes unpleasant.

Would this be required of a non-quorum node?

Any changes concerning this with gpfs version >=4.2?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/850462fc/attachment.htm>

From Robert.Oesterlin at nuance.com  Fri Dec  8 16:17:13 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Fri, 8 Dec 2017 16:17:13 +0000
Subject: [gpfsug-discuss] mm'add|del'node with ccr enabled
Message-ID: <A1F99FA0-3794-4989-A1A2-D81A038765B5@nuance.com>

Yes, there were issues with this in pre 4.2? versions of GPFS. My clusters are at 4.2.2 with CCR (working my way to 4.2.3-5) and I?ve been using this process on my quorum nodes when I need to update them:

mmchnode ?quorum -N new-node (designate a new quorum node, going from 3 to 4)
mmchnode ?nonquorum -N old-node (remove existing node to be updated)

I can then shutdown ?old-node? and then when I bring it back up, I reverse the process. I?ve had no issues using this process.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "J. Eric Wonderley" <eric.wonderley at vt.edu>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Friday, December 8, 2017 at 10:11 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] [gpfsug-discuss] mm'add|del'node with ccr enabled

Hello:

If I recall correctly this does not work...correct?  I think the last time I attempted this was gpfs version <=4.1.  I think I attempted to add a quorum node.

The process was that I remember doing was mmshutdown -a, mmchcluster --ccr-disable, mmaddnode yadayada, mmchcluster --ccr-enable, mmstartup.

I think with ccr disabled mmaddnode can be run with gpfs up.  We would like to run with ccr enabled but it does make adding/removing nodes unpleasant.

Would this be required of a non-quorum node?

Any changes concerning this with gpfs version >=4.2?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/da944b42/attachment.htm>

From TROPPENS at de.ibm.com  Sat Dec  9 00:11:27 2017
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Sat, 9 Dec 2017 01:11:27 +0100
Subject: [gpfsug-discuss] Genomics Blueprint - SSUG @ CIUK next week
 [Tuesday 12th December 2017]
In-Reply-To: <OF0DB03636.FFEC975E-ON002581ED.003C507F@LocalDomain>
References: <OF0DB03636.FFEC975E-ON002581ED.003C507F@LocalDomain>
Message-ID: <OF3EFB1F2F.FBD960FA-ONC12581F1.00006B84-C12581F1.00010C76@notes.na.collabserv.com>

We just published the first blueprint:

IBM Spectrum Scale Best Practices for Genomics Medicine Workloads
http://www.redbooks.ibm.com/abstracts/redp5479.html

This blueprint is optimized for genomics workloads but most recommendations
apply to other workloads.

Looking forward to receive first-hand feedback at CIUK.


--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294




From:	Simon Thompson <chair at spectrumscale.org>
To:	"gpfsug-discuss at spectrumscale.org"
            <gpfsug-discuss at spectrumscale.org>
Date:	05.12.2017 11:58
Subject:	[gpfsug-discuss] SSUG @ CIUK next week [Tuesday 12th December
            2017]
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hi All,

Just a reminder that the last #SSUG of 2017 is happening next week as part
of Computing Insights UK (CIUK) in Manchester. The agenda for the session
is:

2pm Introductions and welcome, Simon Thompson & Claire O?Toole (SSUG
Committee)
2:10pm Spectrum Scale 5.0, Daniel Kidger (IBM)
2:40pm Deep Dive: Blueprints, Ulf Troppens (IBM)
3:40pm User Talk: Institute of Cancer Research, Robert Horton (ICR)
4:00pm User Talk: Red Bull Technology, John Brookes (Red Bull Technology)
4:20pm Questions and discussion
4:50pm Close

To attend, you must be registered as a delegate of CIUK:
https://eventbooking.stfc.ac.uk/news-events/ciuk-2017

Simon_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/9d16277d/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/9d16277d/attachment.gif>

From Renar.Grunenberg at huk-coburg.de  Sat Dec  9 10:58:52 2017
From: Renar.Grunenberg at huk-coburg.de (Grunenberg, Renar)
Date: Sat, 9 Dec 2017 10:58:52 +0000
Subject: [gpfsug-discuss] mm'add|del'node with ccr enabled
In-Reply-To: <CABOSGQf2JTQdp3E+8dkFXY07x5u9SnePh0yhAgyD6YMty2WhHA@mail.gmail.com>
References: <CABOSGQf2JTQdp3E+8dkFXY07x5u9SnePh0yhAgyD6YMty2WhHA@mail.gmail.com>
Message-ID: <f926f5c352f146ceb49f5e14fb1e76fd@SMXRF105.msg.hukrf.de>

Hallo Eric,
our experiences are add and delete new/old nodes is working only if this node is no quorum node in an ccr cluster, no problem. There are no mmshutdown steps necessary. We are on 4.2.3.6. I think this is already available since >4.2. If you want to add a new quorum node, than you must put this node first as a client and after that you can change this node to a quorum node.


Renar Grunenberg
Abteilung Informatik ? Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon:        09561 96-44110
Telefax:        09561 96-44104
E-Mail: Renar.Grunenberg at huk-coburg.de
Internet:       www.huk.de
________________________________
HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Her?y, Dr. J?rg Rheinl?nder (stv.), Sarah R?ssler, Daniel Thomas (stv.).
________________________________
Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.

This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this information in error) please notify the
sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.
________________________________
Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von J. Eric Wonderley
Gesendet: Freitag, 8. Dezember 2017 17:10
An: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Betreff: [gpfsug-discuss] mm'add|del'node with ccr enabled

Hello:

If I recall correctly this does not work...correct?  I think the last time I attempted this was gpfs version <=4.1.  I think I attempted to add a quorum node.

The process was that I remember doing was mmshutdown -a, mmchcluster --ccr-disable, mmaddnode yadayada, mmchcluster --ccr-enable, mmstartup.

I think with ccr disabled mmaddnode can be run with gpfs up.  We would like to run with ccr enabled but it does make adding/removing nodes unpleasant.

Would this be required of a non-quorum node?

Any changes concerning this with gpfs version >=4.2?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/16c21559/attachment.htm>

From bipcuds at gmail.com  Sat Dec  9 21:49:54 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Sat, 9 Dec 2017 16:49:54 -0500
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <CAAxuGpHH6e2K=mBTZQ=tA7wc-UMYmg7_8F9hvrh8Kn7Gh78_rA@mail.gmail.com>

I meant more specifically, remove any indication that quotas have been set,
for anyone (or everyone). That way, I could have a script to clean out old
quota definitions, then set both default and explicit quotas, without any
old settings lurking around. Is there a way to get rid of (e.g. zero out)
all existing explicit PER-FILESET quota definitions, and start over
(without having to look at mmrepquota and zero them out one by one)?


So I see your point about soft=lard limit => no grace period. I guess
what's odd is that I see any indication that the grace period is not
"none"; what grace period would be assigned to a per-fileset quota if I
cannot even view or set it?

Thanks,
   Keith


From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Working with per-fileset quotas
>
> 2) By removing any quota setting, do you mean unlimited quota? Which is
> just setting it (quota) to zero. (meaning unlimited)
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 8 Dec 2017 10:41:04 -0500
> From: "Truong Vu" <truongv at us.ibm.com>
> To: gpfsug-discuss at spectrumscale.org
> Subject: Re: [gpfsug-discuss] Working with per-fileset quotas
> Message-ID:
>         <OF95F138D7.BB03BC26-ON852581F0.00560D00-852581F0.00562863@
> notes.na.collabserv.com>
>
> Content-Type: text/plain; charset="us-ascii"
>
>
> 1) That is correct.  The grace period can't be set for per-fileset
> quota.  As you pointed out, you can only change the grace period for
> user, group or fileset.
>
> If you want a particular fileset to have no grace period, you can
> set the hard limit to be the same as the soft limit.
>
> When the grace column shows "none", this means the soft limit has
> not been reached.  Once the soft limit is reached, the grace period
> is start counting.
>
> 2) To remove explicit quota settings, you need to set the limit to 0.
>
>
>
>
> From:   gpfsug-discuss-request at spectrumscale.org
> To:     gpfsug-discuss at spectrumscale.org
> Date:   12/08/2017 07:00 AM
> Subject:        gpfsug-discuss Digest, Vol 71, Issue 19
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
>
>
>    1. Working with per-fileset quotas (Keith Ball)
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 7 Dec 2017 17:48:49 -0500
> From: Keith Ball <bipcuds at gmail.com>
> To: gpfsug-discuss at spectrumscale.org
> Subject: [gpfsug-discuss] Working with per-fileset quotas
> Message-ID:
>                  <CAAxuGpHj
> +4z6cLff69Z25CqfUaTWFjoV9vNGWhHSEUqWgWOZ9w at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi All,
>
> In working with per-fileset quotas (not user/group/fileset quotas at the
> filesystem level), I have the follwing issues/questions.
>
> 1.) Setting grace periods. I notice that some of the group quotas in a
> specific fileset have a grace period (or remaining grace period) of X days,
> while others report "none":
>
> # mmrepquota -g --block-size G hpc-fs:fileset01
>                          Block Limits
> |                     File Limits
> Name       fileset    type             GB      quota      limit
> in_doubt    grace |    files   quota    limit in_doubt    grace
> groupa      fileset01 GRP            2257          2          2
> 0   4 days |      143     100      100        0   4 days
> root       fileset01 GRP             710          0          0
> 0     none |    15578       0        0        0     none
> groupb   fileset01 GRP            2106        400        400          0   4
> days |        1 1048576  1048576        0     none
> ...
>
> How can I set a grace period of "none" on group quotas? mmsetquota does not
> appear (from the man pages) to provide any way to set grace periods for
> per-fileset quotas:
>
> mmsetquota Device ??grace {user | group | fileset}
>            {[??block GracePeriod] [??files GracePeriod]}
>
> How can I set to "none" or "0days"? (i.e. no grace period given if over
> quota). Or, for that matter, set grace periods for any duration at all?
>
>
> 2.) How to remove any explicit quota settings for (not just deactivating
> default quota settings) at the per-fileset level. The mmdefquotaoff docs
> seem to suggest that the ?-d? option will not remove explicit per-fileset
> quota settings if they are non-zero (so really, what use is the -d option
> then?)
>
> Many Thanks!
>   Keith
>
>

-- 
Keith D. Ball, PhD
RedLine Performance Solutions, LLC
web:  http://www.redlineperf.com/
email: kball at redlineperf.com <aqualkenbush at redlineperf.com>
cell: 540-557-7851 <(540)%20557-7851>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/203f7a33/attachment.htm>

From Robert.Oesterlin at nuance.com  Sun Dec 10 01:27:13 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Sun, 10 Dec 2017 01:27:13 +0000
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <C1EFE45D-20E7-4689-87A6-7C5B2CE2ED5F@nuance.com>

Looking at the default case in a new file system/fileset, if you did an ?mmrepquota? you?d see everyone have a default quota of zero. Meaning ? any time you set a user/fileset/group quota back to zero, you are removing any trace of the previous quota. On you your specific question - yes, one by one is it, I?m not aware of any other way to do it.

Hard quota set, no soft quota: no grace period
Hard Soft quota set and equal: no grace period
Hard quota set or unset, soft quota set: grace period

The default behavior is no grace period unless soft quota is set, maybe that?s why it?s no displayed?


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Keith Ball <bipcuds at gmail.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Saturday, December 9, 2017 at 3:50 PM
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Working with per-fileset quotas

I meant more specifically, remove any indication that quotas have been set, for anyone (or everyone). That way, I could have a script to clean out old quota definitions, then set both default and explicit quotas, without any old settings lurking around. Is there a way to get rid of (e.g. zero out) all existing explicit PER-FILESET quota definitions, and start over (without having to look at mmrepquota and zero them out one by one)?


So I see your point about soft=lard limit => no grace period. I guess what's odd is that I see any indication that the grace period is not "none"; what grace period would be assigned to a per-fileset quota if I cannot even view or set it?


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171210/5b19346b/attachment.htm>

From john.hearns at asml.com  Wed Dec 13 12:18:01 2017
From: john.hearns at asml.com (John Hearns)
Date: Wed, 13 Dec 2017 12:18:01 +0000
Subject: [gpfsug-discuss] Am I a gateway or not?
Message-ID: <HE1PR02MB14506B5498911ABC8CE08D4F88350@HE1PR02MB1450.eurprd02.prod.outlook.com>

Firstly, I hope everyone at the Manchester meeting is enjoying it and that the snow did not hamper too many travel plans.
I like Manchester - I shall tell the tale of blowing the 100Amp fuse in the University server room some other time.
(Yes - a fuse. Not a circuit breaker).

I am writing some monitoring scripts for AFM - actually to interface with Bright.
I would like the script to detect if the node which is running it is a Gateway node or has the AFM component.
If not, the script silently exits.
This is not a Callback script - I can write these and indeed have one for the AFM fileset demount event.

In order to determine which components are on a given node I could:

  1.  Run "mmhealth node show" and grep for AFM
  2.  Run mmlscluster and grep for $HOSTNAME

Both methods seem slightly inelegant. IS there a handy-dandy command I can run to see the components on this node?

John Hearns
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/22341ece/attachment.htm>

From scale at us.ibm.com  Wed Dec 13 14:45:52 2017
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Wed, 13 Dec 2017 09:45:52 -0500
Subject: [gpfsug-discuss] Am I a gateway or not?
In-Reply-To: <OF9C6D99B0.DF71D62B-ON002581F5.0043B3C0@LocalDomain>
References: <OF9C6D99B0.DF71D62B-ON002581F5.0043B3C0@LocalDomain>
Message-ID: <OF1EF7D0AE.0DD977EE-ON852581F5.00510567-852581F5.00511AAB@notes.na.collabserv.com>

Hi Abhishek,

Can you please help answer this query.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   John Hearns <john.hearns at asml.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   12/13/2017 07:19 AM
Subject:        [gpfsug-discuss] Am I a gateway or not?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Firstly, I hope everyone at the Manchester meeting is enjoying it and that 
the snow did not hamper too many travel plans.
I like Manchester ? I shall tell the tale of blowing the 100Amp fuse in 
the University server room some other time.
(Yes ? a fuse. Not a circuit breaker).
 
I am writing some monitoring scripts for AFM ? actually to interface with 
Bright.
I would like the script to detect if the node which is running it is a 
Gateway node or has the AFM component.
If not, the script silently exits.
This is not a Callback script ? I can write these and indeed have one for 
the AFM fileset demount event.
 
In order to determine which components are on a given node I could:
a.      Run ?mmhealth node show? and grep for AFM
b.      Run mmlscluster and grep for $HOSTNAME
 
Both methods seem slightly inelegant. IS there a handy-dandy command I can 
run to see the components on this node?
 
John Hearns
-- The information contained in this communication and any attachments is 
confidential and may be privileged, and is for the sole use of the 
intended recipient(s). Any unauthorized review, use, disclosure or 
distribution is prohibited. Unless explicitly stated otherwise in the body 
of this communication or the attachment thereto (if any), the information 
is provided on an AS-IS basis without any express or implied warranties or 
liabilities. To the extent you are relying on this information, you are 
doing so at your own risk. If you are not the intended recipient, please 
notify the sender immediately by replying to this message and destroy all 
copies of this message and any attachments. Neither the sender nor the 
company/group of companies he or she represents shall be liable for the 
proper and complete transmission of the information contained in this 
communication, or for any delay in its receipt. 
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=iZJyYX6s6-fBYfRNYStIrfVhph3NLzwXJNqTZEfKOyA&s=HoPYEIPpBLIFV-JCq69MS026ZLBRxqqvqNZYGH8UmlY&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/92f3f864/attachment.htm>

From abhisdav at in.ibm.com  Wed Dec 13 16:58:39 2017
From: abhisdav at in.ibm.com (Abhishek Dave)
Date: Wed, 13 Dec 2017 22:28:39 +0530
Subject: [gpfsug-discuss] Am I a gateway or not?
In-Reply-To: <OF1EF7D0AE.0DD977EE-ON852581F5.00510567-852581F5.00511998@LocalDomain>
References: <OF9C6D99B0.DF71D62B-ON002581F5.0043B3C0@LocalDomain>
	<OF1EF7D0AE.0DD977EE-ON852581F5.00510567-852581F5.00511998@LocalDomain>
Message-ID: <OF90FA1155.FA40F211-ON652581F5.005C72A9-652581F5.005D42EF@notes.na.collabserv.com>

Hi John,

I would recommend running 'mmlscluster' command on any cluster node to
check if any AFM Gateway Node(s) is available.

This will list hostname(s) which are running as AFM Gateway.

/usr/lpp/mmfs/bin/mmlscluster | grep -w gateway | awk '{print $4}'

e.g.

~bash# /usr/lpp/mmfs/bin/mmlscluster | grep -w gateway | awk '{print $4}'
node1
node2

Thanks,
Abhishek, Dave




From:	IBM Spectrum Scale/Poughkeepsie/IBM
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc:	Abhishek Dave/India/IBM at IBMIN, IBM Spectrum
            Scale/Poughkeepsie/IBM at IBMUS
Date:	12/13/2017 08:16 PM
Subject:	Re: [gpfsug-discuss] Am I a gateway or not?
Sent by:	Haizhu Liu


Hi Abhishek,

Can you please help answer this query.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------

If you feel that your question can benefit other users of  Spectrum Scale
(GPFS), then please post it to the public IBM developerWroks Forum at
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
.

If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact
1-800-237-5511 in the United States or your local IBM Service Center in
other countries.

The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.




From:	John Hearns <john.hearns at asml.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	12/13/2017 07:19 AM
Subject:	[gpfsug-discuss] Am I a gateway or not?
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Firstly, I hope everyone at the Manchester meeting is enjoying it and that
the snow did not hamper too many travel plans.
I like Manchester ? I shall tell the tale of blowing the 100Amp fuse in the
University server room some other time.
(Yes ? a fuse. Not a circuit breaker).

I am writing some monitoring scripts for AFM ? actually to interface with
Bright.
I would like the script to detect if the node which is running it is a
Gateway node or has the AFM component.
If not, the script silently exits.
This is not a Callback script ? I can write these and indeed have one for
the AFM fileset demount event.

In order to determine which components are on a given node I could:
   a.	Run ?mmhealth node show? and grep for AFM
   b.	Run mmlscluster and grep for $HOSTNAME

Both methods seem slightly inelegant. IS there a handy-dandy command I can
run to see the components on this node?

John Hearns


-- The information contained in this communication and any attachments is
confidential and may be privileged, and is for the sole use of the intended
recipient(s). Any unauthorized review, use, disclosure or distribution is
prohibited. Unless explicitly stated otherwise in the body of this
communication or the attachment thereto (if any), the information is
provided on an AS-IS basis without any express or implied warranties or
liabilities. To the extent you are relying on this information, you are
doing so at your own risk. If you are not the intended recipient, please
notify the sender immediately by replying to this message and destroy all
copies of this message and any attachments. Neither the sender nor the
company/group of companies he or she represents shall be liable for the
proper and complete transmission of the information contained in this
communication, or for any delay in its receipt.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=iZJyYX6s6-fBYfRNYStIrfVhph3NLzwXJNqTZEfKOyA&s=HoPYEIPpBLIFV-JCq69MS026ZLBRxqqvqNZYGH8UmlY&e=










-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/3b793bb7/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/3b793bb7/attachment.gif>

From chair at spectrumscale.org  Thu Dec 14 11:16:05 2017
From: chair at spectrumscale.org (Simon Thompson)
Date: Thu, 14 Dec 2017 11:16:05 +0000
Subject: [gpfsug-discuss] SSUG17 wraps
Message-ID: <F96F7F48-5592-45D9-9A1D-16B9D8F8D507@spectrumscale.org>

Well, our last event of 2017 took place this year at CIUK in Manchester on 12th December. Claire pointed out that it was exactly 7 years since the user group was first formed and we?ve grown significantly since then. There?s now hundreds of subscribers to the mailing list and regular events taking place across the globe.

 

For those interested in the presentations this year, all the slide decks we have or that have been approved are published on the user group website.

 

Thanks to all the speakers from both IBM and the user community for speaking at our events!

 

And finally, we?ve updated the look of our website and changed to something we hope will be a bit easier for use to manage. We?ve made some changes to the navigation to make things easier to find, so please do check out the website and let myself of Claire know directly if you have any comments on this.

 

We?re also interested in hearing from you if you?d like to write a feature or article for the user group website, perhaps comment on a scale feature you use, or something you?ve struggled with and fixed. Please get in touch if you?d like to do this.

 

We?re already planning for 2018 events and hope we?ll have a successful year next year.

 

Simon Thompson

UK Group Chair

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171214/53dbbc98/attachment.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Thu Dec 14 19:38:52 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Thu, 14 Dec 2017 19:38:52 +0000
Subject: [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5 upgrade
Message-ID: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>

Hi All,

         26 mmbackupDors-20171023.log
         26 mmbackupDors-20171024.log
         26 mmbackupDors-20171025.log
         26 mmbackupDors-20171026.log
    2922752 mmbackupDors-20171027.log
    1375555 mmbackupDors-20171028.log
      59328 mmbackupDors-20171029.log
    2748095 mmbackupDors-20171030.log
     124953 mmbackupDors-20171031.log

That?s ?wc -l? output ? and the difference in size occurred with the GPFS 4.2.3.5 upgrade.  I?m not technically ?responsible? for mmbackup here, so I?m not at all familiar with it.  However, we?ve asked a certain vendor (not IBM) about it and they don?t know either, so I don?t feel too awfully bad.

And we have looked at the man page and didn?t see any obvious options to decrease the verbosity.  We did not make any changes to the backup script itself, so the mmbackup invocation is the same.  Any ideas?  Thanks?

Kevin
?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171214/23f59fce/attachment.htm>

From scale at us.ibm.com  Fri Dec 15 14:32:25 2017
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Fri, 15 Dec 2017 09:32:25 -0500
Subject: [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5
	upgrade
In-Reply-To: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>
References: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>
Message-ID: <OFEF3AE524.5EDB2C57-ON852581F7.004FCC00-852581F7.004FDF6C@notes.na.collabserv.com>

Tru,

Can you please help with this query or forward to the right person. 
Thanks. 

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   12/14/2017 02:39 PM
Subject:        [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5 
upgrade
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi All, 

         26 mmbackupDors-20171023.log
         26 mmbackupDors-20171024.log
         26 mmbackupDors-20171025.log
         26 mmbackupDors-20171026.log
    2922752 mmbackupDors-20171027.log
    1375555 mmbackupDors-20171028.log
      59328 mmbackupDors-20171029.log
    2748095 mmbackupDors-20171030.log
     124953 mmbackupDors-20171031.log

That?s ?wc -l? output ? and the difference in size occurred with the GPFS 
4.2.3.5 upgrade.  I?m not technically ?responsible? for mmbackup here, so 
I?m not at all familiar with it.  However, we?ve asked a certain vendor 
(not IBM) about it and they don?t know either, so I don?t feel too awfully 
bad.

And we have looked at the man page and didn?t see any obvious options to 
decrease the verbosity.  We did not make any changes to the backup script 
itself, so the mmbackup invocation is the same.  Any ideas?  Thanks?

Kevin
?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and 
Education
Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=ZFlCTb2hD9WKda9YJ94wM4Kb-ZTkTg6yuH2H0DabbRs&s=4TRCVgkMd1Ly9WvaBPVWr4ciur_ZYZswzqrlFHl_6EI&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/9212a7f7/attachment.htm>

From makaplan at us.ibm.com  Fri Dec 15 15:14:40 2017
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Fri, 15 Dec 2017 10:14:40 -0500
Subject: [gpfsug-discuss] mmbackup log file size after GPFS
	4.2.3.5upgrade
In-Reply-To: <OFEF3AE524.5EDB2C57-ON852581F7.004FCC00-852581F7.004FDF6C@notes.na.collabserv.com>
References: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>
	<OFEF3AE524.5EDB2C57-ON852581F7.004FCC00-852581F7.004FDF6C@notes.na.collabserv.com>
Message-ID: <OF94C2F295.6B1FFCC4-ON852581F7.005351E8-852581F7.0053BE2A@notes.na.collabserv.com>

It's not clear that this is a problem or malfunction.
Customer should contact IBM support and be ready to transmit copies of the 
cited log files and other mmbackup command output (stdout and stderr 
messages) for analysis.
Also mmsnap output.



From:   "IBM Spectrum Scale" <scale at us.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc:     Truong Vu <truongv at us.ibm.com>
Date:   12/15/2017 09:32 AM
Subject:        Re: [gpfsug-discuss] mmbackup log file size after GPFS 
4.2.3.5 upgrade
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Tru,

Can you please help with this query or forward to the right person. 
Thanks.  

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        12/14/2017 02:39 PM
Subject:        [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5 
upgrade
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi All, 

         26 mmbackupDors-20171023.log
         26 mmbackupDors-20171024.log
         26 mmbackupDors-20171025.log
         26 mmbackupDors-20171026.log
    2922752 mmbackupDors-20171027.log
    1375555 mmbackupDors-20171028.log
      59328 mmbackupDors-20171029.log
    2748095 mmbackupDors-20171030.log
     124953 mmbackupDors-20171031.log

That?s ?wc -l? output ? and the difference in size occurred with the GPFS 
4.2.3.5 upgrade.  I?m not technically ?responsible? for mmbackup here, so 
I?m not at all familiar with it.  However, we?ve asked a certain vendor 
(not IBM) about it and they don?t know either, so I don?t feel too awfully 
bad.

And we have looked at the man page and didn?t see any obvious options to 
decrease the verbosity.  We did not make any changes to the backup script 
itself, so the mmbackup invocation is the same.  Any ideas?  Thanks?

Kevin
?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and 
Education
Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=ZFlCTb2hD9WKda9YJ94wM4Kb-ZTkTg6yuH2H0DabbRs&s=4TRCVgkMd1Ly9WvaBPVWr4ciur_ZYZswzqrlFHl_6EI&e=



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=n3w55C-SUpJhWZEaM249Jx7hx44Q8KsZi0-S2WaisJc&s=SFkl9xPj2c7TR7paHtU3Nby1vonHbBBt1Vmv7QdLH9Q&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/f4679d08/attachment.htm>

From Robert.Oesterlin at nuance.com  Fri Dec 15 18:40:28 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Fri, 15 Dec 2017 18:40:28 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0.0 Available on Passport
	Advantage
Message-ID: <985A9D83-4B5E-4234-9884-FFA5AE295063@nuance.com>

Scale 5.0 was released today and is available for download. Time to construct a test cluster!


Bob Oesterlin
Sr Principal Storage Engineer, Nuance
507-269-0413

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/b7a70d41/attachment.htm>

From aaron.s.knister at nasa.gov  Fri Dec 15 19:49:59 2017
From: aaron.s.knister at nasa.gov (Aaron Knister)
Date: Fri, 15 Dec 2017 14:49:59 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
Message-ID: <fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>

Thanks, Bill.

I still don't feel like I've got an clear answer from IBM and frankly
the core issue of a lack of migration tool was totally dodged.

Again in Sven's presentation from SSUG @ SC17
(http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf)
he mentions "It has a significant performance penalty for small files in
large block size filesystems" and the demonstrates that with several
mdtest runs (which show the effect with and without the >32 subblocks code):


4.2.1 base code - SUMMARY: (of 3 iterations)
File creation : Mean = 2237.644

zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY: (of
3 iterations)
File creation : Mean = 12866.842

more sub blocks per block (4.2.2 + morethan32subblock code):
File creation : Mean = 40316.721

Can someone (ideally Sven) give me a straight answer as to whether or
not the > 32 subblock code actually makes a performance difference for
small files in large block filesystems? And if not, help me understand
why his slides and provided benchmark data have consistently indicated
it does?

-Aaron

On 12/1/17 11:44 AM, Bill Hartner wrote:
> ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> results @ user group meeting.
> 
> -Bill
> 
> Bill Hartner
> IBM Systems
> Scalable I/O Development
> Austin, Texas
> bhartner at us.ibm.com
> home office 512-784-0980
> 
> 
> Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
> AM---Bill, could you say something about what the metadataJan-Frode
> Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> about what the metadata-storage here was? ESS/NL-SAS/3way replication?
> 
> From: Jan-Frode Myklebust <janfrode at tanso.net>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 12/01/2017 06:53 AM
> Subject: Re: [gpfsug-discuss] Online data migration tool
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> 
> ------------------------------------------------------------------------
> 
> 
> 
> Bill, could you say something about what the metadata-storage here was?
> ESS/NL-SAS/3way replication?
> 
> I just asked about this in the internal slack channel #scale-help today..
> 
> 
> 
> -jf
> 
> fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
> <mailto:bhartner at us.ibm.com>>:
> 
>     > "It has a significant performance penalty for small files in large
>     > block size filesystems"
> 
>     Aaron,
> 
>     Below are mdtest results for a test we ran for CORAL - file size was
>     32k.
> 
>     We have not gone back and ran the test on a file system formatted
>     without > 32 subblocks. We'll do that at some point...
> 
>     -Bill
> 
>     -- started at 10/28/2017 17:51:38 --
> 
>     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360
>     -u -y
>     Path: /ibm/fs2-16m-10
>     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> 
>     228 tasks, 67239936 files
> 
>     SUMMARY: (of 3 iterations)
>     Operation Max Min Mean Std Dev
>     --------- --- --- ---- -------
>     File creation : 51953.498 50558.517 51423.221 616.643
>     File stat : 0.000 0.000 0.000 0.000
>     File read : 0.000 0.000 0.000 0.000
>     File removal : 96746.376 92149.535 94658.774 1900.187
>     Tree creation : 1.588 0.070 0.599 0.700
>     Tree removal : 0.213 0.034 0.097 0.082
> 
>     -- finished at 10/28/2017 19:51:54 --
> 
>     Bill Hartner
>     IBM Systems
>     Scalable I/O Development
>     Austin, Texas_
>     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com>
>     home office 512-784-0980
> 
>     _
>     __gpfsug-discuss-bounces at spectrumscale.org_
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>?wrote on
>     11/29/2017 04:41:48 PM:
> 
>     > From: Aaron Knister <_aaron.knister at gmail.com_
>     <mailto:aaron.knister at gmail.com>>
> 
> 
>     > To: gpfsug main discussion list
>     <_gpfsug-discuss at spectrumscale.org_
>     <mailto:gpfsug-discuss at spectrumscale.org>>
> 
>     > Date: 11/29/2017 04:42 PM
> 
> 
>     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> 
>     >
> 
>     > Thanks, Nikhil. Most of that was consistent with my understnading,
>     > however I was under the impression that the >32 subblocks code is
>     > required to achieve the touted 50k file creates/second that Sven has
>     > talked about a bunch of times:
>     >
>     >
>     _http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>
>     >
>     _http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_ <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>
>     > _Sven_Oehme_-_News_from_Research.pdf
>     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>
>     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> 
> 
>     > from those presentations regarding 32 subblocks:
>     >
>     > "It has a significant performance penalty for small files in large
>     > block size filesystems"
> 
>     > although I'm not clear on the specific definition of "large". Many
>     > filesystems I encounter only have a 1M block size so it may not
>     > matter there, although that same presentation clearly shows the
>     > benefit of larger block sizes which is yet *another* thing for which
>     > a migration tool would be helpful.
> 
>     > -Aaron
>     >
>     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com>> wrote:
> 
>     > Hi,
>     >
>     > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
>     > For all Spectrum Scale clusters that are currently at 4.X.X, it is
>     > possible to migrate to 5.0.0 with no offline data migration and no
>     > need to move data. Once these clusters are at 5.0.0, they will
>     > benefit from the performance improvements, new features (such as
>     > file audit logging), and various enhancements that are included in
>     5.0.0.
>     >
>     > That being said, there is one enhancement that will not be applied
>     > to these clusters, and that is the increased number of sub-blocks
>     > per block for small file allocation. This means that for file
>     > systems with a large block size and a lot of small files, the
>     > overall space utilization will be the same it currently is in 4.X.X.
>     > Since file systems created at 4.X.X and earlier used a block size
>     > that kept this allocation in mind, there should be very little
>     > impact on existing file systems.
>     >
>     > Outside of that one particular function, the remainder of the
>     > performance improvements, metadata improvements, updated
>     > compatibility, new functionality, and all of the other enhancements
>     > will be immediately available to you once you complete the upgrade
>     > to 5.0.0 -- with no need to reformat, move data, or take your data
>     offline.
>     >
>     > I hope that clarifies things a little and makes the upgrade path
>     > more accessible.
>     >
>     > Please let me know if there are any other questions or concerns.
>     >
>     > Thank you,
>     > Nikhil Khandelwal
>     > Spectrum Scale Development
>     > Client Adoption
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at _spectrumscale.org_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
>     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
> 
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at _spectrumscale.org_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
> 
>     > _https://urldefense.proofpoint.com/v2/url?_
>     >
>     u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>     >
>     siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> 
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at _spectrumscale.org_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>_
>     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>_______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org
>     https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> 
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776


From alex at calicolabs.com  Fri Dec 15 20:48:16 2017
From: alex at calicolabs.com (Alex Chekholko)
Date: Fri, 15 Dec 2017 12:48:16 -0800
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
Message-ID: <CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>

Hey Aaron,

Can you define your sizes for "large blocks" and "small files"?  If you
dial one up and the other down, your performance will be worse.  And in any
case it's a pathological corner case so it shouldn't matter much for your
workflow, unless you've designed your system with the wrong values.

For example, for bioinformatics workloads, I prefer to use 256KB filesystem
block size, and I'd consider 4MB+ to be "large block size", which would
make the filesystem obviously unsuitable for processing millions of 8KB
files.

You can make a histogram of file sizes in your existing filesystems and
then make your subblock size (1/32 of block size) on the smaller end of
that.   Also definitely use the "small file in inode" feature and put your
metadata on SSD.

Regards,
Alex

On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister <aaron.s.knister at nasa.gov>
wrote:

> Thanks, Bill.
>
> I still don't feel like I've got an clear answer from IBM and frankly
> the core issue of a lack of migration tool was totally dodged.
>
> Again in Sven's presentation from SSUG @ SC17
> (http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf)
> he mentions "It has a significant performance penalty for small files in
> large block size filesystems" and the demonstrates that with several
> mdtest runs (which show the effect with and without the >32 subblocks
> code):
>
>
> 4.2.1 base code - SUMMARY: (of 3 iterations)
> File creation : Mean = 2237.644
>
> zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY: (of
> 3 iterations)
> File creation : Mean = 12866.842
>
> more sub blocks per block (4.2.2 + morethan32subblock code):
> File creation : Mean = 40316.721
>
> Can someone (ideally Sven) give me a straight answer as to whether or
> not the > 32 subblock code actually makes a performance difference for
> small files in large block filesystems? And if not, help me understand
> why his slides and provided benchmark data have consistently indicated
> it does?
>
> -Aaron
>
> On 12/1/17 11:44 AM, Bill Hartner wrote:
> > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> > results @ user group meeting.
> >
> > -Bill
> >
> > Bill Hartner
> > IBM Systems
> > Scalable I/O Development
> > Austin, Texas
> > bhartner at us.ibm.com
> > home office 512-784-0980
> >
> >
> > Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
> > AM---Bill, could you say something about what the metadataJan-Frode
> > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> > about what the metadata-storage here was? ESS/NL-SAS/3way replication?
> >
> > From: Jan-Frode Myklebust <janfrode at tanso.net>
> > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> > Date: 12/01/2017 06:53 AM
> > Subject: Re: [gpfsug-discuss] Online data migration tool
> > Sent by: gpfsug-discuss-bounces at spectrumscale.org
> >
> > ------------------------------------------------------------------------
> >
> >
> >
> > Bill, could you say something about what the metadata-storage here was?
> > ESS/NL-SAS/3way replication?
> >
> > I just asked about this in the internal slack channel #scale-help today..
> >
> >
> >
> > -jf
> >
> > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
> > <mailto:bhartner at us.ibm.com>>:
> >
> >     > "It has a significant performance penalty for small files in large
> >     > block size filesystems"
> >
> >     Aaron,
> >
> >     Below are mdtest results for a test we ran for CORAL - file size was
> >     32k.
> >
> >     We have not gone back and ran the test on a file system formatted
> >     without > 32 subblocks. We'll do that at some point...
> >
> >     -Bill
> >
> >     -- started at 10/28/2017 17:51:38 --
> >
> >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
> >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360
> >     -u -y
> >     Path: /ibm/fs2-16m-10
> >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> >
> >     228 tasks, 67239936 files
> >
> >     SUMMARY: (of 3 iterations)
> >     Operation Max Min Mean Std Dev
> >     --------- --- --- ---- -------
> >     File creation : 51953.498 50558.517 51423.221 616.643
> >     File stat : 0.000 0.000 0.000 0.000
> >     File read : 0.000 0.000 0.000 0.000
> >     File removal : 96746.376 92149.535 94658.774 1900.187
> >     Tree creation : 1.588 0.070 0.599 0.700
> >     Tree removal : 0.213 0.034 0.097 0.082
> >
> >     -- finished at 10/28/2017 19:51:54 --
> >
> >     Bill Hartner
> >     IBM Systems
> >     Scalable I/O Development
> >     Austin, Texas_
> >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com>
> >     home office 512-784-0980
> >
> >     _
> >     __gpfsug-discuss-bounces at spectrumscale.org_
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org> wrote on
> >     11/29/2017 04:41:48 PM:
> >
> >     > From: Aaron Knister <_aaron.knister at gmail.com_
> >     <mailto:aaron.knister at gmail.com>>
> >
> >
> >     > To: gpfsug main discussion list
> >     <_gpfsug-discuss at spectrumscale.org_
> >     <mailto:gpfsug-discuss at spectrumscale.org>>
> >
> >     > Date: 11/29/2017 04:42 PM
> >
> >
> >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> >
> >     >
> >
> >     > Thanks, Nikhil. Most of that was consistent with my understnading,
> >     > however I was under the impression that the >32 subblocks code is
> >     > required to achieve the touted 50k file creates/second that Sven
> has
> >     > talked about a bunch of times:
> >     >
> >     >
> >     _http://files.gpfsug.org/presentations/2017/Manchester/
> 08_Research_Topics.pdf_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__files.gpfsug.org_presentations_2017_Manchester_
> 08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>
> >     >
> >     _http://files.gpfsug.org/presentations/2017/Ehningen/
> 31_-_SSUG17DE_-_ <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__files.gpfsug.org_presentations_2017_Ehningen_
> 31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>
> >     > _Sven_Oehme_-_News_from_Research.pdf
> >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__files.gpfsug.org_presentations_2016_SC16_12-5F-
> 2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW
> UGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--
> uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>
> >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> >
> >
> >     > from those presentations regarding 32 subblocks:
> >     >
> >     > "It has a significant performance penalty for small files in large
> >     > block size filesystems"
> >
> >     > although I'm not clear on the specific definition of "large". Many
> >     > filesystems I encounter only have a 1M block size so it may not
> >     > matter there, although that same presentation clearly shows the
> >     > benefit of larger block sizes which is yet *another* thing for
> which
> >     > a migration tool would be helpful.
> >
> >     > -Aaron
> >     >
> >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
> >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com>> wrote:
> >
> >     > Hi,
> >     >
> >     > I would like to clarify migration path to 5.0.0 from 4.X.X
> clusters.
> >     > For all Spectrum Scale clusters that are currently at 4.X.X, it is
> >     > possible to migrate to 5.0.0 with no offline data migration and no
> >     > need to move data. Once these clusters are at 5.0.0, they will
> >     > benefit from the performance improvements, new features (such as
> >     > file audit logging), and various enhancements that are included in
> >     5.0.0.
> >     >
> >     > That being said, there is one enhancement that will not be applied
> >     > to these clusters, and that is the increased number of sub-blocks
> >     > per block for small file allocation. This means that for file
> >     > systems with a large block size and a lot of small files, the
> >     > overall space utilization will be the same it currently is in
> 4.X.X.
> >     > Since file systems created at 4.X.X and earlier used a block size
> >     > that kept this allocation in mind, there should be very little
> >     > impact on existing file systems.
> >     >
> >     > Outside of that one particular function, the remainder of the
> >     > performance improvements, metadata improvements, updated
> >     > compatibility, new functionality, and all of the other enhancements
> >     > will be immediately available to you once you complete the upgrade
> >     > to 5.0.0 -- with no need to reformat, move data, or take your data
> >     offline.
> >     >
> >     > I hope that clarifies things a little and makes the upgrade path
> >     > more accessible.
> >     >
> >     > Please let me know if there are any other questions or concerns.
> >     >
> >     > Thank you,
> >     > Nikhil Khandelwal
> >     > Spectrum Scale Development
> >     > Client Adoption
> >     >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at _spectrumscale.org_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
> >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=
> DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW
> UGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_
> TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
> >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at _spectrumscale.org_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
> >
> >     > _https://urldefense.proofpoint.com/v2/url?_
> >     >
> >     u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=
> DwICAg&c=jf_iaSHvJObTbx-
> >     >
> >     siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=
> DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_
> W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> >
> >     _______________________________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at _spectrumscale.org_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >_
> >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=
> DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW
> UGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_
> TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>___________________
> ____________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at spectrumscale.org
> >     https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/5f33cc4a/attachment.htm>

From aaron.s.knister at nasa.gov  Fri Dec 15 22:47:43 2017
From: aaron.s.knister at nasa.gov (Aaron Knister)
Date: Fri, 15 Dec 2017 17:47:43 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
Message-ID: <03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>

Thanks, Alex. I'm all too familiar with the trade offs between large
blocks and small files and we do use pretty robust SSD storage for our
metadata. We support a wide range of workloads and we have some folks
with many small (<1M) files and other folks with many large (>256MB) files.

My point in this thread is that IBM has said over and over again in
presentations that there is a significant performance gain with the >32
subblocks code on filesystems with large block sizes (although to your
point I'm not clear on exactly what large means since I didn't define
large in this context). Therefore given that the >32 subblock code gives
a significant performance gain one could reasonably assume that having a
filesystem with >32 subblocks is required to see this gain (rather than
just running the >32 subblocks code on an fs w/o > 32 subblocks).

This lead me to ask about a migration tool because in my mind if there's
a performance gain from having >32 subblocks on the FS I'd like that
feature and having to manually copy 10's of PB to new hardware to get
this performance boost is unacceptable. However, IBM can't seem to make
up their mind about whether or not the >32 subblocks code *actually*
provides a performance increase. This seems like a pretty
straightforward question.

-Aaron

On 12/15/17 3:48 PM, Alex Chekholko wrote:
> Hey Aaron,
> 
> Can you define your sizes for "large blocks" and "small files"?? If you
> dial one up and the other down, your performance will be worse.? And in
> any case it's a pathological corner case so it shouldn't matter much for
> your workflow, unless you've designed your system with the wrong values.
> 
> For example, for bioinformatics workloads, I prefer to use 256KB
> filesystem block size, and I'd consider 4MB+ to be "large block size",
> which would make the filesystem obviously unsuitable for processing
> millions of 8KB files.
> 
> You can make a histogram of file sizes in your existing filesystems and
> then make your subblock size (1/32 of block size) on the smaller end of
> that.? ?Also definitely use the "small file in inode" feature and put
> your metadata on SSD.
> 
> Regards,
> Alex
> 
> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
> 
>     Thanks, Bill.
> 
>     I still don't feel like I've got an clear answer from IBM and frankly
>     the core issue of a lack of migration tool was totally dodged.
> 
>     Again in Sven's presentation from SSUG @ SC17
>     (http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
>     he mentions "It has a significant performance penalty for small files in
>     large block size filesystems" and the demonstrates that with several
>     mdtest runs (which show the effect with and without the >32
>     subblocks code):
> 
> 
>     4.2.1 base code - SUMMARY: (of 3 iterations)
>     File creation : Mean = 2237.644
> 
>     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):? SUMMARY: (of
>     3 iterations)
>     File creation : Mean = 12866.842
> 
>     more sub blocks per block (4.2.2 + morethan32subblock code):
>     File creation : Mean = 40316.721
> 
>     Can someone (ideally Sven) give me a straight answer as to whether or
>     not the > 32 subblock code actually makes a performance difference for
>     small files in large block filesystems? And if not, help me understand
>     why his slides and provided benchmark data have consistently indicated
>     it does?
> 
>     -Aaron
> 
>     On 12/1/17 11:44 AM, Bill Hartner wrote:
>     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
>     > results @ user group meeting.
>     >
>     > -Bill
>     >
>     > Bill Hartner
>     > IBM Systems
>     > Scalable I/O Development
>     > Austin, Texas
>     > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
>     > home office 512-784-0980 <tel:512-784-0980>
>     >
>     >
>     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
>     > AM---Bill, could you say something about what the metadataJan-Frode
>     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
>     > about what the metadata-storage here was? ESS/NL-SAS/3way replication?
>     >
>     > From: Jan-Frode Myklebust <janfrode at tanso.net <mailto:janfrode at tanso.net>>
>     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org
>     <mailto:gpfsug-discuss at spectrumscale.org>>
>     > Date: 12/01/2017 06:53 AM
>     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     >
>     >
>     ------------------------------------------------------------------------
>     >
>     >
>     >
>     > Bill, could you say something about what the metadata-storage here was?
>     > ESS/NL-SAS/3way replication?
>     >
>     > I just asked about this in the internal slack channel #scale-help today..
>     >
>     >
>     >
>     > -jf
>     >
>     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
>     > <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>:
>     >
>     >? ? ?> "It has a significant performance penalty for small files in
>     large
>     >? ? ?> block size filesystems"
>     >
>     >? ? ?Aaron,
>     >
>     >? ? ?Below are mdtest results for a test we ran for CORAL - file
>     size was
>     >? ? ?32k.
>     >
>     >? ? ?We have not gone back and ran the test on a file system formatted
>     >? ? ?without > 32 subblocks. We'll do that at some point...
>     >
>     >? ? ?-Bill
>     >
>     >? ? ?-- started at 10/28/2017 17:51:38 --
>     >
>     >? ? ?mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>     >? ? ?Command line used: /tmp/mdtest-binary-dir/mdtest -d
>     >? ? ?/ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
>     -p 360
>     >? ? ?-u -y
>     >? ? ?Path: /ibm/fs2-16m-10
>     >? ? ?FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>     >
>     >? ? ?228 tasks, 67239936 files
>     >
>     >? ? ?SUMMARY: (of 3 iterations)
>     >? ? ?Operation Max Min Mean Std Dev
>     >? ? ?--------- --- --- ---- -------
>     >? ? ?File creation : 51953.498 50558.517 51423.221 616.643
>     >? ? ?File stat : 0.000 0.000 0.000 0.000
>     >? ? ?File read : 0.000 0.000 0.000 0.000
>     >? ? ?File removal : 96746.376 92149.535 94658.774 1900.187
>     >? ? ?Tree creation : 1.588 0.070 0.599 0.700
>     >? ? ?Tree removal : 0.213 0.034 0.097 0.082
>     >
>     >? ? ?-- finished at 10/28/2017 19:51:54 --
>     >
>     >? ? ?Bill Hartner
>     >? ? ?IBM Systems
>     >? ? ?Scalable I/O Development
>     >? ? ?Austin, Texas_
>     >? ? ?__bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
>     <mailto:bhartner at us.ibm.com>>
>     >? ? ?home office 512-784-0980 <tel:512-784-0980>
>     >
>     >? ? ?_
>     >? ? ?__gpfsug-discuss-bounces at spectrumscale.org_
>     >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>?wrote on
>     >? ? ?11/29/2017 04:41:48 PM:
>     >
>     >? ? ?> From: Aaron Knister <_aaron.knister at gmail.com_
>     >? ? ?<mailto:aaron.knister at gmail.com <mailto:aaron.knister at gmail.com>>>
>     >
>     >
>     >? ? ?> To: gpfsug main discussion list
>     >? ? ?<_gpfsug-discuss at spectrumscale.org_
>     >? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>     <mailto:gpfsug-discuss at spectrumscale.org>>>
>     >
>     >? ? ?> Date: 11/29/2017 04:42 PM
>     >
>     >
>     >? ? ?> Subject: Re: [gpfsug-discuss] Online data migration tool
>     >? ? ?> Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>     >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>     >
>     >? ? ?>
>     >
>     >? ? ?> Thanks, Nikhil. Most of that was consistent with my understnading,
>     >? ? ?> however I was under the impression that the >32 subblocks code is
>     >? ? ?> required to achieve the touted 50k file creates/second that Sven has
>     >? ? ?> talked about a bunch of times:
>     >? ? ?>
>     >? ? ?>
>     >? ?
>     ?_http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>     <http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>     >? ? ?>
>     >? ?
>     ?_http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
>     <http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>     >? ? ?> _Sven_Oehme_-_News_from_Research.pdf
>     >? ? ?> _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>     >? ? ?> _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>     >
>     >
>     >? ? ?> from those presentations regarding 32 subblocks:
>     >? ? ?>
>     >? ? ?> "It has a significant performance penalty for small files in large
>     >? ? ?> block size filesystems"
>     >
>     >? ? ?> although I'm not clear on the specific definition of "large". Many
>     >? ? ?> filesystems I encounter only have a 1M block size so it may not
>     >? ? ?> matter there, although that same presentation clearly shows the
>     >? ? ?> benefit of larger block sizes which is yet *another* thing for which
>     >? ? ?> a migration tool would be helpful.
>     >
>     >? ? ?> -Aaron
>     >? ? ?>
>     >? ? ?> On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>     >? ? ?<_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
>     <mailto:nikhilk at us.ibm.com>>> wrote:
>     >
>     >? ? ?> Hi,
>     >? ? ?>
>     >? ? ?> I would like to clarify migration path to 5.0.0 from 4.X.X
>     clusters.
>     >? ? ?> For all Spectrum Scale clusters that are currently at 4.X.X,
>     it is
>     >? ? ?> possible to migrate to 5.0.0 with no offline data migration
>     and no
>     >? ? ?> need to move data. Once these clusters are at 5.0.0, they will
>     >? ? ?> benefit from the performance improvements, new features (such as
>     >? ? ?> file audit logging), and various enhancements that are
>     included in
>     >? ? ?5.0.0.
>     >? ? ?>
>     >? ? ?> That being said, there is one enhancement that will not be
>     applied
>     >? ? ?> to these clusters, and that is the increased number of
>     sub-blocks
>     >? ? ?> per block for small file allocation. This means that for file
>     >? ? ?> systems with a large block size and a lot of small files, the
>     >? ? ?> overall space utilization will be the same it currently is
>     in 4.X.X.
>     >? ? ?> Since file systems created at 4.X.X and earlier used a block
>     size
>     >? ? ?> that kept this allocation in mind, there should be very little
>     >? ? ?> impact on existing file systems.
>     >? ? ?>
>     >? ? ?> Outside of that one particular function, the remainder of the
>     >? ? ?> performance improvements, metadata improvements, updated
>     >? ? ?> compatibility, new functionality, and all of the other
>     enhancements
>     >? ? ?> will be immediately available to you once you complete the
>     upgrade
>     >? ? ?> to 5.0.0 -- with no need to reformat, move data, or take
>     your data
>     >? ? ?offline.
>     >? ? ?>
>     >? ? ?> I hope that clarifies things a little and makes the upgrade path
>     >? ? ?> more accessible.
>     >? ? ?>
>     >? ? ?> Please let me know if there are any other questions or concerns.
>     >? ? ?>
>     >? ? ?> Thank you,
>     >? ? ?> Nikhil Khandelwal
>     >? ? ?> Spectrum Scale Development
>     >? ? ?> Client Adoption
>     >? ? ?>
>     >? ? ?> _______________________________________________
>     >? ? ?> gpfsug-discuss mailing list
>     >? ? ?> gpfsug-discuss at _spectrumscale.org_
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >? ? ?> _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>     >
>     >? ? ?> _______________________________________________
>     >? ? ?> gpfsug-discuss mailing list
>     >? ? ?> gpfsug-discuss at _spectrumscale.org_
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >
>     >? ? ?> _https://urldefense.proofpoint.com/v2/url?_
>     <https://urldefense.proofpoint.com/v2/url?_>
>     >? ? ?>
>     >? ? ?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>     >? ? ?>
>     >? ? ?siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>     >
>     >? ? ?_______________________________________________
>     >? ? ?gpfsug-discuss mailing list
>     >? ? ?gpfsug-discuss at _spectrumscale.org_
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>     >? ? ?__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>     >? ? ?gpfsug-discuss mailing list
>     >? ? ?gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     >? ?
>     ?https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>     >
> 
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776 <tel:%28301%29%20286-2776>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> 
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776


From oehmes at gmail.com  Fri Dec 15 23:06:52 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Fri, 15 Dec 2017 23:06:52 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
Message-ID: <CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>

i thought i answered that already, but maybe i just thought about answering
it and then forgot about it :-D

so yes more than 32 subblocks per block significant increase the
performance of filesystems with small files, for the sake of the argument
let's say 32k in a large block filesystem again for sake of argument say
16MB.

you probably ask why ?

if you create a file and write 32k into it in a pre 5.0.0 Version 16 MB
filesystem your client actually doesn't write 32k to the NSD Server, it
writes 512k, because thats the subblock size and we need to write the full
subblock (for security reasons). so first you waste significant memory on
the client to cache that zero padding, you waste network bandwidth and you
waste NSD Server cache because you store it there too. this means you
overrun the cache more quickly, means you start doing read/modify writes
earlier on all your nice large raid tracks... i guess you get the story by
now.

in fact,  if you have a good raid code that can drive really a lot of
bandwidth out of individual drives like a GNR system you get more
performance for small file writes as larger your blocksize is, because we
can 'pack' more files into larger i/os and therefore turn a small file
create workload into a bandwidth workload, essentially exactly what we did
and i demonstrated in the CORAL presentation .

hope that makes this crystal clear now .

sven



On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister <aaron.s.knister at nasa.gov>
wrote:

> Thanks, Alex. I'm all too familiar with the trade offs between large
> blocks and small files and we do use pretty robust SSD storage for our
> metadata. We support a wide range of workloads and we have some folks
> with many small (<1M) files and other folks with many large (>256MB) files.
>
> My point in this thread is that IBM has said over and over again in
> presentations that there is a significant performance gain with the >32
> subblocks code on filesystems with large block sizes (although to your
> point I'm not clear on exactly what large means since I didn't define
> large in this context). Therefore given that the >32 subblock code gives
> a significant performance gain one could reasonably assume that having a
> filesystem with >32 subblocks is required to see this gain (rather than
> just running the >32 subblocks code on an fs w/o > 32 subblocks).
>
> This lead me to ask about a migration tool because in my mind if there's
> a performance gain from having >32 subblocks on the FS I'd like that
> feature and having to manually copy 10's of PB to new hardware to get
> this performance boost is unacceptable. However, IBM can't seem to make
> up their mind about whether or not the >32 subblocks code *actually*
> provides a performance increase. This seems like a pretty
> straightforward question.
>
> -Aaron
>
> On 12/15/17 3:48 PM, Alex Chekholko wrote:
> > Hey Aaron,
> >
> > Can you define your sizes for "large blocks" and "small files"?  If you
> > dial one up and the other down, your performance will be worse.  And in
> > any case it's a pathological corner case so it shouldn't matter much for
> > your workflow, unless you've designed your system with the wrong values.
> >
> > For example, for bioinformatics workloads, I prefer to use 256KB
> > filesystem block size, and I'd consider 4MB+ to be "large block size",
> > which would make the filesystem obviously unsuitable for processing
> > millions of 8KB files.
> >
> > You can make a histogram of file sizes in your existing filesystems and
> > then make your subblock size (1/32 of block size) on the smaller end of
> > that.   Also definitely use the "small file in inode" feature and put
> > your metadata on SSD.
> >
> > Regards,
> > Alex
> >
> > On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> > <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
> >
> >     Thanks, Bill.
> >
> >     I still don't feel like I've got an clear answer from IBM and frankly
> >     the core issue of a lack of migration tool was totally dodged.
> >
> >     Again in Sven's presentation from SSUG @ SC17
> >     (
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
> >     he mentions "It has a significant performance penalty for small
> files in
> >     large block size filesystems" and the demonstrates that with several
> >     mdtest runs (which show the effect with and without the >32
> >     subblocks code):
> >
> >
> >     4.2.1 base code - SUMMARY: (of 3 iterations)
> >     File creation : Mean = 2237.644
> >
> >     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY:
> (of
> >     3 iterations)
> >     File creation : Mean = 12866.842
> >
> >     more sub blocks per block (4.2.2 + morethan32subblock code):
> >     File creation : Mean = 40316.721
> >
> >     Can someone (ideally Sven) give me a straight answer as to whether or
> >     not the > 32 subblock code actually makes a performance difference
> for
> >     small files in large block filesystems? And if not, help me
> understand
> >     why his slides and provided benchmark data have consistently
> indicated
> >     it does?
> >
> >     -Aaron
> >
> >     On 12/1/17 11:44 AM, Bill Hartner wrote:
> >     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> >     > results @ user group meeting.
> >     >
> >     > -Bill
> >     >
> >     > Bill Hartner
> >     > IBM Systems
> >     > Scalable I/O Development
> >     > Austin, Texas
> >     > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
> >     > home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >
> >     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017
> 06:53:44
> >     > AM---Bill, could you say something about what the metadataJan-Frode
> >     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> >     > about what the metadata-storage here was? ESS/NL-SAS/3way
> replication?
> >     >
> >     > From: Jan-Frode Myklebust <janfrode at tanso.net <mailto:
> janfrode at tanso.net>>
> >     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>
> >     > Date: 12/01/2017 06:53 AM
> >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     > Sent by: gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> >     >
> >     >
> >
>  ------------------------------------------------------------------------
> >     >
> >     >
> >     >
> >     > Bill, could you say something about what the metadata-storage here
> was?
> >     > ESS/NL-SAS/3way replication?
> >     >
> >     > I just asked about this in the internal slack channel #scale-help
> today..
> >     >
> >     >
> >     >
> >     > -jf
> >     >
> >     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
> <_bhartner at us.ibm.com_
> >     > <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>:
> >     >
> >     >     > "It has a significant performance penalty for small files in
> >     large
> >     >     > block size filesystems"
> >     >
> >     >     Aaron,
> >     >
> >     >     Below are mdtest results for a test we ran for CORAL - file
> >     size was
> >     >     32k.
> >     >
> >     >     We have not gone back and ran the test on a file system
> formatted
> >     >     without > 32 subblocks. We'll do that at some point...
> >     >
> >     >     -Bill
> >     >
> >     >     -- started at 10/28/2017 17:51:38 --
> >     >
> >     >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> >     >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
> >     >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
> >     -p 360
> >     >     -u -y
> >     >     Path: /ibm/fs2-16m-10
> >     >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> >     >
> >     >     228 tasks, 67239936 files
> >     >
> >     >     SUMMARY: (of 3 iterations)
> >     >     Operation Max Min Mean Std Dev
> >     >     --------- --- --- ---- -------
> >     >     File creation : 51953.498 50558.517 51423.221 616.643
> >     >     File stat : 0.000 0.000 0.000 0.000
> >     >     File read : 0.000 0.000 0.000 0.000
> >     >     File removal : 96746.376 92149.535 94658.774 1900.187
> >     >     Tree creation : 1.588 0.070 0.599 0.700
> >     >     Tree removal : 0.213 0.034 0.097 0.082
> >     >
> >     >     -- finished at 10/28/2017 19:51:54 --
> >     >
> >     >     Bill Hartner
> >     >     IBM Systems
> >     >     Scalable I/O Development
> >     >     Austin, Texas_
> >     >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
> >     <mailto:bhartner at us.ibm.com>>
> >     >     home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >     _
> >     >     __gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>> wrote on
> >     >     11/29/2017 04:41:48 PM:
> >     >
> >     >     > From: Aaron Knister <_aaron.knister at gmail.com_
> >     >     <mailto:aaron.knister at gmail.com <mailto:
> aaron.knister at gmail.com>>>
> >     >
> >     >
> >     >     > To: gpfsug main discussion list
> >     >     <_gpfsug-discuss at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>>
> >     >
> >     >     > Date: 11/29/2017 04:42 PM
> >     >
> >     >
> >     >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
> >     >
> >     >     >
> >     >
> >     >     > Thanks, Nikhil. Most of that was consistent with my
> understnading,
> >     >     > however I was under the impression that the >32 subblocks
> code is
> >     >     > required to achieve the touted 50k file creates/second that
> Sven has
> >     >     > talked about a bunch of times:
> >     >     >
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >     <
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >>
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
> >     <
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >>
> >     >     > _Sven_Oehme_-_News_from_Research.pdf
> >     >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
> >     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >>
> >     >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> >     >
> >     >
> >     >     > from those presentations regarding 32 subblocks:
> >     >     >
> >     >     > "It has a significant performance penalty for small files in
> large
> >     >     > block size filesystems"
> >     >
> >     >     > although I'm not clear on the specific definition of
> "large". Many
> >     >     > filesystems I encounter only have a 1M block size so it may
> not
> >     >     > matter there, although that same presentation clearly shows
> the
> >     >     > benefit of larger block sizes which is yet *another* thing
> for which
> >     >     > a migration tool would be helpful.
> >     >
> >     >     > -Aaron
> >     >     >
> >     >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
> >     >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
> >     <mailto:nikhilk at us.ibm.com>>> wrote:
> >     >
> >     >     > Hi,
> >     >     >
> >     >     > I would like to clarify migration path to 5.0.0 from 4.X.X
> >     clusters.
> >     >     > For all Spectrum Scale clusters that are currently at 4.X.X,
> >     it is
> >     >     > possible to migrate to 5.0.0 with no offline data migration
> >     and no
> >     >     > need to move data. Once these clusters are at 5.0.0, they
> will
> >     >     > benefit from the performance improvements, new features
> (such as
> >     >     > file audit logging), and various enhancements that are
> >     included in
> >     >     5.0.0.
> >     >     >
> >     >     > That being said, there is one enhancement that will not be
> >     applied
> >     >     > to these clusters, and that is the increased number of
> >     sub-blocks
> >     >     > per block for small file allocation. This means that for file
> >     >     > systems with a large block size and a lot of small files, the
> >     >     > overall space utilization will be the same it currently is
> >     in 4.X.X.
> >     >     > Since file systems created at 4.X.X and earlier used a block
> >     size
> >     >     > that kept this allocation in mind, there should be very
> little
> >     >     > impact on existing file systems.
> >     >     >
> >     >     > Outside of that one particular function, the remainder of the
> >     >     > performance improvements, metadata improvements, updated
> >     >     > compatibility, new functionality, and all of the other
> >     enhancements
> >     >     > will be immediately available to you once you complete the
> >     upgrade
> >     >     > to 5.0.0 -- with no need to reformat, move data, or take
> >     your data
> >     >     offline.
> >     >     >
> >     >     > I hope that clarifies things a little and makes the upgrade
> path
> >     >     > more accessible.
> >     >     >
> >     >     > Please let me know if there are any other questions or
> concerns.
> >     >     >
> >     >     > Thank you,
> >     >     > Nikhil Khandelwal
> >     >     > Spectrum Scale Development
> >     >     > Client Adoption
> >     >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>
> >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >
> >     >     > _https://urldefense.proofpoint.com/v2/url?_
> >     <https://urldefense.proofpoint.com/v2/url?_>
> >     >     >
> >     >
>  u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
> >     >     >
> >     >
>  siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> >     >
> >     >     _______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>_
> >     >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>_______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >
> >     >
> >     >
> >     >
> >     >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >     >
> >
> >     --
> >     Aaron Knister
> >     NASA Center for Climate Simulation (Code 606.2)
> >     Goddard Space Flight Center
> >     (301) 286-2776 <tel:%28301%29%20286-2776>
> >     _______________________________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/1d465799/attachment.htm>

From Robert.Oesterlin at nuance.com  Mon Dec 18 19:43:35 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Mon, 18 Dec 2017 19:43:35 +0000
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=> replied to the IBM Spectrum Scale V5.0 announcements<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=> topic thread in the General Parallel File System - Announce (GPFS - Announce)<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=> forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
as updated, April 2017

Changes to this release of the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library include the following:

Added DMPs for TIP events

       A topic is added listing the directed maintenance procedures for TIP events.
       The DMPs help users resolve issues caused due to TIP events.

AFM and AFM DR

       - Compression and snapshot ILM policy supported.
       - A general recommendation added for the Gateway node.
       - Configuration parameters added - afmMaxParallelRecoveries, afmAsyncOpWaitTimeout,
           afmSyncOpWaitTimeout, and afmRevalOpWaitTimeout.
       - Configuration parameters modified - afmRPO and afmHashVersion.

Authentication: Authentication packages

       Updated the authentication page to include packages specific to Ubuntu.

Authentication: AD-based authentication

       New information is added on NFS with server-side group
       lookup and Active Directory authentication.

Authentication: Primary group selection configurable for AD + RFC2307 based authentication

       Ability to choose primary group as set in "UNIX attributes" of a user on
       Active Directory introduced with AD + RFC2307 based authentication scheme.
       Earlier, Windows primary group was by default selected as the primary group.

Big data and analytics
       - The GPFS Ambari integration package is now called the IBM Spectrum
         Scale Ambari management pack (in short, management pack or MPack).
       - IBM Spectrum Scale Ambari management pack version 2.4.2.1 with
         HDFS Transparency version 2.7.3.1 supports BI 4.2/BI 4.2.5 IOP migration
         to HDP 2.6.2.
       - Supports the remote mount configuration in Ambari.
       - Supports the multiple file systems configuration. In management
         pack version 2.4.2.1, the current limit is two file systems.
       - The Short circuit write is supported for better performance.
       - In HDFS Transparency, the Ranger performance is enhanced.

Changes to IBM Spectrum Scale management API

   Added the following new commands:

       GET /perfmon/data
       GET /filesystems/{filesystemName}/afm/state
       DELETE /nodes/{name}
       POST /nodes
       GET /nodeclasses
       POST /nodeclasses
       DELETE /nodeclasses/{nodeclassName}
       GET /nodeclasses/{nodeclassName}
       PUT /nodeclasses/{nodeclassName}
       DELETE /jobs/jobId
       POST /filesystems/{filesystemName}/filesets/{filesetName}/psnaps
       DELETE /filesystems/{filesystemName}/filesets/{filesetName}/psnaps/{snapshotName}
       GET /thresholds
       GET /thresholds/{name}
       POST /thresholds
       DELETE /thresholds/{name}

IBM Spectrum Scale GUI changes

   - Added new Networks page to monitor the performance, configuration, and
      adapters of network configurations in the cluster. You can monitor the
      network performance with respect to the IP and RDMA interfaces used in the configuration.
    - Added new Monitoring > Thresholds page to create and monitor the threshold rules
      that are defined in the system.
    - Added Access > Remote Connections page to enable the GUI node of the
      local cluster to monitor the remote cluster by establishing a connection
      with the GUI node of the remote cluster.
    - Added Settings > Call Home page to configure call home. Configuring
      the call home feature helps the IBM? Support to monitor the system.
      Configuring call home also helps to reduce the response time of the
      IBM Support to resolve any issues.
      The diagnostic data that is downloaded through the Settings > Diagnostic Data
      can be uploaded to a problem management record (PMR) by using
      the call home feature in the backend. To upload the diagnostic data, right-click
      the relevant data set from the Previously Collected Diagnostic Data, and
      select Upload to PMR.
    - Added file system creation capabilities in GUI. Use the Files > File Systems > Create File System
      option to launch the Create File System wizard. In the Create File System wizard, you can specify
      the following details of the file system:
         - File system name
         - Storage pools
         - NSDs for the file systems
         - Failure groups
         - NSD order for data writes
         - Maximum number of Spectrum Scale clients
         - Maximum number of inodes of the root fileset
         - Whether to enable quota and scope for the quota definition
         - Whether to enable DMAPI
         - Mount point and automatic mount mode
    - Added the aggregation levels Access Point and Filesets and removed Account
      for the resource type Transparent Cloud Tiering in the Monitoring > Statisitics page.
    - The Files > Transparent Cloud Tiering page now displays the file systems
      and filesets that are mapped with the cloud service. It also shows the
      connection of such a container pair configuration to a cloud account and
      the corresponding CSAP that is configured for the cloud account.
    - Changes to capacity monitoring in the GUI
         - Moved the Capacity page from Monitoring to Files menu in the navigation
           and renamed the GUI page to User Capacity.
         - Only the file data user capacity can be viewed from the Files > User Capacity page.
           Removed the pools, filesets, file system capacity monitoring options from the Files > User Capacity page.
           You can monitor the capacity of these components from the respective GUI pages.
         - Replaced the GPFSPoolCap sensor with the GPFSPool sensor and separate data and metadata
           level capacity monitoring are introduced in the performance charts available in the

           Files > File Systems and Storage > Pools pages.
         - New GPFSPool-based data and metadata performance monitoring metrics are available
           for selection in the Files > Statistics > Edit > Capacity section.
           You need to select the aggregation level as Pool to view these metrics.
    - AFM monitoring changes in the GUI
         - Provides the number of AFM filesets and the corresponding export server maps.
           Each export map establishes a mapping between the gateway node and the NFS
           host name to allow parallel data transfers from cache to home.
         - By using the Request Access option available in the Files > Active File Management
           or Access > Remote Connection page in the GUI, you can now establish connection with remote clusters.
           After establishing the connection, you can monitor the following AFM and AFM DR
           configuration details across clusters:
              * On home and secondary, you can see the AFM relationships configuration,
                health status, and performance values of the Cache and Disaster Recovery grids.
              * On the Overview tab of the detailed view, the available home and secondary inodes are available.
              * On the Overview tab of the detailed view, the details such as NFS throughput,
                IOPs, and latency details are available, if the protocol is NFS.
    - New option to create AFM peer snapshots through GUI. Use the Create Peer Snapshot option
      in the Files > Snapshots page to create peer snapshots. You can view and delete these peer snapshots from
      the Snapshots page and also from the detailed view of the Files > Active File Management page.

Encryption: GSKit V8 improves cryptographic performance on IBM POWER8

   The IBM Global Security Kit (GSKit) Version 8 and later improves cryptographic performance
   on IBM POWER8 hardware. The version of GSKit that is shipped with IBM Spectrum Scale v5.0.0
   offers better performance on POWER8, compared with the versions shipped with earlier releases.

File compression: The lz4 library provides fast access to compressed data

   File compression supports the lz4 compression library. Lz4 is intended primarily for active data and
   favors read-access speed over maximized space saving.

File data: Block and subblock sizes improve I/O performance and reduce fragmentation

   The default block size is larger, 4 MiB instead of 256 KiB, and the sizes of subblocks relative to blocks
   s are smaller, for example, 8 KiB subblocks in a 4 MiB block. A larger block size improves the
   file system performance and a smaller subblock size reduces the amount of unused space. For many business
   applications, the default value of 4 MiB provides the best balance of improved performance
   and reduced fragmentation.

File encryption: AES-XTS encryption is faster on x86 in non-FIPS mode

    On x86 architecture in non-FIPS mode, file encryption with the AES algorithm in XTS mode
    is faster than it was.

File systems: File system rebalancing is faster

    Rebalancing is implemented by a lenient round-robin method that typically runs faster than the
    previously used method of strict round robin. The strict round robin method is available as an option.

installation toolkit changes

    - The installation toolkit has added support for the installation and the deployment
      of IBM Spectrum Scale in a cluster containing Elastic Storage Server (ESS).
    - The installation toolkit has added support for enabling and configuring call home.
    - The installation toolkit has added support for enabling and configuring file audit logging.
    - The installation toolkit has added support for the installation and the
      deployment of IBM Spectrum Scale on Ubuntu 16.04 LTS nodes.
    - The installation toolkit has added verification of passwordless SSH during
      prechecks before installation, deployment, or upgrade.
    - The installation toolkit has added support for cumulative object upgrade.

mmafmctl command

    The --outband parameter is deprecated.

mmcallhome command: Enhancements

    - Addition of -Y option
        * The -Y displays the command output in a parseable format with a colon (:) as a field delimiter.
    - Addition of --pmr option
        * The --pmr option allows you to upload data to existing PMRs using the
          mmcallhome run SendFile command.

mmchconfig command: Enhancements

    - Encrypted files can be copied into an LROC device
        * With the lrocEnableStoringClearText attribute, you can control whether file
          data from encrypted files, which is held in memory as cleartext, is
          copied into a local read-only cache (LROC) device.
    - InfiniBand addresses can be specified for RDMA transfers
        * In the verbsPorts attribute, you can specify InfiniBand addresses
          for RDMA transfers between an NSD client and server.

mmchnsd command: Change NSDs without unmounting the file system

    When you add or remove NSDs or do other operations with mmchnsd,
    you do not need to unmount the file system.

mmcrfs command: Enhancements

     - The default data block size is 4 MiB with an 8 KiB subblock size
         * If no block size is specified, a file system is created with a 4 MiB block size and an
           8 KiB subblock size. The minimum release level (minReleaseLevel) of the
           cluster must be 5.0.0 or greater when the file system is created.
     - The default log file size depends on block size and metadata size
         * If the block size is 512 KiB or larger and the metadata block size 256 KiB or larger,
           then the default log file size is 32 MiB. Otherwise, the default log file
           size is 4 MiB or the metadata block size, whichever is larger.
     - The default method for updating atime is relatime
         * If the minimum release level (minReleaseLevel) of the cluster is 5.0.0 or greater
           when the file system is created, the default method for updating atime is relatime.

mmdsh command: Several options are no longer supported

     The --ignoreSignal, -I, and -d options are no longer supported.
     Do not use these options unless instructed to by IBM support personnel.

mmfsck command: Display an interim status report at any time

     While a long-running instance of mmfsck is in progress, you can start another instance
     of mmfsck with the --status-report parameter to display current
     status information from all the nodes that are participating in the mmfsck run.

mmgetstate command: Display the unresponsive state

     The command returns the unresponsive state when the GPFS
     daemon is running but is not responding.

mmhealth command: Addition to measurement options

     Measurement options for filesystem, SMB node, and NFS node
     has been added to the mmhealth command.

mmkeyserv command: The simplified method supports certificate chains from a certificate authority.

     In the simplified method, with the --kmip-cert parameter, you can set up encryption with IBM?
     Security Key Lifecycle Manager (SKLM) as the key management server and with a certificate signed
     by a certificate authority (CA) on the KMIP port of the SKLM server.

mmnetverify command: Enhancements

     - Verify the network operation of nodes in a subnet
         * With the --subnets parameters, you can specify the subnet
           addresses of the nodes that you want to verify.
     - Verify that nodes can handle a new MTU size
         * With the ping-packet-size parameter, you can specify the size
           of the ICMP echo request packets that are sent between local node and the target
           node during the ping test.

mmtracectl command: Display the tracing status

     The --status parameter displays the tracing status of the specified nodes.

New feature for threshold monitoring

      Starting from IBM Spectrum Scale version 5.0.0, if multiple thresholds rules
      have overlapping entities for the same metrics, only one of the concurrent rules is
      made actively eligible.

NFS: Dynamic export changes

      You can dynamically change the export configuration without restarting the NFS service.

Object

      - Support for Ubuntu
      - Support for sudo wrapper for Object on Ubuntu
      - Support for cumulative upgrades from older versions
      - Object snap enhancement to contain keystore logs

Protocol support: Enhanced

      Protocol support is extended to Ubuntu 10.04

Setting up a system for storing crash files for Ubuntu

      A topic is added to describe how to set up a system for storing crash files for Ubuntu.

SMB: DFS redirects for SMB shares

      New option to configure DFS redirects for SMB shares.

SMB: SMB server upgrade changes

      Two events on CTDB version match/mismatch are added to the RAS events.

Sudo wrappers: Root-level processes can call administration commands directly

      Root-level background processes, such as cron and callback programs, can
      successfully call administration commands directly rather than through sudo
      when sudo wrappers are enabled.

Supported clients for NFS

      A topic is added listing the clients that are supported by NFS protocol.

Transparent cloud tiering

      - Support for multiple cloud storage accounts
      - Support for multiple file systems or filesets per node group
      - Enhanced support for large file systems provided by container spillover
      - Support for associating file sets with containers for enhanced granularity
      - Multiple URL and region support at the node level
      - Support for creating a cloud service separately for tiering and sharing operations.
      - Unique encryption key per cloud container
      - Support for remotely mounted clients.
      - Support for Amazon S3 regions requiring Sigv4 security support,
        including the US government cloud region.
      - Ability to enable or disable transparent recall for files for a given file
        system instantly, without having to rewrite a policy.
      - Support for backing up and restoring the Cloud services configuration in case of any disaster.
      - Support for backing up the Cloud services database to the cloud.
      - Support for restoring Transparent cloud tiering service on an identical backup cluster.
      - Support for checking the integrity of the Cloud services
        database after any system crash or outage.
      - Support for auditing events relating to each operation performed in Cloud services.

New commands

      mmaudit mmmsgqueue

Changed commands

      mmafmctl mmadddisk mmcallhome mmchattr mmchcluster mmchconfig mmchfs mmchnsd
      mmcloudgateway mmcrcluster mmcrfs mmdeldisk mmdsh mmfsck mmgetstate
      mmkeyserv mmnetverify mmnfs mmrestripefile mmrestripefs mmsmb mmtracectl
      mmuserauth

Deleted commands

      mmrest

New messages

      6027-1264, 6027-1757, 6027-2394, 6027-2395, 6027-2396, 6027-2397
      6027-2398, 6027-2399, 6027-2400, 6027-2401, 6027-3259, 6027-3408
      6027-3597, 6027-3598, 6027-3599, 6027-3600, 6027-3601, 6027-3602
      6027-3603, 6027-3604, 6027-3730, 6027-3921, 6027-3922, 6027-3923
      6027-3924, 6027-3925, 6027-3926, 6027-3927, 6027-3928
      6027-3929, 6027-3930, 6027-3931, 6027-4019

Changed messages

      6027-928


     Open this item<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab-234019da9b-2D248a-2D49ec-2D9503-2De612becd953e&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=Q6E0fg8sor2mORS3ugkurXkuV2ZUDqRd_C6_jpYXVVM&e=>


Posting Date:

Monday, December 18, 2017 at 2:26:38 PM EST



To unsubscribe or change settings, please go to your developerWorks community Settings.<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_news&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=5C4z0hhuaas4Ghs1g4YfFTCE0eyNYJVmLPYrGPCuE7Q&e=>
This is a notification sent from developerWorks community.

The information contained in this e-mail message is intended only for the use of the named recipients and is subject to the IBM Online Privacy Statement<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ibm.com_privacy_details_us_en_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=7gE9uoypP_yHb1jMMSjdCN_xyJ7UU2Ifrx1TchwMkQA&e=>. If you have received this e-mail in error, please contact the sender to report the error and then delete this message from your system.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/9c66dc47/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 289 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/9c66dc47/attachment.png>

From Kevin.Buterbaugh at Vanderbilt.Edu  Mon Dec 18 19:51:56 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Mon, 18 Dec 2017 19:51:56 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
	<DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu>

Hi All,

Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up.

Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this.  Richards? suggestion below errors for me on the ?-g Administrator? part.  Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked.

No matter what I do, I cannot log in to the GUI.  Thanks for the suggestions, though?

Kevin

On Dec 7, 2017, at 6:10 AM, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:

Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error.

That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself.

[root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin
EFSSG0019I The user admin2 has been successfully created.
EFSSG1000I The command completed successfully.


Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 07 December 2017 11:57
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.
Get Outlook for Android<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fghei36&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=sPRr5u4lo%2BAwBPBQ2%2BdXw%2F2EWUAqy30Fk0UNssRFWHU%3D&reserved=0>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin


On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/8eab1462/attachment.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Mon Dec 18 20:06:30 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Mon, 18 Dec 2017 20:06:30 +0000
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
 Central
In-Reply-To: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
References: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
Message-ID: <662BADD6-27D5-4DE6-9EC8-5BE1E73C5873@vanderbilt.edu>

Hi All,

GPFS 5.0 was announced on Friday ? and today:

IBM Spectrum Scale
: IBM Spectrum Scale: NFS operations may fail with IO-Error<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ibm.com%2Fsupport%2Fdocview.wss%3Fuid%3Dssg1S1011791%26myns%3Ds033%26mynp%3DOCSTXKQY%26mynp%3DOCSWJ00%26mync%3DE%26cm_sp%3Ds033-_-OCSTXKQY-OCSWJ00-_-E&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C07b900d4a1aa4e92852e08d54634aa16%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492114479674351&sdata=i%2B8WSjOObZoaSpNLnEZLHq%2FqXwj%2BetYSJzojl9Rh%2FgQ%3D&reserved=0>

IBM has identified an issue with IBM Spectrum Scale 5.0.0.0 Protocol support for NFSv3/v4 in which IO-errors may be returned to the NFS client if the NFS server accumulates file-descriptor resources beyond the defined limit. Accumulation of file descriptor resources will occur when NFSv3 file create operations are sent against files that are already in use.
Bob?s suggestion in a previous e-mail to the list about installing this on a test cluster is almost certainly very, VERY good advice.  That?s certainly what we will do after the holiday break...

Kevin

On Dec 18, 2017, at 1:43 PM, Oesterlin, Robert <Robert.Oesterlin at nuance.com<mailto:Robert.Oesterlin at nuance.com>> wrote:

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Reply-To: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com<mailto:gpfs at us.ibm.com>' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum.

<image001.png> gpfs at us.ibm.com<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DYIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=4BVRC4gq9NXkxODASI9hJ5BV5dQQ94rUjzVc1fDndgs%3D&reserved=0> replied to the IBM Spectrum Scale V5.0 announcements<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3D05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=eCCEi0WsUIaKWOw1BHWueMPQ5O1LF6rYFZnHouwvZ8Y%3D&reserved=0> topic thread in the General Parallel File System - Announce (GPFS - Announce)<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0000-2D0000-2D000000001606%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DzTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=uxfVFydD9TI9l6%2BF%2F3rP7y%2F9Uo0NnBNlmr2tiMrEzbM%3D&reserved=0> forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www-2D933.ibm.com_support_fixcentral%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DiHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=cAwptvUoVH%2B57%2Fat%2FZgt3ft8Gv%2FLsNRWXQzv%2BiuAFuQ%3D&reserved=0>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
as updated, April 2017

Changes to this release of the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library include the following:

Added DMPs for TIP events

       A topic is added listing the directed maintenance procedures for TIP events.
       The DMPs help users resolve issues caused due to TIP events.

AFM and AFM DR

       - Compression and snapshot ILM policy supported.
       - A general recommendation added for the Gateway node.
       - Configuration parameters added - afmMaxParallelRecoveries, afmAsyncOpWaitTimeout,
           afmSyncOpWaitTimeout, and afmRevalOpWaitTimeout.
       - Configuration parameters modified - afmRPO and afmHashVersion.

Authentication: Authentication packages

       Updated the authentication page to include packages specific to Ubuntu.

Authentication: AD-based authentication

       New information is added on NFS with server-side group
       lookup and Active Directory authentication.

Authentication: Primary group selection configurable for AD + RFC2307 based authentication

       Ability to choose primary group as set in "UNIX attributes" of a user on
       Active Directory introduced with AD + RFC2307 based authentication scheme.
       Earlier, Windows primary group was by default selected as the primary group.

Big data and analytics
       - The GPFS Ambari integration package is now called the IBM Spectrum
         Scale Ambari management pack (in short, management pack or MPack).
       - IBM Spectrum Scale Ambari management pack version 2.4.2.1 with
         HDFS Transparency version 2.7.3.1 supports BI 4.2/BI 4.2.5 IOP migration
         to HDP 2.6.2.
       - Supports the remote mount configuration in Ambari.
       - Supports the multiple file systems configuration. In management
         pack version 2.4.2.1, the current limit is two file systems.
       - The Short circuit write is supported for better performance.
       - In HDFS Transparency, the Ranger performance is enhanced.

Changes to IBM Spectrum Scale management API

   Added the following new commands:

       GET /perfmon/data
       GET /filesystems/{filesystemName}/afm/state
       DELETE /nodes/{name}
       POST /nodes
       GET /nodeclasses
       POST /nodeclasses
       DELETE /nodeclasses/{nodeclassName}
       GET /nodeclasses/{nodeclassName}
       PUT /nodeclasses/{nodeclassName}
       DELETE /jobs/jobId
       POST /filesystems/{filesystemName}/filesets/{filesetName}/psnaps
       DELETE /filesystems/{filesystemName}/filesets/{filesetName}/psnaps/{snapshotName}
       GET /thresholds
       GET /thresholds/{name}
       POST /thresholds
       DELETE /thresholds/{name}

IBM Spectrum Scale GUI changes

   - Added new Networks page to monitor the performance, configuration, and
      adapters of network configurations in the cluster. You can monitor the
      network performance with respect to the IP and RDMA interfaces used in the configuration.
    - Added new Monitoring > Thresholds page to create and monitor the threshold rules
      that are defined in the system.
    - Added Access > Remote Connections page to enable the GUI node of the
      local cluster to monitor the remote cluster by establishing a connection
      with the GUI node of the remote cluster.
    - Added Settings > Call Home page to configure call home. Configuring
      the call home feature helps the IBM? Support to monitor the system.
      Configuring call home also helps to reduce the response time of the
      IBM Support to resolve any issues.
      The diagnostic data that is downloaded through the Settings > Diagnostic Data
      can be uploaded to a problem management record (PMR) by using
      the call home feature in the backend. To upload the diagnostic data, right-click
      the relevant data set from the Previously Collected Diagnostic Data, and
      select Upload to PMR.
    - Added file system creation capabilities in GUI. Use the Files > File Systems > Create File System
      option to launch the Create File System wizard. In the Create File System wizard, you can specify
      the following details of the file system:
         - File system name
         - Storage pools
         - NSDs for the file systems
         - Failure groups
         - NSD order for data writes
         - Maximum number of Spectrum Scale clients
         - Maximum number of inodes of the root fileset
         - Whether to enable quota and scope for the quota definition
         - Whether to enable DMAPI
         - Mount point and automatic mount mode
    - Added the aggregation levels Access Point and Filesets and removed Account
      for the resource type Transparent Cloud Tiering in the Monitoring > Statisitics page.
    - The Files > Transparent Cloud Tiering page now displays the file systems
      and filesets that are mapped with the cloud service. It also shows the
      connection of such a container pair configuration to a cloud account and
      the corresponding CSAP that is configured for the cloud account.
    - Changes to capacity monitoring in the GUI
         - Moved the Capacity page from Monitoring to Files menu in the navigation
           and renamed the GUI page to User Capacity.
         - Only the file data user capacity can be viewed from the Files > User Capacity page.
           Removed the pools, filesets, file system capacity monitoring options from the Files > User Capacity page.
           You can monitor the capacity of these components from the respective GUI pages.
         - Replaced the GPFSPoolCap sensor with the GPFSPool sensor and separate data and metadata
           level capacity monitoring are introduced in the performance charts available in the

           Files > File Systems and Storage > Pools pages.
         - New GPFSPool-based data and metadata performance monitoring metrics are available
           for selection in the Files > Statistics > Edit > Capacity section.
           You need to select the aggregation level as Pool to view these metrics.
    - AFM monitoring changes in the GUI
         - Provides the number of AFM filesets and the corresponding export server maps.
           Each export map establishes a mapping between the gateway node and the NFS
           host name to allow parallel data transfers from cache to home.
         - By using the Request Access option available in the Files > Active File Management
           or Access > Remote Connection page in the GUI, you can now establish connection with remote clusters.
           After establishing the connection, you can monitor the following AFM and AFM DR
           configuration details across clusters:
              * On home and secondary, you can see the AFM relationships configuration,
                health status, and performance values of the Cache and Disaster Recovery grids.
              * On the Overview tab of the detailed view, the available home and secondary inodes are available.
              * On the Overview tab of the detailed view, the details such as NFS throughput,
                IOPs, and latency details are available, if the protocol is NFS.
    - New option to create AFM peer snapshots through GUI. Use the Create Peer Snapshot option
      in the Files > Snapshots page to create peer snapshots. You can view and delete these peer snapshots from
      the Snapshots page and also from the detailed view of the Files > Active File Management page.

Encryption: GSKit V8 improves cryptographic performance on IBM POWER8

   The IBM Global Security Kit (GSKit) Version 8 and later improves cryptographic performance
   on IBM POWER8 hardware. The version of GSKit that is shipped with IBM Spectrum Scale v5.0.0
   offers better performance on POWER8, compared with the versions shipped with earlier releases.

File compression: The lz4 library provides fast access to compressed data

   File compression supports the lz4 compression library. Lz4 is intended primarily for active data and
   favors read-access speed over maximized space saving.

File data: Block and subblock sizes improve I/O performance and reduce fragmentation

   The default block size is larger, 4 MiB instead of 256 KiB, and the sizes of subblocks relative to blocks
   s are smaller, for example, 8 KiB subblocks in a 4 MiB block. A larger block size improves the
   file system performance and a smaller subblock size reduces the amount of unused space. For many business
   applications, the default value of 4 MiB provides the best balance of improved performance
   and reduced fragmentation.

File encryption: AES-XTS encryption is faster on x86 in non-FIPS mode

    On x86 architecture in non-FIPS mode, file encryption with the AES algorithm in XTS mode
    is faster than it was.

File systems: File system rebalancing is faster

    Rebalancing is implemented by a lenient round-robin method that typically runs faster than the
    previously used method of strict round robin. The strict round robin method is available as an option.

installation toolkit changes

    - The installation toolkit has added support for the installation and the deployment
      of IBM Spectrum Scale in a cluster containing Elastic Storage Server (ESS).
    - The installation toolkit has added support for enabling and configuring call home.
    - The installation toolkit has added support for enabling and configuring file audit logging.
    - The installation toolkit has added support for the installation and the
      deployment of IBM Spectrum Scale on Ubuntu 16.04 LTS nodes.
    - The installation toolkit has added verification of passwordless SSH during
      prechecks before installation, deployment, or upgrade.
    - The installation toolkit has added support for cumulative object upgrade.

mmafmctl command

    The --outband parameter is deprecated.

mmcallhome command: Enhancements

    - Addition of -Y option
        * The -Y displays the command output in a parseable format with a colon (:) as a field delimiter.
    - Addition of --pmr option
        * The --pmr option allows you to upload data to existing PMRs using the
          mmcallhome run SendFile command.

mmchconfig command: Enhancements

    - Encrypted files can be copied into an LROC device
        * With the lrocEnableStoringClearText attribute, you can control whether file
          data from encrypted files, which is held in memory as cleartext, is
          copied into a local read-only cache (LROC) device.
    - InfiniBand addresses can be specified for RDMA transfers
        * In the verbsPorts attribute, you can specify InfiniBand addresses
          for RDMA transfers between an NSD client and server.

mmchnsd command: Change NSDs without unmounting the file system

    When you add or remove NSDs or do other operations with mmchnsd,
    you do not need to unmount the file system.

mmcrfs command: Enhancements

     - The default data block size is 4 MiB with an 8 KiB subblock size
         * If no block size is specified, a file system is created with a 4 MiB block size and an
           8 KiB subblock size. The minimum release level (minReleaseLevel) of the
           cluster must be 5.0.0 or greater when the file system is created.
     - The default log file size depends on block size and metadata size
         * If the block size is 512 KiB or larger and the metadata block size 256 KiB or larger,
           then the default log file size is 32 MiB. Otherwise, the default log file
           size is 4 MiB or the metadata block size, whichever is larger.
     - The default method for updating atime is relatime
         * If the minimum release level (minReleaseLevel) of the cluster is 5.0.0 or greater
           when the file system is created, the default method for updating atime is relatime.

mmdsh command: Several options are no longer supported

     The --ignoreSignal, -I, and -d options are no longer supported.
     Do not use these options unless instructed to by IBM support personnel.

mmfsck command: Display an interim status report at any time

     While a long-running instance of mmfsck is in progress, you can start another instance
     of mmfsck with the --status-report parameter to display current
     status information from all the nodes that are participating in the mmfsck run.

mmgetstate command: Display the unresponsive state

     The command returns the unresponsive state when the GPFS
     daemon is running but is not responding.

mmhealth command: Addition to measurement options

     Measurement options for filesystem, SMB node, and NFS node
     has been added to the mmhealth command.

mmkeyserv command: The simplified method supports certificate chains from a certificate authority.

     In the simplified method, with the --kmip-cert parameter, you can set up encryption with IBM?
     Security Key Lifecycle Manager (SKLM) as the key management server and with a certificate signed
     by a certificate authority (CA) on the KMIP port of the SKLM server.

mmnetverify command: Enhancements

     - Verify the network operation of nodes in a subnet
         * With the --subnets parameters, you can specify the subnet
           addresses of the nodes that you want to verify.
     - Verify that nodes can handle a new MTU size
         * With the ping-packet-size parameter, you can specify the size
           of the ICMP echo request packets that are sent between local node and the target
           node during the ping test.

mmtracectl command: Display the tracing status

     The --status parameter displays the tracing status of the specified nodes.

New feature for threshold monitoring

      Starting from IBM Spectrum Scale version 5.0.0, if multiple thresholds rules
      have overlapping entities for the same metrics, only one of the concurrent rules is
      made actively eligible.

NFS: Dynamic export changes

      You can dynamically change the export configuration without restarting the NFS service.

Object

      - Support for Ubuntu
      - Support for sudo wrapper for Object on Ubuntu
      - Support for cumulative upgrades from older versions
      - Object snap enhancement to contain keystore logs

Protocol support: Enhanced

      Protocol support is extended to Ubuntu 10.04

Setting up a system for storing crash files for Ubuntu

      A topic is added to describe how to set up a system for storing crash files for Ubuntu.

SMB: DFS redirects for SMB shares

      New option to configure DFS redirects for SMB shares.

SMB: SMB server upgrade changes

      Two events on CTDB version match/mismatch are added to the RAS events.

Sudo wrappers: Root-level processes can call administration commands directly

      Root-level background processes, such as cron and callback programs, can
      successfully call administration commands directly rather than through sudo
      when sudo wrappers are enabled.

Supported clients for NFS

      A topic is added listing the clients that are supported by NFS protocol.

Transparent cloud tiering

      - Support for multiple cloud storage accounts
      - Support for multiple file systems or filesets per node group
      - Enhanced support for large file systems provided by container spillover
      - Support for associating file sets with containers for enhanced granularity
      - Multiple URL and region support at the node level
      - Support for creating a cloud service separately for tiering and sharing operations.
      - Unique encryption key per cloud container
      - Support for remotely mounted clients.
      - Support for Amazon S3 regions requiring Sigv4 security support,
        including the US government cloud region.
      - Ability to enable or disable transparent recall for files for a given file
        system instantly, without having to rewrite a policy.
      - Support for backing up and restoring the Cloud services configuration in case of any disaster.
      - Support for backing up the Cloud services database to the cloud.
      - Support for restoring Transparent cloud tiering service on an identical backup cluster.
      - Support for checking the integrity of the Cloud services
        database after any system crash or outage.
      - Support for auditing events relating to each operation performed in Cloud services.

New commands

      mmaudit mmmsgqueue

Changed commands

      mmafmctl mmadddisk mmcallhome mmchattr mmchcluster mmchconfig mmchfs mmchnsd
      mmcloudgateway mmcrcluster mmcrfs mmdeldisk mmdsh mmfsck mmgetstate
      mmkeyserv mmnetverify mmnfs mmrestripefile mmrestripefs mmsmb mmtracectl
      mmuserauth

Deleted commands

      mmrest

New messages

      6027-1264, 6027-1757, 6027-2394, 6027-2395, 6027-2396, 6027-2397
      6027-2398, 6027-2399, 6027-2400, 6027-2401, 6027-3259, 6027-3408
      6027-3597, 6027-3598, 6027-3599, 6027-3600, 6027-3601, 6027-3602
      6027-3603, 6027-3604, 6027-3730, 6027-3921, 6027-3922, 6027-3923
      6027-3924, 6027-3925, 6027-3926, 6027-3927, 6027-3928
      6027-3929, 6027-3930, 6027-3931, 6027-4019

Changed messages

      6027-928



     Open this item<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab-234019da9b-2D248a-2D49ec-2D9503-2De612becd953e%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DQ6E0fg8sor2mORS3ugkurXkuV2ZUDqRd_C6_jpYXVVM%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=wQawdxgurt9CU5mLHBaUeqAshZxOd7tn7HrOeP4CZR8%3D&reserved=0>


Posting Date:

Monday, December 18, 2017 at 2:26:38 PM EST



To unsubscribe or change settings, please go to your developerWorks community Settings.<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_news%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3D5C4z0hhuaas4Ghs1g4YfFTCE0eyNYJVmLPYrGPCuE7Q%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=KMU2UypqoZXXJlTopsMbsV3x0RUuNMJXEv1UyZEQoRY%3D&reserved=0>

This is a notification sent from developerWorks community.

The information contained in this e-mail message is intended only for the use of the named recipients and is subject to the IBM Online Privacy Statement<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__www.ibm.com_privacy_details_us_en_%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3D7gE9uoypP_yHb1jMMSjdCN_xyJ7UU2Ifrx1TchwMkQA%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=NzaHT12VmX5MY%2FRIm3m4ovKIYNz%2FGMTnJj1%2FM3cOM7I%3D&reserved=0>. If you have received this e-mail in error, please contact the sender to report the error and then delete this message from your system.


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=pHmKmOoSbQ5W4e2DiOfphjWgRdqT8D9NMBjtYZcH%2Bog%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/fd0d1cca/attachment.htm>

From valdis.kletnieks at vt.edu  Mon Dec 18 20:08:48 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Mon, 18 Dec 2017 15:08:48 -0500
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID: <32207.1513627728@turing-police.cc.vt.edu>

Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux

that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4."

We don't use AFM or the install toolkit.  But we *do* make fairly heavy use
of mmces and nfs-ganesha - is that what they mean by "Integrated Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are doing
their upgrades - and going to 7.4 would be nice.  If there's a mine field there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)


From Kevin.Buterbaugh at Vanderbilt.Edu  Mon Dec 18 20:11:28 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Mon, 18 Dec 2017 20:11:28 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
	<DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu>
	<1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov>
Message-ID: <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu>

Hi Jesse,

Thanks for the suggestion ? I find the following error very interesting:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/rmuser admin
EFSSP0010C CLI parser: The object "admin" specified for "userID" does not exist.
/root
root at testnsd1#

That says to me that I don?t have an admin user, which - if true - would explain why not a single password I can think of works.  ;-)

But as I mentioned in my original post I had this up and working earlier this fall.  While I can?t prove anything, I can?t imagine a scenario where I would deliberately choose a non-default username.  So if ?admin? has been the default login for the GPFS GUI all along then I am really mystified.

Thanks!

Kevin

On Dec 18, 2017, at 1:58 PM, Hanley, Jesse A. <hanleyja at ornl.gov<mailto:hanleyja at ornl.gov>> wrote:

Kevin,

I ran into this a couple times using 4.2.3.  This is what we used to get around it:

/usr/lpp/mmfs/gui/cli/rmuser admin
/usr/lpp/mmfs/gui/cli/mkuser admin -p <password> -g Administrator,SecurityAdmin

You may need to run the initgui command if those objects are present.  That typically gets run on first login to the GUI.

Thanks,
--
Jesse


From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Monday, December 18, 2017 at 2:52 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

Hi All,

Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up.

Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this.  Richards? suggestion below errors for me on the ?-g Administrator? part.  Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked.

No matter what I do, I cannot log in to the GUI.  Thanks for the suggestions, though?

Kevin


On Dec 7, 2017, at 6:10 AM, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:

Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error.

That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself.

[root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin
EFSSG0019I The user admin2 has been successfully created.
EFSSG1000I The command completed successfully.


Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 07 December 2017 11:57
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.
Get Outlook for Android<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fghei36&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=sPRr5u4lo%2BAwBPBQ2%2BdXw%2F2EWUAqy30Fk0UNssRFWHU%3D&reserved=0>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin



On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C3f1a258c20a240166fbe08d54651c592%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492239506213881&sdata=qGxMvy3f6cfKQEmwH%2Bcc0SKVDAcaJ4Z66p%2BE9mWDwHg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/656e148e/attachment.htm>

From aaron.s.knister at nasa.gov  Mon Dec 18 20:13:01 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Mon, 18 Dec 2017 20:13:01 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>,
	<CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
Message-ID: <6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>

Thanks Sven! That makes sense to me and is what I thought was the case which is why I was confused when I saw the reply to the thread that said the >32 subblocks code had no performance impact.

A couple more question for you? in your presentation there?s a benchmark that shows the file create performance without the zero padding. Since you mention this is done for security reasons was that feature ever promoted to a GA Scale release? I?m also wondering if you could explain the performance difference between the no zero padding code and the > 32 subblock code since given your the example of 32K files and 16MB block size I figure both cases ought to write the same amount to disk.

Thanks!

-Aaron





On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com> wrote:
i thought i answered that already, but maybe i just thought about answering it and then forgot about it :-D

so yes more than 32 subblocks per block significant increase the performance of filesystems with small files, for the sake of the argument let's say 32k in a large block filesystem again for sake of argument say 16MB.

you probably ask why ?

if you create a file and write 32k into it in a pre 5.0.0 Version 16 MB filesystem your client actually doesn't write 32k to the NSD Server, it writes 512k, because thats the subblock size and we need to write the full subblock (for security reasons). so first you waste significant memory on the client to cache that zero padding, you waste network bandwidth and you waste NSD Server cache because you store it there too. this means you overrun the cache more quickly, means you start doing read/modify writes earlier on all your nice large raid tracks... i guess you get the story by now.

in fact,  if you have a good raid code that can drive really a lot of bandwidth out of individual drives like a GNR system you get more performance for small file writes as larger your blocksize is, because we can 'pack' more files into larger i/os and therefore turn a small file create workload into a bandwidth workload, essentially exactly what we did and i demonstrated in the CORAL presentation .

hope that makes this crystal clear now .

sven



On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>> wrote:
Thanks, Alex. I'm all too familiar with the trade offs between large
blocks and small files and we do use pretty robust SSD storage for our
metadata. We support a wide range of workloads and we have some folks
with many small (<1M) files and other folks with many large (>256MB) files.

My point in this thread is that IBM has said over and over again in
presentations that there is a significant performance gain with the >32
subblocks code on filesystems with large block sizes (although to your
point I'm not clear on exactly what large means since I didn't define
large in this context). Therefore given that the >32 subblock code gives
a significant performance gain one could reasonably assume that having a
filesystem with >32 subblocks is required to see this gain (rather than
just running the >32 subblocks code on an fs w/o > 32 subblocks).

This lead me to ask about a migration tool because in my mind if there's
a performance gain from having >32 subblocks on the FS I'd like that
feature and having to manually copy 10's of PB to new hardware to get
this performance boost is unacceptable. However, IBM can't seem to make
up their mind about whether or not the >32 subblocks code *actually*
provides a performance increase. This seems like a pretty
straightforward question.

-Aaron

On 12/15/17 3:48 PM, Alex Chekholko wrote:
> Hey Aaron,
>
> Can you define your sizes for "large blocks" and "small files"?  If you
> dial one up and the other down, your performance will be worse.  And in
> any case it's a pathological corner case so it shouldn't matter much for
> your workflow, unless you've designed your system with the wrong values.
>
> For example, for bioinformatics workloads, I prefer to use 256KB
> filesystem block size, and I'd consider 4MB+ to be "large block size",
> which would make the filesystem obviously unsuitable for processing
> millions of 8KB files.
>
> You can make a histogram of file sizes in your existing filesystems and
> then make your subblock size (1/32 of block size) on the smaller end of
> that.   Also definitely use the "small file in inode" feature and put
> your metadata on SSD.
>
> Regards,
> Alex
>
> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov> <mailto:aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>>> wrote:
>
>     Thanks, Bill.
>
>     I still don't feel like I've got an clear answer from IBM and frankly
>     the core issue of a lack of migration tool was totally dodged.
>
>     Again in Sven's presentation from SSUG @ SC17
>     (http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
>     he mentions "It has a significant performance penalty for small files in
>     large block size filesystems" and the demonstrates that with several
>     mdtest runs (which show the effect with and without the >32
>     subblocks code):
>
>
>     4.2.1 base code - SUMMARY: (of 3 iterations)
>     File creation : Mean = 2237.644
>
>     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY: (of
>     3 iterations)
>     File creation : Mean = 12866.842
>
>     more sub blocks per block (4.2.2 + morethan32subblock code):
>     File creation : Mean = 40316.721
>
>     Can someone (ideally Sven) give me a straight answer as to whether or
>     not the > 32 subblock code actually makes a performance difference for
>     small files in large block filesystems? And if not, help me understand
>     why his slides and provided benchmark data have consistently indicated
>     it does?
>
>     -Aaron
>
>     On 12/1/17 11:44 AM, Bill Hartner wrote:
>     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
>     > results @ user group meeting.
>     >
>     > -Bill
>     >
>     > Bill Hartner
>     > IBM Systems
>     > Scalable I/O Development
>     > Austin, Texas
>     > bhartner at us.ibm.com<mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>>
>     > home office 512-784-0980<tel:(512)%20784-0980> <tel:512-784-0980<tel:(512)%20784-0980>>
>     >
>     >
>     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
>     > AM---Bill, could you say something about what the metadataJan-Frode
>     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
>     > about what the metadata-storage here was? ESS/NL-SAS/3way replication?
>     >
>     > From: Jan-Frode Myklebust <janfrode at tanso.net<mailto:janfrode at tanso.net> <mailto:janfrode at tanso.net<mailto:janfrode at tanso.net>>>
>     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
>     <mailto:gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>>
>     > Date: 12/01/2017 06:53 AM
>     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     > Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>     >
>     >
>     ------------------------------------------------------------------------
>     >
>     >
>     >
>     > Bill, could you say something about what the metadata-storage here was?
>     > ESS/NL-SAS/3way replication?
>     >
>     > I just asked about this in the internal slack channel #scale-help today..
>     >
>     >
>     >
>     > -jf
>     >
>     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
>     > <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>>>>:
>     >
>     >     > "It has a significant performance penalty for small files in
>     large
>     >     > block size filesystems"
>     >
>     >     Aaron,
>     >
>     >     Below are mdtest results for a test we ran for CORAL - file
>     size was
>     >     32k.
>     >
>     >     We have not gone back and ran the test on a file system formatted
>     >     without > 32 subblocks. We'll do that at some point...
>     >
>     >     -Bill
>     >
>     >     -- started at 10/28/2017 17:51:38 --
>     >
>     >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>     >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>     >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
>     -p 360
>     >     -u -y
>     >     Path: /ibm/fs2-16m-10
>     >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>     >
>     >     228 tasks, 67239936 files
>     >
>     >     SUMMARY: (of 3 iterations)
>     >     Operation Max Min Mean Std Dev
>     >     --------- --- --- ---- -------
>     >     File creation : 51953.498 50558.517 51423.221 616.643
>     >     File stat : 0.000 0.000 0.000 0.000
>     >     File read : 0.000 0.000 0.000 0.000
>     >     File removal : 96746.376 92149.535 94658.774 1900.187
>     >     Tree creation : 1.588 0.070 0.599 0.700
>     >     Tree removal : 0.213 0.034 0.097 0.082
>     >
>     >     -- finished at 10/28/2017 19:51:54 --
>     >
>     >     Bill Hartner
>     >     IBM Systems
>     >     Scalable I/O Development
>     >     Austin, Texas_
>     >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>
>     <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>>>
>     >     home office 512-784-0980<tel:(512)%20784-0980> <tel:512-784-0980<tel:(512)%20784-0980>>
>     >
>     >     _
>     >     __gpfsug-discuss-bounces at spectrumscale.org_
>     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>> wrote on
>     >     11/29/2017 04:41:48 PM:
>     >
>     >     > From: Aaron Knister <_aaron.knister at gmail.com_
>     >     <mailto:aaron.knister at gmail.com<mailto:aaron.knister at gmail.com> <mailto:aaron.knister at gmail.com<mailto:aaron.knister at gmail.com>>>>
>     >
>     >
>     >     > To: gpfsug main discussion list
>     >     <_gpfsug-discuss at spectrumscale.org_
>     >     <mailto:gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
>     <mailto:gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>>>
>     >
>     >     > Date: 11/29/2017 04:42 PM
>     >
>     >
>     >     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>>
>     >
>     >     >
>     >
>     >     > Thanks, Nikhil. Most of that was consistent with my understnading,
>     >     > however I was under the impression that the >32 subblocks code is
>     >     > required to achieve the touted 50k file creates/second that Sven has
>     >     > talked about a bunch of times:
>     >     >
>     >     >
>     >
>      _http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>     <http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>     >     >
>     >
>      _http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
>     <http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>     >     > _Sven_Oehme_-_News_from_Research.pdf
>     >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>     >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>     >
>     >
>     >     > from those presentations regarding 32 subblocks:
>     >     >
>     >     > "It has a significant performance penalty for small files in large
>     >     > block size filesystems"
>     >
>     >     > although I'm not clear on the specific definition of "large". Many
>     >     > filesystems I encounter only have a 1M block size so it may not
>     >     > matter there, although that same presentation clearly shows the
>     >     > benefit of larger block sizes which is yet *another* thing for which
>     >     > a migration tool would be helpful.
>     >
>     >     > -Aaron
>     >     >
>     >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>     >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com<mailto:nikhilk at us.ibm.com>
>     <mailto:nikhilk at us.ibm.com<mailto:nikhilk at us.ibm.com>>>> wrote:
>     >
>     >     > Hi,
>     >     >
>     >     > I would like to clarify migration path to 5.0.0 from 4.X.X
>     clusters.
>     >     > For all Spectrum Scale clusters that are currently at 4.X.X,
>     it is
>     >     > possible to migrate to 5.0.0 with no offline data migration
>     and no
>     >     > need to move data. Once these clusters are at 5.0.0, they will
>     >     > benefit from the performance improvements, new features (such as
>     >     > file audit logging), and various enhancements that are
>     included in
>     >     5.0.0.
>     >     >
>     >     > That being said, there is one enhancement that will not be
>     applied
>     >     > to these clusters, and that is the increased number of
>     sub-blocks
>     >     > per block for small file allocation. This means that for file
>     >     > systems with a large block size and a lot of small files, the
>     >     > overall space utilization will be the same it currently is
>     in 4.X.X.
>     >     > Since file systems created at 4.X.X and earlier used a block
>     size
>     >     > that kept this allocation in mind, there should be very little
>     >     > impact on existing file systems.
>     >     >
>     >     > Outside of that one particular function, the remainder of the
>     >     > performance improvements, metadata improvements, updated
>     >     > compatibility, new functionality, and all of the other
>     enhancements
>     >     > will be immediately available to you once you complete the
>     upgrade
>     >     > to 5.0.0 -- with no need to reformat, move data, or take
>     your data
>     >     offline.
>     >     >
>     >     > I hope that clarifies things a little and makes the upgrade path
>     >     > more accessible.
>     >     >
>     >     > Please let me know if there are any other questions or concerns.
>     >     >
>     >     > Thank you,
>     >     > Nikhil Khandelwal
>     >     > Spectrum Scale Development
>     >     > Client Adoption
>     >     >
>     >     > _______________________________________________
>     >     > gpfsug-discuss mailing list
>     >     > gpfsug-discuss at _spectrumscale.org_
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>     >
>     >     > _______________________________________________
>     >     > gpfsug-discuss mailing list
>     >     > gpfsug-discuss at _spectrumscale.org_
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >
>     >     > _https://urldefense.proofpoint.com/v2/url?_
>     <https://urldefense.proofpoint.com/v2/url?_>
>     >     >
>     >     u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>     >     >
>     >     siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>     >
>     >     _______________________________________________
>     >     gpfsug-discuss mailing list
>     >     gpfsug-discuss at _spectrumscale.org_
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>     >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>     >     gpfsug-discuss mailing list
>     >     gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> <http://spectrumscale.org>
>     >
>      https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> <http://spectrumscale.org>
>     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>     >
>
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776<tel:(301)%20286-2776> <tel:%28301%29%20286-2776>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776<tel:(301)%20286-2776>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/62da1e91/attachment.htm>

From taylorm at us.ibm.com  Mon Dec 18 20:27:42 2017
From: taylorm at us.ibm.com (Michael L Taylor)
Date: Mon, 18 Dec 2017 13:27:42 -0700
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
	Central
In-Reply-To: <mailman.3372.1513626221.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3372.1513626221.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <OFD183EA98.E6691217-ON002581FA.006FA7A4-072581FA.00706687@notes.na.collabserv.com>



Hi Bob,
Thanks for the note on 5.0.0
One correction however.... clusters can do a rolling upgrade to 5.0.0 from
any 4.2.x level (not just 4.2.3).




Today's Topics:

   1. FW: Spectrum Scale 5.0 now available on Fix Central
      (Oesterlin, Robert)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 19:43:35 +0000
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
		 Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53 at nuance.com>
Content-Type: text/plain; charset="utf-8"

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to
this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum
Scale V5.0 announcements' topic thread in the 'General Parallel File System
- Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=
> replied to the IBM Spectrum Scale V5.0 announcements<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=
> topic thread in the General Parallel File System - Announce (GPFS -
Announce)<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0


000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=>
 forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=
>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/157250d2/attachment.htm>

From stockf at us.ibm.com  Mon Dec 18 21:10:55 2017
From: stockf at us.ibm.com (Frederick Stock)
Date: Mon, 18 Dec 2017 16:10:55 -0500
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
In-Reply-To: <32207.1513627728@turing-police.cc.vt.edu>
References: <32207.1513627728@turing-police.cc.vt.edu>
Message-ID: <OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF@notes.na.collabserv.com>

Yes the integrated protocols are the Samba and Ganesha that are bundled 
with Spectrum Scale.  These require the use of the CES component for 
monitoring the protocols.  If you do use them then you need to wait for a 
release of Spectrum Scale in which the integrated protocols are also 
supported on RHEL 7.4.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com



From:   valdis.kletnieks at vt.edu
To:     gpfsug-discuss at spectrumscale.org
Date:   12/18/2017 03:09 PM
Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux


that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on 
RHEL 7.4."

We don't use AFM or the install toolkit.  But we *do* make fairly heavy 
use
of mmces and nfs-ganesha - is that what they mean by "Integrated 
Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are 
doing
their upgrades - and going to 7.4 would be nice.  If there's a mine field 
there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e=






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/7c272339/attachment.htm>

From erich at uw.edu  Mon Dec 18 21:46:02 2017
From: erich at uw.edu (Eric Horst)
Date: Mon, 18 Dec 2017 21:46:02 +0000
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
In-Reply-To: <OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF@notes.na.collabserv.com>
References: <32207.1513627728@turing-police.cc.vt.edu>
	<OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF@notes.na.collabserv.com>
Message-ID: <CAF-To-3w7FGHf_ks8cv4WZ=ZF42ZHVOZo4rbu2K8VTQtgAQUWg@mail.gmail.com>

Grr, this might explain why I experienced unhappiness when I tried to start
my long-delayed AFM based migration over the weekend. I had previously
tested AFM and found everything working, but 7.4 may have slipped in last
month. The AFM relationship seems to work but `mmafmctl premigrate`
commands fail. I would revert packages if I could figure out where the
issue lies.

-Eric

On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock <stockf at us.ibm.com> wrote:

> Yes the integrated protocols are the Samba and Ganesha that are bundled
> with Spectrum Scale.  These require the use of the CES component for
> monitoring the protocols.  If you do use them then you need to wait for a
> release of Spectrum Scale in which the integrated protocols are also
> supported on RHEL 7.4.
>
> Fred
> __________________________________________________
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821>
> stockf at us.ibm.com
>
>
>
> From:        valdis.kletnieks at vt.edu
> To:        gpfsug-discuss at spectrumscale.org
> Date:        12/18/2017 03:09 PM
> Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Currently, the IBM support matrix says:
>
> https://www.ibm.com/support/knowledgecenter/STXKQY/
> gpfsclustersfaq.html#linux
>
> that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:
>
> "AFM, Integrated Protocols, and Installation Toolkit are not supported on
> RHEL 7.4."
>
> We don't use AFM or the install toolkit.  But we *do* make fairly heavy use
> of mmces and nfs-ganesha - is that what they mean by "Integrated
> Protocols"?
>
> (We're looking at doing upgrades next month while our HPC clusters are
> doing
> their upgrades - and going to 7.4 would be nice.  If there's a mine field
> there, I need to
> make sure we stay at 7.3 - plus applicable non-7.4 updates)
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_
> iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=
> 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_
> I4NKvn8aFGXCqcJARCbu4AOE&e=
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/c01cc906/attachment.htm>

From a.khiredine at meteo.dz  Tue Dec 19 08:08:03 2017
From: a.khiredine at meteo.dz (atmane khiredine)
Date: Tue, 19 Dec 2017 08:08:03 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35
In-Reply-To: <mailman.3382.1513633600.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3382.1513633600.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz>

IBM Spectrum Scale V5.0 not support RHEL 6.x

Only RHEL 7.1 or later

https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux


Atmane Khiredine
HPC System Administrator | Office National de la M?t?orologie
T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz

________________________________________
De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org]
Envoy? : lundi 18 d?cembre 2017 22:46
? : gpfsug-discuss at spectrumscale.org
Objet : gpfsug-discuss Digest, Vol 71, Issue 35

Send gpfsug-discuss mailing list submissions to
        gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
        gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
        gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Re: FW: Spectrum Scale 5.0 now available on Fix   Central
      (Michael L Taylor)
   2. Re: gpfs 4.2.3.5 and RHEL 7.4... (Frederick Stock)
   3. Re: gpfs 4.2.3.5 and RHEL 7.4... (Eric Horst)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 13:27:42 -0700
From: "Michael L Taylor" <taylorm at us.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on
        Fix     Central
Message-ID:
        <OFD183EA98.E6691217-ON002581FA.006FA7A4-072581FA.00706687 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"



Hi Bob,
Thanks for the note on 5.0.0
One correction however.... clusters can do a rolling upgrade to 5.0.0 from
any 4.2.x level (not just 4.2.3).




Today's Topics:

   1. FW: Spectrum Scale 5.0 now available on Fix Central
      (Oesterlin, Robert)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 19:43:35 +0000
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
                 Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53 at nuance.com>
Content-Type: text/plain; charset="utf-8"

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to
this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum
Scale V5.0 announcements' topic thread in the 'General Parallel File System
- Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=
> replied to the IBM Spectrum Scale V5.0 announcements<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=
> topic thread in the General Parallel File System - Announce (GPFS -
Announce)<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0


000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=>
 forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=
>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/157250d2/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 18 Dec 2017 16:10:55 -0500
From: "Frederick Stock" <stockf at us.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
        <OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

Yes the integrated protocols are the Samba and Ganesha that are bundled
with Spectrum Scale.  These require the use of the CES component for
monitoring the protocols.  If you do use them then you need to wait for a
release of Spectrum Scale in which the integrated protocols are also
supported on RHEL 7.4.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com



From:   valdis.kletnieks at vt.edu
To:     gpfsug-discuss at spectrumscale.org
Date:   12/18/2017 03:09 PM
Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux


that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on
RHEL 7.4."

We don't use AFM or the install toolkit.  But we *do* make fairly heavy
use
of mmces and nfs-ganesha - is that what they mean by "Integrated
Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are
doing
their upgrades - and going to 7.4 would be nice.  If there's a mine field
there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e=






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/7c272339/attachment-0001.html>

------------------------------

Message: 3
Date: Mon, 18 Dec 2017 21:46:02 +0000
From: Eric Horst <erich at uw.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
        <CAF-To-3w7FGHf_ks8cv4WZ=ZF42ZHVOZo4rbu2K8VTQtgAQUWg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Grr, this might explain why I experienced unhappiness when I tried to start
my long-delayed AFM based migration over the weekend. I had previously
tested AFM and found everything working, but 7.4 may have slipped in last
month. The AFM relationship seems to work but `mmafmctl premigrate`
commands fail. I would revert packages if I could figure out where the
issue lies.

-Eric

On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock <stockf at us.ibm.com> wrote:

> Yes the integrated protocols are the Samba and Ganesha that are bundled
> with Spectrum Scale.  These require the use of the CES component for
> monitoring the protocols.  If you do use them then you need to wait for a
> release of Spectrum Scale in which the integrated protocols are also
> supported on RHEL 7.4.
>
> Fred
> __________________________________________________
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821>
> stockf at us.ibm.com
>
>
>
> From:        valdis.kletnieks at vt.edu
> To:        gpfsug-discuss at spectrumscale.org
> Date:        12/18/2017 03:09 PM
> Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Currently, the IBM support matrix says:
>
> https://www.ibm.com/support/knowledgecenter/STXKQY/
> gpfsclustersfaq.html#linux
>
> that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:
>
> "AFM, Integrated Protocols, and Installation Toolkit are not supported on
> RHEL 7.4."
>
> We don't use AFM or the install toolkit.  But we *do* make fairly heavy use
> of mmces and nfs-ganesha - is that what they mean by "Integrated
> Protocols"?
>
> (We're looking at doing upgrades next month while our HPC clusters are
> doing
> their upgrades - and going to 7.4 would be nice.  If there's a mine field
> there, I need to
> make sure we stay at 7.3 - plus applicable non-7.4 updates)
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_
> iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=
> 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_
> I4NKvn8aFGXCqcJARCbu4AOE&e=
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/c01cc906/attachment.html>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 71, Issue 35
**********************************************


From aaron.s.knister at nasa.gov  Tue Dec 19 08:31:37 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Tue, 19 Dec 2017 08:31:37 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35
In-Reply-To: <4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz>
References: <mailman.3382.1513633600.19082.gpfsug-discuss@spectrumscale.org>,
	<4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz>
Message-ID: <44E99F55-25CC-48DB-9AD6-E7D6794694DC@nasa.gov>

It?s not supported on SLES11 either.


IBM didn?t (that I saw) talk much about this publicly or give customers a chance to provide feedback about the decision. I know it was raised at the UG in NY and I recall a number of people saying it would be a significant issue for them (myself included) as is the fact they no longer support Debian with scale 5.0.

I?d raised the issue on the mailing list after the UG trying to start the discussion but IBM said they weren?t ready to talk about it publicly and I can only guess they had already set their sights and didn?t actually want feedback. This is actually pretty frustrating. I?m tempted to open an RFE but most of my RFEs either have been rejected or just sit idle so I?m not clear there?s a benefit.



On December 19, 2017 at 03:08:27 EST, atmane khiredine <a.khiredine at meteo.dz> wrote:
IBM Spectrum Scale V5.0 not support RHEL 6.x

Only RHEL 7.1 or later

https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux


Atmane Khiredine
HPC System Administrator | Office National de la M?t?orologie
T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz

________________________________________
De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org]
Envoy? : lundi 18 d?cembre 2017 22:46
? : gpfsug-discuss at spectrumscale.org
Objet : gpfsug-discuss Digest, Vol 71, Issue 35

Send gpfsug-discuss mailing list submissions to
gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

1. Re: FW: Spectrum Scale 5.0 now available on Fix Central
(Michael L Taylor)
2. Re: gpfs 4.2.3.5 and RHEL 7.4... (Frederick Stock)
3. Re: gpfs 4.2.3.5 and RHEL 7.4... (Eric Horst)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 13:27:42 -0700
From: "Michael L Taylor" <taylorm at us.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on
Fix Central
Message-ID:
<OFD183EA98.E6691217-ON002581FA.006FA7A4-072581FA.00706687 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"



Hi Bob,
Thanks for the note on 5.0.0
One correction however.... clusters can do a rolling upgrade to 5.0.0 from
any 4.2.x level (not just 4.2.3).




Today's Topics:

1. FW: Spectrum Scale 5.0 now available on Fix Central
(Oesterlin, Robert)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 19:43:35 +0000
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53 at nuance.com>
Content-Type: text/plain; charset="utf-8"

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to
this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum
Scale V5.0 announcements' topic thread in the 'General Parallel File System
- Announce (GPFS - Announce)' forum.

[mid:4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75 at SDEB-EXC02.meteo.dz/forums.png] gpfs at us.ibm.com<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=
> replied to the IBM Spectrum Scale V5.0 announcements<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=
> topic thread in the General Parallel File System - Announce (GPFS -
Announce)<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0


000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=>
forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=
>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/157250d2/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 18 Dec 2017 16:10:55 -0500
From: "Frederick Stock" <stockf at us.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
<OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

Yes the integrated protocols are the Samba and Ganesha that are bundled
with Spectrum Scale. These require the use of the CES component for
monitoring the protocols. If you do use them then you need to wait for a
release of Spectrum Scale in which the integrated protocols are also
supported on RHEL 7.4.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com



From: valdis.kletnieks at vt.edu
To: gpfsug-discuss at spectrumscale.org
Date: 12/18/2017 03:09 PM
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Sent by: gpfsug-discuss-bounces at spectrumscale.org



Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux


that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on
RHEL 7.4."

We don't use AFM or the install toolkit. But we *do* make fairly heavy
use
of mmces and nfs-ganesha - is that what they mean by "Integrated
Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are
doing
their upgrades - and going to 7.4 would be nice. If there's a mine field
there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e=






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/7c272339/attachment-0001.html>

------------------------------

Message: 3
Date: Mon, 18 Dec 2017 21:46:02 +0000
From: Eric Horst <erich at uw.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
<CAF-To-3w7FGHf_ks8cv4WZ=ZF42ZHVOZo4rbu2K8VTQtgAQUWg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Grr, this might explain why I experienced unhappiness when I tried to start
my long-delayed AFM based migration over the weekend. I had previously
tested AFM and found everything working, but 7.4 may have slipped in last
month. The AFM relationship seems to work but `mmafmctl premigrate`
commands fail. I would revert packages if I could figure out where the
issue lies.

-Eric

On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock <stockf at us.ibm.com> wrote:

> Yes the integrated protocols are the Samba and Ganesha that are bundled
> with Spectrum Scale. These require the use of the CES component for
> monitoring the protocols. If you do use them then you need to wait for a
> release of Spectrum Scale in which the integrated protocols are also
> supported on RHEL 7.4.
>
> Fred
> __________________________________________________
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821>
> stockf at us.ibm.com
>
>
>
> From: valdis.kletnieks at vt.edu
> To: gpfsug-discuss at spectrumscale.org
> Date: 12/18/2017 03:09 PM
> Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Currently, the IBM support matrix says:
>
> https://www.ibm.com/support/knowledgecenter/STXKQY/
> gpfsclustersfaq.html#linux
>
> that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:
>
> "AFM, Integrated Protocols, and Installation Toolkit are not supported on
> RHEL 7.4."
>
> We don't use AFM or the install toolkit. But we *do* make fairly heavy use
> of mmces and nfs-ganesha - is that what they mean by "Integrated
> Protocols"?
>
> (We're looking at doing upgrades next month while our HPC clusters are
> doing
> their upgrades - and going to 7.4 would be nice. If there's a mine field
> there, I need to
> make sure we stay at 7.3 - plus applicable non-7.4 updates)
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_
> iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=
> 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_
> I4NKvn8aFGXCqcJARCbu4AOE&e=
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/c01cc906/attachment.html>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 71, Issue 35
**********************************************
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/78969e45/attachment.htm>

From r.sobey at imperial.ac.uk  Tue Dec 19 09:06:08 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 19 Dec 2017 09:06:08 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
In-Reply-To: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
References: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
Message-ID: <DB6PR0601MB2101BA78601B08F6F06469F8DF0F0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Hi Robert

Do you mean the minReleaseLevel from mmlsconfig or just making sure all the nodes are running 4.2.3?

Cheers!
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert
Sent: 18 December 2017 19:44
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Reply-To: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=> replied to the IBM Spectrum Scale V5.0 announcements<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=> topic thread in the General Parallel File System - Announce (GPFS - Announce)<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=> forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
as updated, April 2017

Changes to this release of the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library include the following:

Added DMPs for TIP events

       A topic is added listing the directed maintenance procedures for TIP events.
       The DMPs help users resolve issues caused due to TIP events.

AFM and AFM DR

       - Compression and snapshot ILM policy supported.
       - A general recommendation added for the Gateway node.
       - Configuration parameters added - afmMaxParallelRecoveries, afmAsyncOpWaitTimeout,
           afmSyncOpWaitTimeout, and afmRevalOpWaitTimeout.
       - Configuration parameters modified - afmRPO and afmHashVersion.

Authentication: Authentication packages

       Updated the authentication page to include packages specific to Ubuntu.

Authentication: AD-based authentication

       New information is added on NFS with server-side group
       lookup and Active Directory authentication.

Authentication: Primary group selection configurable for AD + RFC2307 based authentication

       Ability to choose primary group as set in "UNIX attributes" of a user on
       Active Directory introduced with AD + RFC2307 based authentication scheme.
       Earlier, Windows primary group was by default selected as the primary group.

Big data and analytics
       - The GPFS Ambari integration package is now called the IBM Spectrum
         Scale Ambari management pack (in short, management pack or MPack).
       - IBM Spectrum Scale Ambari management pack version 2.4.2.1 with
         HDFS Transparency version 2.7.3.1 supports BI 4.2/BI 4.2.5 IOP migration
         to HDP 2.6.2.
       - Supports the remote mount configuration in Ambari.
       - Supports the multiple file systems configuration. In management
         pack version 2.4.2.1, the current limit is two file systems.
       - The Short circuit write is supported for better performance.
       - In HDFS Transparency, the Ranger performance is enhanced.

Changes to IBM Spectrum Scale management API

   Added the following new commands:

       GET /perfmon/data
       GET /filesystems/{filesystemName}/afm/state
       DELETE /nodes/{name}
       POST /nodes
       GET /nodeclasses
       POST /nodeclasses
       DELETE /nodeclasses/{nodeclassName}
       GET /nodeclasses/{nodeclassName}
       PUT /nodeclasses/{nodeclassName}
       DELETE /jobs/jobId
       POST /filesystems/{filesystemName}/filesets/{filesetName}/psnaps
       DELETE /filesystems/{filesystemName}/filesets/{filesetName}/psnaps/{snapshotName}
       GET /thresholds
       GET /thresholds/{name}
       POST /thresholds
       DELETE /thresholds/{name}

IBM Spectrum Scale GUI changes

   - Added new Networks page to monitor the performance, configuration, and
      adapters of network configurations in the cluster. You can monitor the
      network performance with respect to the IP and RDMA interfaces used in the configuration.
    - Added new Monitoring > Thresholds page to create and monitor the threshold rules
      that are defined in the system.
    - Added Access > Remote Connections page to enable the GUI node of the
      local cluster to monitor the remote cluster by establishing a connection
      with the GUI node of the remote cluster.
    - Added Settings > Call Home page to configure call home. Configuring
      the call home feature helps the IBM? Support to monitor the system.
      Configuring call home also helps to reduce the response time of the
      IBM Support to resolve any issues.
      The diagnostic data that is downloaded through the Settings > Diagnostic Data
      can be uploaded to a problem management record (PMR) by using
      the call home feature in the backend. To upload the diagnostic data, right-click
      the relevant data set from the Previously Collected Diagnostic Data, and
      select Upload to PMR.
    - Added file system creation capabilities in GUI. Use the Files > File Systems > Create File System
      option to launch the Create File System wizard. In the Create File System wizard, you can specify
      the following details of the file system:
         - File system name
         - Storage pools
         - NSDs for the file systems
         - Failure groups
         - NSD order for data writes
         - Maximum number of Spectrum Scale clients
         - Maximum number of inodes of the root fileset
         - Whether to enable quota and scope for the quota definition
         - Whether to enable DMAPI
         - Mount point and automatic mount mode
    - Added the aggregation levels Access Point and Filesets and removed Account
      for the resource type Transparent Cloud Tiering in the Monitoring > Statisitics page.
    - The Files > Transparent Cloud Tiering page now displays the file systems
      and filesets that are mapped with the cloud service. It also shows the
      connection of such a container pair configuration to a cloud account and
      the corresponding CSAP that is configured for the cloud account.
    - Changes to capacity monitoring in the GUI
         - Moved the Capacity page from Monitoring to Files menu in the navigation
           and renamed the GUI page to User Capacity.
         - Only the file data user capacity can be viewed from the Files > User Capacity page.
           Removed the pools, filesets, file system capacity monitoring options from the Files > User Capacity page.
           You can monitor the capacity of these components from the respective GUI pages.
         - Replaced the GPFSPoolCap sensor with the GPFSPool sensor and separate data and metadata
           level capacity monitoring are introduced in the performance charts available in the

           Files > File Systems and Storage > Pools pages.
         - New GPFSPool-based data and metadata performance monitoring metrics are available
           for selection in the Files > Statistics > Edit > Capacity section.
           You need to select the aggregation level as Pool to view these metrics.
    - AFM monitoring changes in the GUI
         - Provides the number of AFM filesets and the corresponding export server maps.
           Each export map establishes a mapping between the gateway node and the NFS
           host name to allow parallel data transfers from cache to home.
         - By using the Request Access option available in the Files > Active File Management
           or Access > Remote Connection page in the GUI, you can now establish connection with remote clusters.
           After establishing the connection, you can monitor the following AFM and AFM DR
           configuration details across clusters:
              * On home and secondary, you can see the AFM relationships configuration,
                health status, and performance values of the Cache and Disaster Recovery grids.
              * On the Overview tab of the detailed view, the available home and secondary inodes are available.
              * On the Overview tab of the detailed view, the details such as NFS throughput,
                IOPs, and latency details are available, if the protocol is NFS.
    - New option to create AFM peer snapshots through GUI. Use the Create Peer Snapshot option
      in the Files > Snapshots page to create peer snapshots. You can view and delete these peer snapshots from
      the Snapshots page and also from the detailed view of the Files > Active File Management page.

Encryption: GSKit V8 improves cryptographic performance on IBM POWER8

   The IBM Global Security Kit (GSKit) Version 8 and later improves cryptographic performance
   on IBM POWER8 hardware. The version of GSKit that is shipped with IBM Spectrum Scale v5.0.0
   offers better performance on POWER8, compared with the versions shipped with earlier releases.

File compression: The lz4 library provides fast access to compressed data

   File compression supports the lz4 compression library. Lz4 is intended primarily for active data and
   favors read-access speed over maximized space saving.

File data: Block and subblock sizes improve I/O performance and reduce fragmentation

   The default block size is larger, 4 MiB instead of 256 KiB, and the sizes of subblocks relative to blocks
   s are smaller, for example, 8 KiB subblocks in a 4 MiB block. A larger block size improves the
   file system performance and a smaller subblock size reduces the amount of unused space. For many business
   applications, the default value of 4 MiB provides the best balance of improved performance
   and reduced fragmentation.

File encryption: AES-XTS encryption is faster on x86 in non-FIPS mode

    On x86 architecture in non-FIPS mode, file encryption with the AES algorithm in XTS mode
    is faster than it was.

File systems: File system rebalancing is faster

    Rebalancing is implemented by a lenient round-robin method that typically runs faster than the
    previously used method of strict round robin. The strict round robin method is available as an option.

installation toolkit changes

    - The installation toolkit has added support for the installation and the deployment
      of IBM Spectrum Scale in a cluster containing Elastic Storage Server (ESS).
    - The installation toolkit has added support for enabling and configuring call home.
    - The installation toolkit has added support for enabling and configuring file audit logging.
    - The installation toolkit has added support for the installation and the
      deployment of IBM Spectrum Scale on Ubuntu 16.04 LTS nodes.
    - The installation toolkit has added verification of passwordless SSH during
      prechecks before installation, deployment, or upgrade.
    - The installation toolkit has added support for cumulative object upgrade.

mmafmctl command

    The --outband parameter is deprecated.

mmcallhome command: Enhancements

    - Addition of -Y option
        * The -Y displays the command output in a parseable format with a colon (:) as a field delimiter.
    - Addition of --pmr option
        * The --pmr option allows you to upload data to existing PMRs using the
          mmcallhome run SendFile command.

mmchconfig command: Enhancements

    - Encrypted files can be copied into an LROC device
        * With the lrocEnableStoringClearText attribute, you can control whether file
          data from encrypted files, which is held in memory as cleartext, is
          copied into a local read-only cache (LROC) device.
    - InfiniBand addresses can be specified for RDMA transfers
        * In the verbsPorts attribute, you can specify InfiniBand addresses
          for RDMA transfers between an NSD client and server.

mmchnsd command: Change NSDs without unmounting the file system

    When you add or remove NSDs or do other operations with mmchnsd,
    you do not need to unmount the file system.

mmcrfs command: Enhancements

     - The default data block size is 4 MiB with an 8 KiB subblock size
         * If no block size is specified, a file system is created with a 4 MiB block size and an
           8 KiB subblock size. The minimum release level (minReleaseLevel) of the
           cluster must be 5.0.0 or greater when the file system is created.
     - The default log file size depends on block size and metadata size
         * If the block size is 512 KiB or larger and the metadata block size 256 KiB or larger,
           then the default log file size is 32 MiB. Otherwise, the default log file
           size is 4 MiB or the metadata block size, whichever is larger.
     - The default method for updating atime is relatime
         * If the minimum release level (minReleaseLevel) of the cluster is 5.0.0 or greater
           when the file system is created, the default method for updating atime is relatime.

mmdsh command: Several options are no longer supported

     The --ignoreSignal, -I, and -d options are no longer supported.
     Do not use these options unless instructed to by IBM support personnel.

mmfsck command: Display an interim status report at any time

     While a long-running instance of mmfsck is in progress, you can start another instance
     of mmfsck with the --status-report parameter to display current
     status information from all the nodes that are participating in the mmfsck run.

mmgetstate command: Display the unresponsive state

     The command returns the unresponsive state when the GPFS
     daemon is running but is not responding.

mmhealth command: Addition to measurement options

     Measurement options for filesystem, SMB node, and NFS node
     has been added to the mmhealth command.

mmkeyserv command: The simplified method supports certificate chains from a certificate authority.

     In the simplified method, with the --kmip-cert parameter, you can set up encryption with IBM?
     Security Key Lifecycle Manager (SKLM) as the key management server and with a certificate signed
     by a certificate authority (CA) on the KMIP port of the SKLM server.

mmnetverify command: Enhancements

     - Verify the network operation of nodes in a subnet
         * With the --subnets parameters, you can specify the subnet
           addresses of the nodes that you want to verify.
     - Verify that nodes can handle a new MTU size
         * With the ping-packet-size parameter, you can specify the size
           of the ICMP echo request packets that are sent between local node and the target
           node during the ping test.

mmtracectl command: Display the tracing status

     The --status parameter displays the tracing status of the specified nodes.

New feature for threshold monitoring

      Starting from IBM Spectrum Scale version 5.0.0, if multiple thresholds rules
      have overlapping entities for the same metrics, only one of the concurrent rules is
      made actively eligible.

NFS: Dynamic export changes

      You can dynamically change the export configuration without restarting the NFS service.

Object

      - Support for Ubuntu
      - Support for sudo wrapper for Object on Ubuntu
      - Support for cumulative upgrades from older versions
      - Object snap enhancement to contain keystore logs

Protocol support: Enhanced

      Protocol support is extended to Ubuntu 10.04

Setting up a system for storing crash files for Ubuntu

      A topic is added to describe how to set up a system for storing crash files for Ubuntu.

SMB: DFS redirects for SMB shares

      New option to configure DFS redirects for SMB shares.

SMB: SMB server upgrade changes

      Two events on CTDB version match/mismatch are added to the RAS events.

Sudo wrappers: Root-level processes can call administration commands directly

      Root-level background processes, such as cron and callback programs, can
      successfully call administration commands directly rather than through sudo
      when sudo wrappers are enabled.

Supported clients for NFS

      A topic is added listing the clients that are supported by NFS protocol.

Transparent cloud tiering

      - Support for multiple cloud storage accounts
      - Support for multiple file systems or filesets per node group
      - Enhanced support for large file systems provided by container spillover
      - Support for associating file sets with containers for enhanced granularity
      - Multiple URL and region support at the node level
      - Support for creating a cloud service separately for tiering and sharing operations.
      - Unique encryption key per cloud container
      - Support for remotely mounted clients.
      - Support for Amazon S3 regions requiring Sigv4 security support,
        including the US government cloud region.
      - Ability to enable or disable transparent recall for files for a given file
        system instantly, without having to rewrite a policy.
      - Support for backing up and restoring the Cloud services configuration in case of any disaster.
      - Support for backing up the Cloud services database to the cloud.
      - Support for restoring Transparent cloud tiering service on an identical backup cluster.
      - Support for checking the integrity of the Cloud services
        database after any system crash or outage.
      - Support for auditing events relating to each operation performed in Cloud services.

New commands

      mmaudit mmmsgqueue

Changed commands

      mmafmctl mmadddisk mmcallhome mmchattr mmchcluster mmchconfig mmchfs mmchnsd
      mmcloudgateway mmcrcluster mmcrfs mmdeldisk mmdsh mmfsck mmgetstate
      mmkeyserv mmnetverify mmnfs mmrestripefile mmrestripefs mmsmb mmtracectl
      mmuserauth

Deleted commands

      mmrest

New messages

      6027-1264, 6027-1757, 6027-2394, 6027-2395, 6027-2396, 6027-2397
      6027-2398, 6027-2399, 6027-2400, 6027-2401, 6027-3259, 6027-3408
      6027-3597, 6027-3598, 6027-3599, 6027-3600, 6027-3601, 6027-3602
      6027-3603, 6027-3604, 6027-3730, 6027-3921, 6027-3922, 6027-3923
      6027-3924, 6027-3925, 6027-3926, 6027-3927, 6027-3928
      6027-3929, 6027-3930, 6027-3931, 6027-4019

Changed messages

      6027-928


     Open this item<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab-234019da9b-2D248a-2D49ec-2D9503-2De612becd953e&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=Q6E0fg8sor2mORS3ugkurXkuV2ZUDqRd_C6_jpYXVVM&e=>


Posting Date:

Monday, December 18, 2017 at 2:26:38 PM EST



To unsubscribe or change settings, please go to your developerWorks community Settings.<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_news&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=5C4z0hhuaas4Ghs1g4YfFTCE0eyNYJVmLPYrGPCuE7Q&e=>
This is a notification sent from developerWorks community.

The information contained in this e-mail message is intended only for the use of the named recipients and is subject to the IBM Online Privacy Statement<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ibm.com_privacy_details_us_en_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=7gE9uoypP_yHb1jMMSjdCN_xyJ7UU2Ifrx1TchwMkQA&e=>. If you have received this e-mail in error, please contact the sender to report the error and then delete this message from your system.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/ced8c773/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 289 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/ced8c773/attachment.png>

From Robert.Oesterlin at nuance.com  Tue Dec 19 12:38:12 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 19 Dec 2017 12:38:12 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
Message-ID: <73C34DA5-36DF-48C4-9E60-4DE75AA92405@nuance.com>

IBM corrected me a bit after I posted ? the cluster can be at any 4.2.X level (minreleaselevel)


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Sobey, Richard A" <r.sobey at imperial.ac.uk>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, December 19, 2017 at 3:07 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central

Hi Robert

Do you mean the minReleaseLevel from mmlsconfig or just making sure all the nodes are running 4.2.3?

Cheers!
Richard


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/c8a838b8/attachment.htm>

From Robert.Oesterlin at nuance.com  Tue Dec 19 12:49:40 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 19 Dec 2017 12:49:40 +0000
Subject: [gpfsug-discuss] Scale 5.0 - Linux supported levels
Message-ID: <6844F00F-F07D-4FEF-B42D-CD32179A057E@nuance.com>

It?s a major problem for me that the dropped RH 6.X support with Scale 5.0. Now, I need to get my clients all to RH7 (which is problematic due to some older code needing the 6.X level) before I can take full advantage of Scale 5.0.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]" <aaron.s.knister at nasa.gov>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, December 19, 2017 at 2:32 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35

It?s not supported on SLES11 either.


IBM didn?t (that I saw) talk much about this publicly or give customers a chance to provide feedback about the decision. I know it was raised at the UG in NY and I recall a number of people saying it would be a significant issue for them (myself included) as is the fact they no longer support Debian with scale 5.0.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/66dbccf7/attachment.htm>

From duersch at us.ibm.com  Tue Dec 19 13:18:08 2017
From: duersch at us.ibm.com (Steve Duersch)
Date: Tue, 19 Dec 2017 08:18:08 -0500
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
In-Reply-To: <mailman.3390.1513674371.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3390.1513674371.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <OF252F90AC.16B92D2A-ON852581FB.00486631-852581FB.00491295@notes.na.collabserv.com>


As Mike Taylor pointed out in a previous post this was an incorrect
statement.
You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a
rolling upgrade.
The minReleaseLevel is not pertinent to a rolling upgrade.  The running
daemon is the important part.  So you can't have any 4.1.x nodes in your
cluster and do a rolling upgrade to 5.0.

Also, Aaron, as to the OS support.  This decision was not made without some
angst.  As I mentioned at the user group meeting in NYC...the key point is
that we would like to get to a more current compiler.  This will allow us
to take advantage of newer features and functions and hopefully make the
code better for customers.  SLES 12 has been around for over 2 years.

I hope this helps give some thinking behind the decision.


Steve Duersch
Spectrum Scale
845-433-7902
IBM Poughkeepsie, New York


> Today's Topics:
>
>    1. Re: Spectrum Scale 5.0 now available on Fix Central
>       (Sobey, Richard A)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 19 Dec 2017 09:06:08 +0000
> From: "Sobey, Richard A" <r.sobey at imperial.ac.uk>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix
>    Central
> Message-ID:
>
>
<DB6PR0601MB2101BA78601B08F6F06469F8DF0F0 at DB6PR0601MB2101.eurprd06.prod.outlook.com>

>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Robert
>
> Do you mean the minReleaseLevel from mmlsconfig or just making sure
> all the nodes are running 4.2.3?
>
> Cheers!
> Richard
>
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-
> discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert
> Sent: 18 December 2017 19:44
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
Central
>
> The Scale 5.0 fix level is now up on Fix Central.
>
> You need to be at Scale 4.2.3 (cluster level) to do a rolling
> upgrade to this level.
>
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/07b56fa1/attachment.htm>

From Robert.Oesterlin at nuance.com  Tue Dec 19 13:41:50 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 19 Dec 2017 13:41:50 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
Message-ID: <1A2E660D-A3F6-410E-9504-4B3BB5A34B09@nuance.com>

Thanks for clearing that up!


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Steve Duersch <duersch at us.ibm.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, December 19, 2017 at 7:20 AM
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central

As Mike Taylor pointed out in a previous post this was an incorrect statement.
You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a rolling upgrade.
The minReleaseLevel is not pertinent to a rolling upgrade. The running daemon is the important part. So you can't have any 4.1.x nodes in your cluster and do a rolling upgrade to 5.0.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/ea1c3e28/attachment.htm>

From S.J.Thompson at bham.ac.uk  Tue Dec 19 13:55:34 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Tue, 19 Dec 2017 13:55:34 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
In-Reply-To: <OF252F90AC.16B92D2A-ON852581FB.00486631-852581FB.00491295@notes.na.collabserv.com>
References: <mailman.3390.1513674371.19082.gpfsug-discuss@spectrumscale.org>
	<OF252F90AC.16B92D2A-ON852581FB.00486631-852581FB.00491295@notes.na.collabserv.com>
Message-ID: <591BE98E-9C39-41D4-BAEA-DCFF53623E01@bham.ac.uk>

Maybe it would have been a good idea to make this clear in the ?What?s new in 5.0? slide decks used at SC. I don?t recall it being there. And the lack of forward public notification on this is not great, particularly for those not in NYC. Sure most of my clusters are on EL7 now, but I still have some nodes still running 6.x (notably some of our Spectrum Protect nodes which are not just systems we can reinstall).

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "duersch at us.ibm.com" <duersch at us.ibm.com>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, 19 December 2017 at 13:19
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central


As Mike Taylor pointed out in a previous post this was an incorrect statement.
You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a rolling upgrade.
The minReleaseLevel is not pertinent to a rolling upgrade. The running daemon is the important part. So you can't have any 4.1.x nodes in your cluster and do a rolling upgrade to 5.0.

Also, Aaron, as to the OS support. This decision was not made without some angst. As I mentioned at the user group meeting in NYC...the key point is that we would like to get to a more current compiler. This will allow us to take advantage of newer features and functions and hopefully make the code better for customers. SLES 12 has been around for over 2 years.

I hope this helps give some thinking behind the decision.


Steve Duersch
Spectrum Scale
845-433-7902
IBM Poughkeepsie, New York


> Today's Topics:
>
>    1. Re: Spectrum Scale 5.0 now available on Fix Central
>       (Sobey, Richard A)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 19 Dec 2017 09:06:08 +0000
> From: "Sobey, Richard A" <r.sobey at imperial.ac.uk>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix
>    Central
> Message-ID:
>
> <DB6PR0601MB2101BA78601B08F6F06469F8DF0F0 at DB6PR0601MB2101.eurprd06.prod.outlook.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Robert
>
> Do you mean the minReleaseLevel from mmlsconfig or just making sure
> all the nodes are running 4.2.3?
>
> Cheers!
> Richard
>
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-
> discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert
> Sent: 18 December 2017 19:44
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central
>
> The Scale 5.0 fix level is now up on Fix Central.
>
> You need to be at Scale 4.2.3 (cluster level) to do a rolling
> upgrade to this level.
>
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/3894c70b/attachment.htm>

From Mark.Bush at siriuscom.com  Tue Dec 19 14:29:21 2017
From: Mark.Bush at siriuscom.com (Mark Bush)
Date: Tue, 19 Dec 2017 14:29:21 +0000
Subject: [gpfsug-discuss] pmcollector and NSD perf
Message-ID: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>

I've noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in the GUI on the monitoring screen with the default view the NSD Server Throughput graph shows "Performance Collector did not return any data".  I've seen that in other items (SMB before for example) but never for NSD.  Is there something that must be enabled in the zimon sensor or collector config file to grab this or is this a bug?


Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/d814d122/attachment.htm>

From rohwedder at de.ibm.com  Tue Dec 19 15:23:55 2017
From: rohwedder at de.ibm.com (Markus Rohwedder)
Date: Tue, 19 Dec 2017 16:23:55 +0100
Subject: [gpfsug-discuss] pmcollector and NSD perf
In-Reply-To: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
References: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
Message-ID: <OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>

Hello Mark,

the NSD sensor is GPFSNSDDisk.

Some things to check:

1. Is the sensor activated?
In a GPFS managed sensor config you should be able to see something like
this when you call mmperfmon config show:

{
        name = "GPFSNSDDisk"
        period = 10
        restrict = "nsdNodes"
},

2. Perfmon designation
The NSD server nodes should have the perfmon designation.


[root at cache-41 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfsgui-cluster-4.localnet.com
  GPFS cluster id:           10583479681538672379
  GPFS UID domain:           localnet.com
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

 Node  Daemon node name       IP address   Admin node name
Designation
------------------------------------------------------------------------------
   1   cache-41.localnet.com  10.0.100.41  cache-41.localnet.com
quorum-perfmon
   2   cache-42.localnet.com  10.0.100.42  cache-42.localnet.com
quorum-gateway-perfmon
   3   cache-43.localnet.com  10.0.100.43  cache-43.localnet.com
gateway-perfmon

3. Direct Disk writes?
One reason why there may be no data on your system is if you are not using
the NSD protocol,
meaning the clients can directly write to disk as in a SAN environment.
In this case the sensor does not catch the transactions.

4. Cross cluster mount
Or maybe you are using a cross cluster mount.



Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

Spectrum Scale GUI Development
                                                                              
                                                                              
                                                                              
                                                                              
                                                                              
 Phone:            +49 7034 6430190      IBM Deutschland                      
                                                                              
 E-Mail:           rohwedder at de.ibm.com  Am Weiher 24                         
                                                                              
                                         65451 Kelsterbach                    
                                                                              
                                         Germany                              
                                                                              
                                                                              
                                                                              
                                                                              
                                                                              
 IBM Deutschland                                                              
 Research &                                                                   
 Development                                                                  
 GmbH /                                                                       
 Vorsitzender des                                                             
 Aufsichtsrats:                                                               
 Martina K?deritz                                                             
 Gesch?ftsf?hrung:                                                            
 Dirk Wittkopp                                                                
 Sitz der                                                                     
 Gesellschaft:                                                                
 B?blingen /                                                                  
 Registergericht:                                                             
 Amtsgericht                                                                  
 Stuttgart, HRB                                                               
 243294                                                                       
                                                                              





From:	Mark Bush <Mark.Bush at siriuscom.com>
To:	"gpfsug-discuss at spectrumscale.org"
            <gpfsug-discuss at spectrumscale.org>
Date:	12/19/2017 03:30 PM
Subject:	[gpfsug-discuss] pmcollector and NSD perf
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



I?ve noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in
the GUI on the monitoring screen with the default view the NSD Server
Throughput graph shows ?Performance Collector did not return any data?.
I?ve seen that in other items (SMB before for example) but never for NSD.
Is there something that must be enabled in the zimon sensor or collector
config file to grab this or is this a bug?


Mark_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=a6GCq72qeADy6hsfA-24PmWHU06W5z2xqx9tKIJ8qJ4&s=OQccy8ikWB-ByYgLsJFgI8szDs1ZrwnsaFrLCwTfTwI&e=



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1A013420.gif
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment-0002.gif>

From Mark.Bush at siriuscom.com  Tue Dec 19 15:49:59 2017
From: Mark.Bush at siriuscom.com (Mark Bush)
Date: Tue, 19 Dec 2017 15:49:59 +0000
Subject: [gpfsug-discuss] pmcollector and NSD perf
In-Reply-To: <OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>
References: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
	<OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>
Message-ID: <DM5PR05MB3577119DC6FE73462E83BC72E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>

It appears number 3 on your list is the case.  My nodes are all SAN connected and until I get separate CES nodes no NSD is necessary (currently run CES on the NSD servers ? just for a test cluster).

Mark

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Markus Rohwedder
Sent: Tuesday, December 19, 2017 9:24 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] pmcollector and NSD perf


Hello Mark,

the NSD sensor is GPFSNSDDisk.

Some things to check:

1. Is the sensor activated?
In a GPFS managed sensor config you should be able to see something like this when you call mmperfmon config show:

{
name = "GPFSNSDDisk"
period = 10
restrict = "nsdNodes"
},

2. Perfmon designation
The NSD server nodes should have the perfmon designation.


[root at cache-41 ~]# mmlscluster

GPFS cluster information
========================
GPFS cluster name: gpfsgui-cluster-4.localnet.com
GPFS cluster id: 10583479681538672379
GPFS UID domain: localnet.com
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCR

Node Daemon node name IP address Admin node name Designation
------------------------------------------------------------------------------
1 cache-41.localnet.com 10.0.100.41 cache-41.localnet.com quorum-perfmon
2 cache-42.localnet.com 10.0.100.42 cache-42.localnet.com quorum-gateway-perfmon
3 cache-43.localnet.com 10.0.100.43 cache-43.localnet.com gateway-perfmon

3. Direct Disk writes?
One reason why there may be no data on your system is if you are not using the NSD protocol,
meaning the clients can directly write to disk as in a SAN environment.
In this case the sensor does not catch the transactions.

4. Cross cluster mount
Or maybe you are using a cross cluster mount.



Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

Spectrum Scale GUI Development

________________________________



Phone:

+49 7034 6430190

IBM Deutschland

[cid:image002.gif at 01D378AE.BAB5B6F0]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________



IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina K?deritz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294



[Inactive hide details for Mark Bush ---12/19/2017 03:30:14 PM---I've noticed this in my test cluster both in 4.2.3.4 and 5.0.0.]Mark Bush ---12/19/2017 03:30:14 PM---I've noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in the GUI on the monitoring s

From: Mark Bush <Mark.Bush at siriuscom.com<mailto:Mark.Bush at siriuscom.com>>
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: 12/19/2017 03:30 PM
Subject: [gpfsug-discuss] pmcollector and NSD perf
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



I?ve noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in the GUI on the monitoring screen with the default view the NSD Server Throughput graph shows ?Performance Collector did not return any data?. I?ve seen that in other items (SMB before for example) but never for NSD. Is there something that must be enabled in the zimon sensor or collector config file to grab this or is this a bug?


Mark_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=a6GCq72qeADy6hsfA-24PmWHU06W5z2xqx9tKIJ8qJ4&s=OQccy8ikWB-ByYgLsJFgI8szDs1ZrwnsaFrLCwTfTwI&e=


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 166 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.gif
Type: image/gif
Size: 1851 bytes
Desc: image002.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.gif
Type: image/gif
Size: 105 bytes
Desc: image003.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment-0001.gif>

From hoov at us.ibm.com  Tue Dec 19 18:40:56 2017
From: hoov at us.ibm.com (Theodore Hoover Jr)
Date: Tue, 19 Dec 2017 18:40:56 +0000
Subject: [gpfsug-discuss] Spectrum Scale on AWS - Join Sponsor User Program
Message-ID: <OFA4E895DE.FCD6420A-ON002581FB.00665ACE-002581FB.00669FEE@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/98449b9b/attachment.htm>

From oehmes at gmail.com  Tue Dec 19 20:20:29 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Tue, 19 Dec 2017 20:20:29 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
	<CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
	<6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>
Message-ID: <CALssuR3wX4_6T668BEupSvdt75pxgZuBSj5rzbVVy=tWWoFhxQ@mail.gmail.com>

Hi,

the zero padding was never promoted into a GA stream, it was an experiment
to proof we are on the right track when we eliminate the overhead from
client to NSD Server, but also showed that alone is not good enough. the
work for the client is the same compared to the >32 subblocks, but the NSD
Server has more work as it can't pack as many subblocks and therefore files
into larger blocks, so you need to do more writes to store the same number
of files.
thats why there is the additional substantial improvement  when we then
went to >32 subblocks.

sven

On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:

> Thanks Sven! That makes sense to me and is what I thought was the case
> which is why I was confused when I saw the reply to the thread that said
> the >32 subblocks code had no performance impact.
>
> A couple more question for you? in your presentation there?s a benchmark
> that shows the file create performance without the zero padding. Since you
> mention this is done for security reasons was that feature ever promoted to
> a GA Scale release? I?m also wondering if you could explain the performance
> difference between the no zero padding code and the > 32 subblock code
> since given your the example of 32K files and 16MB block size I figure both
> cases ought to write the same amount to disk.
>
> Thanks!
>
> -Aaron
>
>
>
>
>
> On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com> wrote:
>
> i thought i answered that already, but maybe i just thought about
> answering it and then forgot about it :-D
>
> so yes more than 32 subblocks per block significant increase the
> performance of filesystems with small files, for the sake of the argument
> let's say 32k in a large block filesystem again for sake of argument say
> 16MB.
>
> you probably ask why ?
>
> if you create a file and write 32k into it in a pre 5.0.0 Version 16 MB
> filesystem your client actually doesn't write 32k to the NSD Server, it
> writes 512k, because thats the subblock size and we need to write the full
> subblock (for security reasons). so first you waste significant memory on
> the client to cache that zero padding, you waste network bandwidth and you
> waste NSD Server cache because you store it there too. this means you
> overrun the cache more quickly, means you start doing read/modify writes
> earlier on all your nice large raid tracks... i guess you get the story by
> now.
>
> in fact,  if you have a good raid code that can drive really a lot of
> bandwidth out of individual drives like a GNR system you get more
> performance for small file writes as larger your blocksize is, because we
> can 'pack' more files into larger i/os and therefore turn a small file
> create workload into a bandwidth workload, essentially exactly what we did
> and i demonstrated in the CORAL presentation .
>
> hope that makes this crystal clear now .
>
> sven
>
>
>
> On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister <aaron.s.knister at nasa.gov>
> wrote:
>
> Thanks, Alex. I'm all too familiar with the trade offs between large
> blocks and small files and we do use pretty robust SSD storage for our
> metadata. We support a wide range of workloads and we have some folks
> with many small (<1M) files and other folks with many large (>256MB) files.
>
> My point in this thread is that IBM has said over and over again in
> presentations that there is a significant performance gain with the >32
> subblocks code on filesystems with large block sizes (although to your
> point I'm not clear on exactly what large means since I didn't define
> large in this context). Therefore given that the >32 subblock code gives
> a significant performance gain one could reasonably assume that having a
> filesystem with >32 subblocks is required to see this gain (rather than
> just running the >32 subblocks code on an fs w/o > 32 subblocks).
>
> This lead me to ask about a migration tool because in my mind if there's
> a performance gain from having >32 subblocks on the FS I'd like that
> feature and having to manually copy 10's of PB to new hardware to get
> this performance boost is unacceptable. However, IBM can't seem to make
> up their mind about whether or not the >32 subblocks code *actually*
> provides a performance increase. This seems like a pretty
> straightforward question.
>
> -Aaron
>
> On 12/15/17 3:48 PM, Alex Chekholko wrote:
> > Hey Aaron,
> >
> > Can you define your sizes for "large blocks" and "small files"?  If you
> > dial one up and the other down, your performance will be worse.  And in
> > any case it's a pathological corner case so it shouldn't matter much for
> > your workflow, unless you've designed your system with the wrong values.
> >
> > For example, for bioinformatics workloads, I prefer to use 256KB
> > filesystem block size, and I'd consider 4MB+ to be "large block size",
> > which would make the filesystem obviously unsuitable for processing
> > millions of 8KB files.
> >
> > You can make a histogram of file sizes in your existing filesystems and
> > then make your subblock size (1/32 of block size) on the smaller end of
> > that.   Also definitely use the "small file in inode" feature and put
> > your metadata on SSD.
> >
> > Regards,
> > Alex
> >
> > On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> > <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
> >
> >     Thanks, Bill.
> >
> >     I still don't feel like I've got an clear answer from IBM and frankly
> >     the core issue of a lack of migration tool was totally dodged.
> >
> >     Again in Sven's presentation from SSUG @ SC17
> >     (
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
> >     he mentions "It has a significant performance penalty for small
> files in
> >     large block size filesystems" and the demonstrates that with several
> >     mdtest runs (which show the effect with and without the >32
> >     subblocks code):
> >
> >
> >     4.2.1 base code - SUMMARY: (of 3 iterations)
> >     File creation : Mean = 2237.644
> >
> >     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY:
> (of
> >     3 iterations)
> >     File creation : Mean = 12866.842
> >
> >     more sub blocks per block (4.2.2 + morethan32subblock code):
> >     File creation : Mean = 40316.721
> >
> >     Can someone (ideally Sven) give me a straight answer as to whether or
> >     not the > 32 subblock code actually makes a performance difference
> for
> >     small files in large block filesystems? And if not, help me
> understand
> >     why his slides and provided benchmark data have consistently
> indicated
> >     it does?
> >
> >     -Aaron
> >
> >     On 12/1/17 11:44 AM, Bill Hartner wrote:
> >     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> >     > results @ user group meeting.
> >     >
> >     > -Bill
> >     >
> >     > Bill Hartner
> >     > IBM Systems
> >     > Scalable I/O Development
> >     > Austin, Texas
> >     > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
> >     > home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >
> >     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017
> 06:53:44
> >     > AM---Bill, could you say something about what the metadataJan-Frode
> >     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> >     > about what the metadata-storage here was? ESS/NL-SAS/3way
> replication?
> >     >
> >     > From: Jan-Frode Myklebust <janfrode at tanso.net <mailto:
> janfrode at tanso.net>>
> >     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>
> >     > Date: 12/01/2017 06:53 AM
> >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     > Sent by: gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> >     >
> >     >
> >
>  ------------------------------------------------------------------------
> >     >
> >     >
> >     >
> >     > Bill, could you say something about what the metadata-storage here
> was?
> >     > ESS/NL-SAS/3way replication?
> >     >
> >     > I just asked about this in the internal slack channel #scale-help
> today..
> >     >
> >     >
> >     >
> >     > -jf
> >     >
> >     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
> <_bhartner at us.ibm.com_
> >     > <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>:
> >     >
> >     >     > "It has a significant performance penalty for small files in
> >     large
> >     >     > block size filesystems"
> >     >
> >     >     Aaron,
> >     >
> >     >     Below are mdtest results for a test we ran for CORAL - file
> >     size was
> >     >     32k.
> >     >
> >     >     We have not gone back and ran the test on a file system
> formatted
> >     >     without > 32 subblocks. We'll do that at some point...
> >     >
> >     >     -Bill
> >     >
> >     >     -- started at 10/28/2017 17:51:38 --
> >     >
> >     >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> >     >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
> >     >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
> >     -p 360
> >     >     -u -y
> >     >     Path: /ibm/fs2-16m-10
> >     >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> >     >
> >     >     228 tasks, 67239936 files
> >     >
> >     >     SUMMARY: (of 3 iterations)
> >     >     Operation Max Min Mean Std Dev
> >     >     --------- --- --- ---- -------
> >     >     File creation : 51953.498 50558.517 51423.221 616.643
> >     >     File stat : 0.000 0.000 0.000 0.000
> >     >     File read : 0.000 0.000 0.000 0.000
> >     >     File removal : 96746.376 92149.535 94658.774 1900.187
> >     >     Tree creation : 1.588 0.070 0.599 0.700
> >     >     Tree removal : 0.213 0.034 0.097 0.082
> >     >
> >     >     -- finished at 10/28/2017 19:51:54 --
> >     >
> >     >     Bill Hartner
> >     >     IBM Systems
> >     >     Scalable I/O Development
> >     >     Austin, Texas_
> >     >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
> >     <mailto:bhartner at us.ibm.com>>
> >     >     home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >     _
> >     >     __gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>> wrote on
> >     >     11/29/2017 04:41:48 PM:
> >     >
> >     >     > From: Aaron Knister <_aaron.knister at gmail.com_
> >     >     <mailto:aaron.knister at gmail.com <mailto:
> aaron.knister at gmail.com>>>
> >     >
> >     >
> >     >     > To: gpfsug main discussion list
> >     >     <_gpfsug-discuss at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>>
> >     >
> >     >     > Date: 11/29/2017 04:42 PM
> >     >
> >     >
> >     >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
> >     >
> >     >     >
> >     >
> >     >     > Thanks, Nikhil. Most of that was consistent with my
> understnading,
> >     >     > however I was under the impression that the >32 subblocks
> code is
> >     >     > required to achieve the touted 50k file creates/second that
> Sven has
> >     >     > talked about a bunch of times:
> >     >     >
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >     <
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >>
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
> >     <
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >>
> >     >     > _Sven_Oehme_-_News_from_Research.pdf
> >     >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
> >     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >>
> >     >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> >     >
> >     >
> >     >     > from those presentations regarding 32 subblocks:
> >     >     >
> >     >     > "It has a significant performance penalty for small files in
> large
> >     >     > block size filesystems"
> >     >
> >     >     > although I'm not clear on the specific definition of
> "large". Many
> >     >     > filesystems I encounter only have a 1M block size so it may
> not
> >     >     > matter there, although that same presentation clearly shows
> the
> >     >     > benefit of larger block sizes which is yet *another* thing
> for which
> >     >     > a migration tool would be helpful.
> >     >
> >     >     > -Aaron
> >     >     >
> >     >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
> >     >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
> >     <mailto:nikhilk at us.ibm.com>>> wrote:
> >     >
> >     >     > Hi,
> >     >     >
> >     >     > I would like to clarify migration path to 5.0.0 from 4.X.X
> >     clusters.
> >     >     > For all Spectrum Scale clusters that are currently at 4.X.X,
> >     it is
> >     >     > possible to migrate to 5.0.0 with no offline data migration
> >     and no
> >     >     > need to move data. Once these clusters are at 5.0.0, they
> will
> >     >     > benefit from the performance improvements, new features
> (such as
> >     >     > file audit logging), and various enhancements that are
> >     included in
> >     >     5.0.0.
> >     >     >
> >     >     > That being said, there is one enhancement that will not be
> >     applied
> >     >     > to these clusters, and that is the increased number of
> >     sub-blocks
> >     >     > per block for small file allocation. This means that for file
> >     >     > systems with a large block size and a lot of small files, the
> >     >     > overall space utilization will be the same it currently is
> >     in 4.X.X.
> >     >     > Since file systems created at 4.X.X and earlier used a block
> >     size
> >     >     > that kept this allocation in mind, there should be very
> little
> >     >     > impact on existing file systems.
> >     >     >
> >     >     > Outside of that one particular function, the remainder of the
> >     >     > performance improvements, metadata improvements, updated
> >     >     > compatibility, new functionality, and all of the other
> >     enhancements
> >     >     > will be immediately available to you once you complete the
> >     upgrade
> >     >     > to 5.0.0 -- with no need to reformat, move data, or take
> >     your data
> >     >     offline.
> >     >     >
> >     >     > I hope that clarifies things a little and makes the upgrade
> path
> >     >     > more accessible.
> >     >     >
> >     >     > Please let me know if there are any other questions or
> concerns.
> >     >     >
> >     >     > Thank you,
> >     >     > Nikhil Khandelwal
> >     >     > Spectrum Scale Development
> >     >     > Client Adoption
> >     >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>
> >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >
> >     >     > _https://urldefense.proofpoint.com/v2/url?_
> >     <https://urldefense.proofpoint.com/v2/url?_>
> >     >     >
> >     >
>  u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
> >     >     >
> >     >
>  siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> >     >
> >     >     _______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>_
> >     >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>_______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >
> >     >
> >     >
> >     >
> >     >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >     >
> >
> >     --
> >     Aaron Knister
> >     NASA Center for Climate Simulation (Code 606.2)
> >     Goddard Space Flight Center
> >     (301) 286-2776 <tel:%28301%29%20286-2776>
> >     _______________________________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/92903846/attachment.htm>

From aaron.s.knister at nasa.gov  Tue Dec 19 21:32:00 2017
From: aaron.s.knister at nasa.gov (Aaron Knister)
Date: Tue, 19 Dec 2017 16:32:00 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CALssuR3wX4_6T668BEupSvdt75pxgZuBSj5rzbVVy=tWWoFhxQ@mail.gmail.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
	<CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
	<6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>
	<CALssuR3wX4_6T668BEupSvdt75pxgZuBSj5rzbVVy=tWWoFhxQ@mail.gmail.com>
Message-ID: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>

Thanks, Sven. Understood!

On 12/19/17 3:20 PM, Sven Oehme wrote:
> Hi,
> 
> the zero padding was never promoted into a GA stream, it was an
> experiment to proof we are on the right track when we eliminate the
> overhead from client to NSD Server, but also showed that alone is not
> good enough. the work for the client is the same compared to the >32
> subblocks, but the NSD Server has more work as it can't pack as many
> subblocks and therefore files into larger blocks, so you need to do more
> writes to store the same number of files.?
> thats why there is the additional substantial improvement? when we then
> went to >32 subblocks.?
> 
> sven
> 
> On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
> SCIENCE CORP] <aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>> wrote:
> 
>     Thanks Sven! That makes sense to me and is what I thought was the
>     case which is why I was confused when I saw the reply to the thread
>     that said the >32 subblocks code had no performance impact.?
> 
>     A couple more question for you? in your presentation there?s a
>     benchmark that shows the file create performance without the zero
>     padding. Since you mention this is done for security reasons was
>     that feature ever promoted to a GA Scale release? I?m also wondering
>     if you could explain the performance difference between the no zero
>     padding code and the > 32 subblock code since given your the example
>     of 32K files and 16MB block size I figure both cases ought to write
>     the same amount to disk.?
> 
>     Thanks!
> 
>     -Aaron
> 
> 
> 
> 
> 
>     On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com
>     <mailto:oehmes at gmail.com>> wrote:
>>     i thought i answered that already, but maybe i just thought about
>>     answering it and then forgot about it :-D
>>
>>     so yes more than 32 subblocks per block significant increase the
>>     performance of filesystems with small files, for the sake of the
>>     argument let's say 32k in a large block filesystem again for sake
>>     of argument say 16MB.?
>>
>>     you probably ask why ??
>>
>>     if you create a file and write 32k into it in a pre 5.0.0 Version
>>     16 MB filesystem your client actually doesn't write 32k to the NSD
>>     Server, it writes 512k, because thats the subblock size and we
>>     need to write the full subblock (for security reasons). so first
>>     you waste significant memory on the client to cache that zero
>>     padding, you waste network bandwidth and you waste NSD Server
>>     cache because you store it there too. this means you overrun the
>>     cache more quickly, means you start doing read/modify writes
>>     earlier on all your nice large raid tracks... i guess you get the
>>     story by now.?
>>
>>     in fact,? if you have a good raid code that can drive really a lot
>>     of bandwidth out of individual drives like a GNR system you get
>>     more performance for small file writes as larger your blocksize
>>     is, because we can 'pack' more files into larger i/os and
>>     therefore turn a small file create workload into a bandwidth
>>     workload, essentially exactly what we did and i demonstrated in
>>     the CORAL presentation .?
>>
>>     hope that makes this crystal clear now .?
>>
>>     sven
>>
>>
>>
>>     On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister
>>     <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
>>
>>         Thanks, Alex. I'm all too familiar with the trade offs between
>>         large
>>         blocks and small files and we do use pretty robust SSD storage
>>         for our
>>         metadata. We support a wide range of workloads and we have
>>         some folks
>>         with many small (<1M) files and other folks with many large
>>         (>256MB) files.
>>
>>         My point in this thread is that IBM has said over and over
>>         again in
>>         presentations that there is a significant performance gain
>>         with the >32
>>         subblocks code on filesystems with large block sizes (although
>>         to your
>>         point I'm not clear on exactly what large means since I didn't
>>         define
>>         large in this context). Therefore given that the >32 subblock
>>         code gives
>>         a significant performance gain one could reasonably assume
>>         that having a
>>         filesystem with >32 subblocks is required to see this gain
>>         (rather than
>>         just running the >32 subblocks code on an fs w/o > 32 subblocks).
>>
>>         This lead me to ask about a migration tool because in my mind
>>         if there's
>>         a performance gain from having >32 subblocks on the FS I'd
>>         like that
>>         feature and having to manually copy 10's of PB to new hardware
>>         to get
>>         this performance boost is unacceptable. However, IBM can't
>>         seem to make
>>         up their mind about whether or not the >32 subblocks code
>>         *actually*
>>         provides a performance increase. This seems like a pretty
>>         straightforward question.
>>
>>         -Aaron
>>
>>         On 12/15/17 3:48 PM, Alex Chekholko wrote:
>>         > Hey Aaron,
>>         >
>>         > Can you define your sizes for "large blocks" and "small
>>         files"?? If you
>>         > dial one up and the other down, your performance will be
>>         worse.? And in
>>         > any case it's a pathological corner case so it shouldn't
>>         matter much for
>>         > your workflow, unless you've designed your system with the
>>         wrong values.
>>         >
>>         > For example, for bioinformatics workloads, I prefer to use 256KB
>>         > filesystem block size, and I'd consider 4MB+ to be "large
>>         block size",
>>         > which would make the filesystem obviously unsuitable for
>>         processing
>>         > millions of 8KB files.
>>         >
>>         > You can make a histogram of file sizes in your existing
>>         filesystems and
>>         > then make your subblock size (1/32 of block size) on the
>>         smaller end of
>>         > that.? ?Also definitely use the "small file in inode"
>>         feature and put
>>         > your metadata on SSD.
>>         >
>>         > Regards,
>>         > Alex
>>         >
>>         > On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
>>         > <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>
>>         <mailto:aaron.s.knister at nasa.gov
>>         <mailto:aaron.s.knister at nasa.gov>>> wrote:
>>         >
>>         >? ? ?Thanks, Bill.
>>         >
>>         >? ? ?I still don't feel like I've got an clear answer from
>>         IBM and frankly
>>         >? ? ?the core issue of a lack of migration tool was totally
>>         dodged.
>>         >
>>         >? ? ?Again in Sven's presentation from SSUG @ SC17
>>         >? ?
>>         ?(http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf
>>         <http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
>>         >? ? ?he mentions "It has a significant performance penalty
>>         for small files in
>>         >? ? ?large block size filesystems" and the demonstrates that
>>         with several
>>         >? ? ?mdtest runs (which show the effect with and without the >32
>>         >? ? ?subblocks code):
>>         >
>>         >
>>         >? ? ?4.2.1 base code - SUMMARY: (of 3 iterations)
>>         >? ? ?File creation : Mean = 2237.644
>>         >
>>         >? ? ?zero-end-of-file-padding (4.2.2 + ifdef for zero
>>         padding):? SUMMARY: (of
>>         >? ? ?3 iterations)
>>         >? ? ?File creation : Mean = 12866.842
>>         >
>>         >? ? ?more sub blocks per block (4.2.2 + morethan32subblock code):
>>         >? ? ?File creation : Mean = 40316.721
>>         >
>>         >? ? ?Can someone (ideally Sven) give me a straight answer as
>>         to whether or
>>         >? ? ?not the > 32 subblock code actually makes a performance
>>         difference for
>>         >? ? ?small files in large block filesystems? And if not, help
>>         me understand
>>         >? ? ?why his slides and provided benchmark data have
>>         consistently indicated
>>         >? ? ?it does?
>>         >
>>         >? ? ?-Aaron
>>         >
>>         >? ? ?On 12/1/17 11:44 AM, Bill Hartner wrote:
>>         >? ? ?> ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported
>>         some of the
>>         >? ? ?> results @ user group meeting.
>>         >? ? ?>
>>         >? ? ?> -Bill
>>         >? ? ?>
>>         >? ? ?> Bill Hartner
>>         >? ? ?> IBM Systems
>>         >? ? ?> Scalable I/O Development
>>         >? ? ?> Austin, Texas
>>         >? ? ?> bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
>>         <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>
>>         >? ? ?> home office 512-784-0980 <tel:(512)%20784-0980>
>>         <tel:512-784-0980 <tel:(512)%20784-0980>>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> Inactive hide details for Jan-Frode Myklebust
>>         ---12/01/2017 06:53:44
>>         >? ? ?> AM---Bill, could you say something about what the
>>         metadataJan-Frode
>>         >? ? ?> Myklebust ---12/01/2017 06:53:44 AM---Bill, could you
>>         say something
>>         >? ? ?> about what the metadata-storage here was?
>>         ESS/NL-SAS/3way replication?
>>         >? ? ?>
>>         >? ? ?> From: Jan-Frode Myklebust <janfrode at tanso.net
>>         <mailto:janfrode at tanso.net> <mailto:janfrode at tanso.net
>>         <mailto:janfrode at tanso.net>>>
>>         >? ? ?> To: gpfsug main discussion list
>>         <gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>>>
>>         >? ? ?> Date: 12/01/2017 06:53 AM
>>         >? ? ?> Subject: Re: [gpfsug-discuss] Online data migration tool
>>         >? ? ?> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ?
>>         ?------------------------------------------------------------------------
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> Bill, could you say something about what the
>>         metadata-storage here was?
>>         >? ? ?> ESS/NL-SAS/3way replication?
>>         >? ? ?>
>>         >? ? ?> I just asked about this in the internal slack channel
>>         #scale-help today..
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> -jf
>>         >? ? ?>
>>         >? ? ?> fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
>>         <_bhartner at us.ibm.com_
>>         >? ? ?> <mailto:bhartner at us.ibm.com
>>         <mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com
>>         <mailto:bhartner at us.ibm.com>>>>:
>>         >? ? ?>
>>         >? ? ?>? ? ?> "It has a significant performance penalty for
>>         small files in
>>         >? ? ?large
>>         >? ? ?>? ? ?> block size filesystems"
>>         >? ? ?>
>>         >? ? ?>? ? ?Aaron,
>>         >? ? ?>
>>         >? ? ?>? ? ?Below are mdtest results for a test we ran for
>>         CORAL - file
>>         >? ? ?size was
>>         >? ? ?>? ? ?32k.
>>         >? ? ?>
>>         >? ? ?>? ? ?We have not gone back and ran the test on a file
>>         system formatted
>>         >? ? ?>? ? ?without > 32 subblocks. We'll do that at some point...
>>         >? ? ?>
>>         >? ? ?>? ? ?-Bill
>>         >? ? ?>
>>         >? ? ?>? ? ?-- started at 10/28/2017 17:51:38 --
>>         >? ? ?>
>>         >? ? ?>? ? ?mdtest-1.9.3 was launched with 228 total task(s)
>>         on 12 node(s)
>>         >? ? ?>? ? ?Command line used: /tmp/mdtest-binary-dir/mdtest -d
>>         >? ? ?>? ? ?/ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w
>>         32768 -C -F -r
>>         >? ? ?-p 360
>>         >? ? ?>? ? ?-u -y
>>         >? ? ?>? ? ?Path: /ibm/fs2-16m-10
>>         >? ? ?>? ? ?FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used
>>         Inodes: 0.0%
>>         >? ? ?>
>>         >? ? ?>? ? ?228 tasks, 67239936 files
>>         >? ? ?>
>>         >? ? ?>? ? ?SUMMARY: (of 3 iterations)
>>         >? ? ?>? ? ?Operation Max Min Mean Std Dev
>>         >? ? ?>? ? ?--------- --- --- ---- -------
>>         >? ? ?>? ? ?File creation : 51953.498 50558.517 51423.221 616.643
>>         >? ? ?>? ? ?File stat : 0.000 0.000 0.000 0.000
>>         >? ? ?>? ? ?File read : 0.000 0.000 0.000 0.000
>>         >? ? ?>? ? ?File removal : 96746.376 92149.535 94658.774 1900.187
>>         >? ? ?>? ? ?Tree creation : 1.588 0.070 0.599 0.700
>>         >? ? ?>? ? ?Tree removal : 0.213 0.034 0.097 0.082
>>         >? ? ?>
>>         >? ? ?>? ? ?-- finished at 10/28/2017 19:51:54 --
>>         >? ? ?>
>>         >? ? ?>? ? ?Bill Hartner
>>         >? ? ?>? ? ?IBM Systems
>>         >? ? ?>? ? ?Scalable I/O Development
>>         >? ? ?>? ? ?Austin, Texas_
>>         >? ? ?>? ? ?__bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
>>         <mailto:bhartner at us.ibm.com>
>>         >? ? ?<mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>
>>         >? ? ?>? ? ?home office 512-784-0980 <tel:(512)%20784-0980>
>>         <tel:512-784-0980 <tel:(512)%20784-0980>>
>>         >? ? ?>
>>         >? ? ?>? ? ?_
>>         >? ? ?>? ? ?__gpfsug-discuss-bounces at spectrumscale.org_
>>         >? ? ?>? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>>>?wrote on
>>         >? ? ?>? ? ?11/29/2017 04:41:48 PM:
>>         >? ? ?>
>>         >? ? ?>? ? ?> From: Aaron Knister <_aaron.knister at gmail.com_
>>         >? ? ?>? ? ?<mailto:aaron.knister at gmail.com
>>         <mailto:aaron.knister at gmail.com>
>>         <mailto:aaron.knister at gmail.com
>>         <mailto:aaron.knister at gmail.com>>>>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> To: gpfsug main discussion list
>>         >? ? ?>? ? ?<_gpfsug-discuss at spectrumscale.org_
>>         >? ? ?>? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>>>>
>>         >? ? ?>
>>         >? ? ?>? ? ?> Date: 11/29/2017 04:42 PM
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> Subject: Re: [gpfsug-discuss] Online data
>>         migration tool
>>         >? ? ?>? ? ?> Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>>         >? ? ?>? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>>>
>>         >? ? ?>
>>         >? ? ?>? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> Thanks, Nikhil. Most of that was consistent with
>>         my understnading,
>>         >? ? ?>? ? ?> however I was under the impression that the >32
>>         subblocks code is
>>         >? ? ?>? ? ?> required to achieve the touted 50k file
>>         creates/second that Sven has
>>         >? ? ?>? ? ?> talked about a bunch of times:
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         >? ?
>>         ??_http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>>         >? ?
>>         ?<http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         >? ?
>>         ??_http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
>>         >? ?
>>         ?<http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>>         >? ? ?>? ? ?> _Sven_Oehme_-_News_from_Research.pdf
>>         >? ? ?>? ? ?>
>>         _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>>         >? ? ?<http://files.gpfsug.org/presentations/2016/SC16/12_-_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>>         >? ? ?>? ? ?>
>>         _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> from those presentations regarding 32 subblocks:
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> "It has a significant performance penalty for
>>         small files in large
>>         >? ? ?>? ? ?> block size filesystems"
>>         >? ? ?>
>>         >? ? ?>? ? ?> although I'm not clear on the specific
>>         definition of "large". Many
>>         >? ? ?>? ? ?> filesystems I encounter only have a 1M block
>>         size so it may not
>>         >? ? ?>? ? ?> matter there, although that same presentation
>>         clearly shows the
>>         >? ? ?>? ? ?> benefit of larger block sizes which is yet
>>         *another* thing for which
>>         >? ? ?>? ? ?> a migration tool would be helpful.
>>         >? ? ?>
>>         >? ? ?>? ? ?> -Aaron
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>>         >? ? ?>? ? ?<_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
>>         <mailto:nikhilk at us.ibm.com>
>>         >? ? ?<mailto:nikhilk at us.ibm.com
>>         <mailto:nikhilk at us.ibm.com>>>> wrote:
>>         >? ? ?>
>>         >? ? ?>? ? ?> Hi,
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> I would like to clarify migration path to 5.0.0
>>         from 4.X.X
>>         >? ? ?clusters.
>>         >? ? ?>? ? ?> For all Spectrum Scale clusters that are
>>         currently at 4.X.X,
>>         >? ? ?it is
>>         >? ? ?>? ? ?> possible to migrate to 5.0.0 with no offline
>>         data migration
>>         >? ? ?and no
>>         >? ? ?>? ? ?> need to move data. Once these clusters are at
>>         5.0.0, they will
>>         >? ? ?>? ? ?> benefit from the performance improvements, new
>>         features (such as
>>         >? ? ?>? ? ?> file audit logging), and various enhancements
>>         that are
>>         >? ? ?included in
>>         >? ? ?>? ? ?5.0.0.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> That being said, there is one enhancement that
>>         will not be
>>         >? ? ?applied
>>         >? ? ?>? ? ?> to these clusters, and that is the increased
>>         number of
>>         >? ? ?sub-blocks
>>         >? ? ?>? ? ?> per block for small file allocation. This means
>>         that for file
>>         >? ? ?>? ? ?> systems with a large block size and a lot of
>>         small files, the
>>         >? ? ?>? ? ?> overall space utilization will be the same it
>>         currently is
>>         >? ? ?in 4.X.X.
>>         >? ? ?>? ? ?> Since file systems created at 4.X.X and earlier
>>         used a block
>>         >? ? ?size
>>         >? ? ?>? ? ?> that kept this allocation in mind, there should
>>         be very little
>>         >? ? ?>? ? ?> impact on existing file systems.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> Outside of that one particular function, the
>>         remainder of the
>>         >? ? ?>? ? ?> performance improvements, metadata improvements,
>>         updated
>>         >? ? ?>? ? ?> compatibility, new functionality, and all of the
>>         other
>>         >? ? ?enhancements
>>         >? ? ?>? ? ?> will be immediately available to you once you
>>         complete the
>>         >? ? ?upgrade
>>         >? ? ?>? ? ?> to 5.0.0 -- with no need to reformat, move data,
>>         or take
>>         >? ? ?your data
>>         >? ? ?>? ? ?offline.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> I hope that clarifies things a little and makes
>>         the upgrade path
>>         >? ? ?>? ? ?> more accessible.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> Please let me know if there are any other
>>         questions or concerns.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> Thank you,
>>         >? ? ?>? ? ?> Nikhil Khandelwal
>>         >? ? ?>? ? ?> Spectrum Scale Development
>>         >? ? ?>? ? ?> Client Adoption
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> _______________________________________________
>>         >? ? ?>? ? ?> gpfsug-discuss mailing list
>>         >? ? ?>? ? ?> gpfsug-discuss at _spectrumscale.org_
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>         >? ? ?>? ? ?> _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>>         >? ? ?>
>>         >? ? ?>? ? ?> _______________________________________________
>>         >? ? ?>? ? ?> gpfsug-discuss mailing list
>>         >? ? ?>? ? ?> gpfsug-discuss at _spectrumscale.org_
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>         >? ? ?>
>>         >? ? ?>? ? ?> _https://urldefense.proofpoint.com/v2/url?_
>>         >? ? ?<https://urldefense.proofpoint.com/v2/url?_>
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         ?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         ?siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>>         >? ? ?>
>>         >? ? ?>? ? ?_______________________________________________
>>         >? ? ?>? ? ?gpfsug-discuss mailing list
>>         >? ? ?>? ? ?gpfsug-discuss at _spectrumscale.org_
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>>         >? ? ?>? ? ?__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>>         >? ? ?>? ? ?gpfsug-discuss mailing list
>>         >? ? ?>? ? ?gpfsug-discuss at spectrumscale.org
>>         <http://spectrumscale.org> <http://spectrumscale.org>
>>         >? ? ?>? ?
>>         >? ?
>>         ??https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> _______________________________________________
>>         >? ? ?> gpfsug-discuss mailing list
>>         >? ? ?> gpfsug-discuss at spectrumscale.org
>>         <http://spectrumscale.org> <http://spectrumscale.org>
>>         >? ? ?> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>>         >? ? ?>
>>         >
>>         >? ? ?--
>>         >? ? ?Aaron Knister
>>         >? ? ?NASA Center for Climate Simulation (Code 606.2)
>>         >? ? ?Goddard Space Flight Center
>>         >? ? ?(301) 286-2776 <tel:(301)%20286-2776>
>>         <tel:%28301%29%20286-2776>
>>         >? ? ?_______________________________________________
>>         >? ? ?gpfsug-discuss mailing list
>>         >? ? ?gpfsug-discuss at spectrumscale.org
>>         <http://spectrumscale.org> <http://spectrumscale.org>
>>         >? ? ?http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>>         >
>>         >
>>         >
>>         >
>>         > _______________________________________________
>>         > gpfsug-discuss mailing list
>>         > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>>         > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>         >
>>
>>         --
>>         Aaron Knister
>>         NASA Center for Climate Simulation (Code 606.2)
>>         Goddard Space Flight Center
>>         (301) 286-2776 <tel:(301)%20286-2776>
>>         _______________________________________________
>>         gpfsug-discuss mailing list
>>         gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776


From bipcuds at gmail.com  Tue Dec 19 23:08:57 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Tue, 19 Dec 2017 18:08:57 -0500
Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the
	Spectrum Scale FAQ
Message-ID: <CAAxuGpF6nw6R_Zmowx3Xv_pUbxikgvTXeR7g-p08TpMvh_ZgzQ@mail.gmail.com>

Hi All,

I was recently trying to determine the latest RHEL release that will work
with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code
uses Scale 4.1.0.8.

A specific X.Y GSS code build, from my experience, is intended to use a
specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken),
which no longer has EUS support from RedHat (only 6.7 still does).

GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be
used, which is a surprising statement given GSS/ESS code's sensitivity to
OS levels (any ESS I have ever seen has never been supported on more than
one version of RHEL).

According to the Scale FAQ (
https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux),
A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying
RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported
by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From
V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27!

Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install
GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is
supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option
here).

Many Thanks,
  Keith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/fa1226b4/attachment.htm>

From olaf.weiser at de.ibm.com  Wed Dec 20 07:35:26 2017
From: olaf.weiser at de.ibm.com (Olaf Weiser)
Date: Wed, 20 Dec 2017 08:35:26 +0100
Subject: [gpfsug-discuss] pmcollector and NSD perf
In-Reply-To: <DM5PR05MB3577119DC6FE73462E83BC72E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
References: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com><OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>
	<DM5PR05MB3577119DC6FE73462E83BC72E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
Message-ID: <OFF3295D55.B6D7C020-ONC12581FC.0028AADD-C12581FC.0029B20E@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0001.gif>

From john.hearns at asml.com  Wed Dec 20 08:41:07 2017
From: john.hearns at asml.com (John Hearns)
Date: Wed, 20 Dec 2017 08:41:07 +0000
Subject: [gpfsug-discuss] Expeliarmus
Message-ID: <HE1PR02MB1450E823234415275F6EADA3880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/a9561284/attachment.htm>

From r.sobey at imperial.ac.uk  Wed Dec 20 08:58:20 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Wed, 20 Dec 2017 08:58:20 +0000
Subject: [gpfsug-discuss] Expeliarmus
In-Reply-To: <HE1PR02MB1450E823234415275F6EADA3880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
References: <HE1PR02MB1450E823234415275F6EADA3880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
Message-ID: <DB6PR0601MB21012910B19A735F4F5835DCDF0C0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Don't take it personally, I'm not entitled to even download 4.X. I can't be bothered to jump through the hoops necessary to get it going.

Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of John Hearns
Sent: 20 December 2017 08:41
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Expeliarmus

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/b4f62495/attachment.htm>

From S.J.Thompson at bham.ac.uk  Wed Dec 20 09:31:25 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Wed, 20 Dec 2017 09:31:25 +0000
Subject: [gpfsug-discuss] Expeliarmus
Message-ID: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>

This is assuming you directly have ?IBM? licenses (as opposed to OEM licenses where the route is different ? or where the licenses are held by your VAR rather than you ?)

You need to have an IBM account which is attached to a (Passport Advantage) PA site that has current support for the product?

If you go to fix central, its also listed there now as well so again assuming your account is appropriately attached to a support contract, you can download it.

If you are struggling with these, then go speak to your business partner or IBM account manager.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "john.hearns at asml.com" <john.hearns at asml.com>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, 20 December 2017 at 08:41
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Expeliarmus

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/376c5117/attachment.htm>

From john.hearns at asml.com  Wed Dec 20 13:02:54 2017
From: john.hearns at asml.com (John Hearns)
Date: Wed, 20 Dec 2017 13:02:54 +0000
Subject: [gpfsug-discuss] Expeliarmus
In-Reply-To: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>
References: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>
Message-ID: <HE1PR02MB1450FFCA4DEC69B3E2F70E4A880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>

Thnkyou both. I tapped on the door of Fix Central with my wand, and now have the eye of newt, wing of bat and a Spectrum Scale 5 installer.


From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support)
Sent: Wednesday, December 20, 2017 10:31 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Expeliarmus

This is assuming you directly have ?IBM? licenses (as opposed to OEM licenses where the route is different ? or where the licenses are held by your VAR rather than you ?)

You need to have an IBM account which is attached to a (Passport Advantage) PA site that has current support for the product?

If you go to fix central, its also listed there now as well so again assuming your account is appropriately attached to a support contract, you can download it.

If you are struggling with these, then go speak to your business partner or IBM account manager.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "john.hearns at asml.com<mailto:john.hearns at asml.com>" <john.hearns at asml.com<mailto:john.hearns at asml.com>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, 20 December 2017 at 08:41
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] Expeliarmus

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/55a82b6a/attachment.htm>

From ewahl at osc.edu  Wed Dec 20 14:53:20 2017
From: ewahl at osc.edu (Edward Wahl)
Date: Wed, 20 Dec 2017 09:53:20 -0500
Subject: [gpfsug-discuss] Expeliarmus
In-Reply-To: <HE1PR02MB1450FFCA4DEC69B3E2F70E4A880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
References: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>
	<HE1PR02MB1450FFCA4DEC69B3E2F70E4A880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
Message-ID: <20171220095320.1093db0d@osc.edu>


And it's not JUST a PA account. 

  You also have to have "someone" convert your old
GPFS part numbers into Spectrum Scale part numbers so that Passport Advantage
will show you the entitlements and downloads properly.  I went through this,
almost not painful at all, but rather slow.

Ed


On Wed, 20 Dec 2017 13:02:54 +0000
John Hearns <john.hearns at asml.com> wrote:

> Thnkyou both. I tapped on the door of Fix Central with my wand, and now have
> the eye of newt, wing of bat and a Spectrum Scale 5 installer.
> 
> 
> From: gpfsug-discuss-bounces at spectrumscale.org
> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson
> (IT Research Support) Sent: Wednesday, December 20, 2017 10:31 AM To: gpfsug
> main discussion list <gpfsug-discuss at spectrumscale.org> Subject: Re:
> [gpfsug-discuss] Expeliarmus
> 
> This is assuming you directly have ?IBM? licenses (as opposed to OEM licenses
> where the route is different ? or where the licenses are held by your VAR
> rather than you ?)
> 
> You need to have an IBM account which is attached to a (Passport Advantage)
> PA site that has current support for the product?
> 
> If you go to fix central, its also listed there now as well so again assuming
> your account is appropriately attached to a support contract, you can
> download it.
> 
> If you are struggling with these, then go speak to your business partner or
> IBM account manager.
> 
> Simon
> 
> From:
> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>
> on behalf of "john.hearns at asml.com<mailto:john.hearns at asml.com>"
> <john.hearns at asml.com<mailto:john.hearns at asml.com>> Reply-To:
> "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>"
> <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
> Date: Wednesday, 20 December 2017 at 08:41 To:
> "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>"
> <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
> Subject: [gpfsug-discuss] Expeliarmus
> 
> I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
> For the life of me I cannot summon the spell needed to put Spectrum Scale 5
> on my orders list.
> 
> Can some kindly witch give me the incantation please?
> I would like to install on a test cluster, as the wisdom of the mages and
> mavens here has it.
> 
> -- The information contained in this communication and any attachments is
> confidential and may be privileged, and is for the sole use of the intended
> recipient(s). Any unauthorized review, use, disclosure or distribution is
> prohibited. Unless explicitly stated otherwise in the body of this
> communication or the attachment thereto (if any), the information is provided
> on an AS-IS basis without any express or implied warranties or liabilities.
> To the extent you are relying on this information, you are doing so at your
> own risk. If you are not the intended recipient, please notify the sender
> immediately by replying to this message and destroy all copies of this
> message and any attachments. Neither the sender nor the company/group of
> companies he or she represents shall be liable for the proper and complete
> transmission of the information contained in this communication, or for any
> delay in its receipt. -- The information contained in this communication and
> any attachments is confidential and may be privileged, and is for the sole
> use of the intended recipient(s). Any unauthorized review, use, disclosure or
> distribution is prohibited. Unless explicitly stated otherwise in the body of
> this communication or the attachment thereto (if any), the information is
> provided on an AS-IS basis without any express or implied warranties or
> liabilities. To the extent you are relying on this information, you are doing
> so at your own risk. If you are not the intended recipient, please notify the
> sender immediately by replying to this message and destroy all copies of this
> message and any attachments. Neither the sender nor the company/group of
> companies he or she represents shall be liable for the proper and complete
> transmission of the information contained in this communication, or for any
> delay in its receipt.



-- 

Ed Wahl
Ohio Supercomputer Center
614-292-9302


From eric.wonderley at vt.edu  Wed Dec 20 19:40:27 2017
From: eric.wonderley at vt.edu (J. Eric Wonderley)
Date: Wed, 20 Dec 2017 14:40:27 -0500
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same host
Message-ID: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>

Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all
mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core
switch.  I'm somewhat fearful as to how this is implemented in the juniper
switch we are about to test.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/3cf2f760/attachment.htm>

From aaron.s.knister at nasa.gov  Wed Dec 20 20:25:04 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Wed, 20 Dec 2017 20:25:04 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
 host
In-Reply-To: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
References: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
Message-ID: <A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>



We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper.

Are you attempting to do RoCE or just plain TCP/IP?


On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu> wrote:
Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/a0eb8e44/attachment.htm>

From eric.wonderley at vt.edu  Wed Dec 20 20:37:09 2017
From: eric.wonderley at vt.edu (J. Eric Wonderley)
Date: Wed, 20 Dec 2017 15:37:09 -0500
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
	host
In-Reply-To: <A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
References: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
Message-ID: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>

Just plain tcpip.

We have dual port connectx4s in our nsd servers.  Upon adding a second
connectx4 hba...no links go up or show "up".  I have one port on each hba
configured for eth and ibv_devinfo looks sane.

I cannot find anything indicating that this should not work.  I have a
ticket opened with mellanox.

On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER
SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:

>
>
> We?ve done a fair amount of VPI work but admittedly not with connectx4. Is
> it possible the cards are trying to talk IB rather than Eth? I figured
> you?re Ethernet based because of the mention of Juniper.
>
> Are you attempting to do RoCE or just plain TCP/IP?
>
>
> On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <
> eric.wonderley at vt.edu> wrote:
>
> Hello:
>
> Does anyone have this type of config?
>
> The host configuration looks sane but we seem to observe link-down on all
> mlx adapters no matter what we do.
>
> Big picture is that we are attempting to do mc(multichassis)-lags to a
> core switch.  I'm somewhat fearful as to how this is implemented in the
> juniper switch we are about to test.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/f26c0b63/attachment.htm>

From S.J.Thompson at bham.ac.uk  Wed Dec 20 20:45:37 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Wed, 20 Dec 2017 20:45:37 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in
	same	host
In-Reply-To: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
References: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>,
	<CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A42875@EX13.adf.bham.ac.uk>

I can't remember if this was on mlx4 or mlx5 driver cards, but we found we had to use LINKDELAY=20 when using bonding for Ethernet.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of J. Eric Wonderley [eric.wonderley at vt.edu]
Sent: 20 December 2017 20:37
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same      host

Just plain tcpip.

We have dual port connectx4s in our nsd servers.  Upon adding a second connectx4 hba...no links go up or show "up".  I have one port on each hba configured for eth and ibv_devinfo looks sane.

I cannot find anything indicating that this should not work.  I have a ticket opened with mellanox.

On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>> wrote:


We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper.

Are you attempting to do RoCE or just plain TCP/IP?


On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu<mailto:eric.wonderley at vt.edu>> wrote:
Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test.

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




From abeattie at au1.ibm.com  Wed Dec 20 21:46:53 2017
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Wed, 20 Dec 2017 21:46:53 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in
	samehost
In-Reply-To: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>,
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com><A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
Message-ID: <OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/4a535991/attachment.htm>

From Paul.Sanchez at deshaw.com  Wed Dec 20 21:53:10 2017
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 20 Dec 2017 21:53:10 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter
	in	samehost
In-Reply-To: <OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>,
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com><A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
	<OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
Message-ID: <55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>

We have run multiple ConnectX-4 NICs in bonded MLAG (Arista) and VPC (Cisco) switch configurations on our NSD servers.  We used to see issues with firmware versions that didn?t support the optics we wanted to use (e.g. early CX3/CX4 and Cisco 40G-BiDi).  You may also want check mstflint to see whether the firmware levels match on the MLX cards, and if you upgrade firmware in some cases a power-cycle (not reboot) can be required to finish the process.

-Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Andrew Beattie
Sent: Wednesday, December 20, 2017 4:47 PM
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in samehost

IBM ESS building blocks can have up to 3 dual port 10GBEth, 40GB Eth, 56GB IB, 100GB IB Mlx adapater cards, because we have 2 IO nodes this is up to a total of 12 ports per building block
so there should not be any reason for this to fail.

I regularly see a Mix of 10GB / 40GB or 10GB / IB configurations



Regards
Andrew Beattie
Software Defined Storage  - IT Specialist
Phone: 614-2133-7927
E-mail: abeattie at au1.ibm.com<mailto:abeattie at au1.ibm.com>


----- Original message -----
From: "J. Eric Wonderley" <eric.wonderley at vt.edu<mailto:eric.wonderley at vt.edu>>
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc:
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same host
Date: Thu, Dec 21, 2017 6:37 AM

Just plain tcpip.

We have dual port connectx4s in our nsd servers.  Upon adding a second connectx4 hba...no links go up or show "up".  I have one port on each hba configured for eth and ibv_devinfo looks sane.

I cannot find anything indicating that this should not work.  I have a ticket opened with mellanox.

On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>> wrote:



We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper.

Are you attempting to do RoCE or just plain TCP/IP?

On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu<mailto:eric.wonderley at vt.edu>> wrote:
Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test.

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=I5Dq2T7aYvC87Wp12fsz6CRLw4uo2-RVnrnpxRYfYuA&e=>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/0b3daff0/attachment.htm>

From david_johnson at brown.edu  Wed Dec 20 22:00:49 2017
From: david_johnson at brown.edu (David D Johnson)
Date: Wed, 20 Dec 2017 17:00:49 -0500
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter
	in	samehost
In-Reply-To: <55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
	<OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
	<55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>
Message-ID: <D14EB35E-865A-400D-B81A-60FAC4002D22@brown.edu>

We're trying to get 40 gbe connection between Mellanox switches and Cisco switches down at the other end of the machine room  
The BiDi part seems to be the best given about 30m  run on multimode.  However Mellanox support says it's not supported.
Want to use this to get close to IB speeds for GPFS on nodes that aren't on the IB fabric.  
Does anyone have any luck getting 40 or 100 gig at 20-30m when the switches are different brands?

Thanks,
 -- ddj

> On Dec 20, 2017, at 4:53 PM, Sanchez, Paul <Paul.Sanchez at deshaw.com> wrote:
> 
> We have run multiple ConnectX-4 NICs in bonded MLAG (Arista) and VPC (Cisco) switch configurations on our NSD servers.  We used to see issues with firmware versions that didn?t support the optics we wanted to use (e.g. early CX3/CX4 and Cisco 40G-BiDi).  You may also want check mstflint to see whether the firmware levels match on the MLX cards, and if you upgrade firmware in some cases a power-cycle (not reboot) can be required to finish the process.
>  
> -Paul
> ? <>
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org]On Behalf Of Andrew Beattie
> Sent: Wednesday, December 20, 2017 4:47 PM
> To: gpfsug-discuss at spectrumscale.org
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in samehost
>  
> IBM ESS building blocks can have up to 3 dual port 10GBEth, 40GB Eth, 56GB IB, 100GB IB Mlx adapater cards, because we have 2 IO nodes this is up to a total of 12 ports per building block
> so there should not be any reason for this to fail.
>  
> I regularly see a Mix of 10GB / 40GB or 10GB / IB configurations
>  
>  
>  
> Regards
> Andrew Beattie
> Software Defined Storage  - IT Specialist
> Phone: 614-2133-7927
> E-mail: abeattie at au1.ibm.com <mailto:abeattie at au1.ibm.com>
>  
>  
> ----- Original message -----
> From: "J. Eric Wonderley" <eric.wonderley at vt.edu <mailto:eric.wonderley at vt.edu>>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org <mailto:gpfsug-discuss at spectrumscale.org>>
> Cc:
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same host
> Date: Thu, Dec 21, 2017 6:37 AM
>   
> Just plain tcpip.
>  
> We have dual port connectx4s in our nsd servers.  Upon adding a second connectx4 hba...no links go up or show "up".  I have one port on each hba configured for eth and ibv_devinfo looks sane.
>  
> I cannot find anything indicating that this should not work.  I have a ticket opened with mellanox.
>  
> On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote: 
>  
> 
> 
> We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper. 
>  
> Are you attempting to do RoCE or just plain TCP/IP?
>  
> On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu <mailto:eric.wonderley at vt.edu>> wrote:
> Hello:
>  
> Does anyone have this type of config?
>  
> The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.
>  
> Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test. 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=I5Dq2T7aYvC87Wp12fsz6CRLw4uo2-RVnrnpxRYfYuA&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>
>  
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/2b7ae14b/attachment.htm>

From oehmes at gmail.com  Thu Dec 21 01:09:19 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Thu, 21 Dec 2017 01:09:19 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in
	samehost
In-Reply-To: <D14EB35E-865A-400D-B81A-60FAC4002D22@brown.edu>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
	<OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
	<55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>
	<D14EB35E-865A-400D-B81A-60FAC4002D22@brown.edu>
Message-ID: <CALssuR3vrn_DO0o+mnquTNObOm=pALfxic1z4miMYnEwbC2vDw@mail.gmail.com>

i don't know if that works with cisco, but i use 50 an 100m cables for 40
as well as 100Gbit in my lab between 2 Mellanox switches :
http://www.mellanox.com/products/interconnect/ethernet-active-optical-cables.php
as paul pointed out one of the very first things one needs to do after
adding an adapter is to flash the firmware to a recent level. especially of
you have 2 adapters with different FW i have seen even once with higher
level not work properly, so before you do anything else get them to a
recent level and especially the same if its the same adapter types.

sven

On Wed, Dec 20, 2017 at 10:01 PM David D Johnson <david_johnson at brown.edu>
wrote:

> We're trying to get 40 gbe connection between Mellanox switches and Cisco
> switches down at the other end of the machine room
> The BiDi part seems to be the best given about 30m  run on multimode.
> However Mellanox support says it's not supported.
> Want to use this to get close to IB speeds for GPFS on nodes that aren't
> on the IB fabric.
> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
> switches are different brands?
>
> Thanks,
>  -- ddj
>
> On Dec 20, 2017, at 4:53 PM, Sanchez, Paul <Paul.Sanchez at deshaw.com>
> wrote:
>
> We have run multiple ConnectX-4 NICs in bonded MLAG (Arista) and VPC
> (Cisco) switch configurations on our NSD servers.  We used to see issues
> with firmware versions that didn?t support the optics we wanted to use
> (e.g. early CX3/CX4 and Cisco 40G-BiDi).  You may also want check mstflint
> to see whether the firmware levels match on the MLX cards, and if you
> upgrade firmware in some cases a power-cycle (not reboot) can be required
> to finish the process.
>
> -Paul
>
> *From:* gpfsug-discuss-bounces at spectrumscale.org [
> mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>]*On Behalf Of *Andrew Beattie
> *Sent:* Wednesday, December 20, 2017 4:47 PM
> *To:* gpfsug-discuss at spectrumscale.org
> *Subject:* Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in
> samehost
>
> IBM ESS building blocks can have up to 3 dual port 10GBEth, 40GB Eth, 56GB
> IB, 100GB IB Mlx adapater cards, because we have 2 IO nodes this is up to a
> total of 12 ports per building block
> so there should not be any reason for this to fail.
>
> I regularly see a Mix of 10GB / 40GB or 10GB / IB configurations
>
>
>
> Regards
> *Andrew Beattie*
> *Software Defined Storage  - IT Specialist*
> *Phone: *614-2133-7927
> *E-mail: *abeattie at au1.ibm.com
>
>
>
> ----- Original message -----
> From: "J. Eric Wonderley" <eric.wonderley at vt.edu>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
> host
> Date: Thu, Dec 21, 2017 6:37 AM
>
> Just plain tcpip.
>
> We have dual port connectx4s in our nsd servers.  Upon adding a second
> connectx4 hba...no links go up or show "up".  I have one port on each hba
> configured for eth and ibv_devinfo looks sane.
>
> I cannot find anything indicating that this should not work.  I have a
> ticket opened with mellanox.
>
> On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER
> SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:
>
>
>
>
> We?ve done a fair amount of VPI work but admittedly not with connectx4. Is
> it possible the cards are trying to talk IB rather than Eth? I figured
> you?re Ethernet based because of the mention of Juniper.
>
> Are you attempting to do RoCE or just plain TCP/IP?
>
> On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <
> eric.wonderley at vt.edu> wrote:
>
> Hello:
>
> Does anyone have this type of config?
>
> The host configuration looks sane but we seem to observe link-down on all
> mlx adapters no matter what we do.
>
> Big picture is that we are attempting to do mc(multichassis)-lags to a
> core switch.  I'm somewhat fearful as to how this is implemented in the
> juniper switch we are about to test.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=I5Dq2T7aYvC87Wp12fsz6CRLw4uo2-RVnrnpxRYfYuA&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/ef7b4e3c/attachment.htm>

From kraemerf at de.ibm.com  Thu Dec 21 07:07:24 2017
From: kraemerf at de.ibm.com (Frank Kraemer)
Date: Thu, 21 Dec 2017 08:07:24 +0100
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
	host
In-Reply-To: <mailman.3419.1513818573.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3419.1513818573.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <OFE99AF1B2.2CCA57A0-ONC12581FD.00272158-C12581FD.00272158@notes.na.collabserv.com>

David,

> We're trying to get 40 gbe connection between Mellanox switches and Cisco
> switches down at the other end of the machine room
> The BiDi part seems to be the best given about 30m  run on multimode.
> However Mellanox support says it's not supported.
> Want to use this to get close to IB speeds for GPFS on nodes that aren't
> on the IB fabric.
> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
> switches are different brands?

maybe that's a good reason to get in contact with the team from Interoptic.
They claim a good expertise for these kind of problems - feedback is good.
http://packetpushers.net/podcast/podcasts/show-360-all-about-optics-interoptic-sponsored/
https://interoptic.com/

Frank Kraemer
IBM Consulting IT Specialist  / Client Technical Architect
Am Weiher 24, 65451 Kelsterbach
mailto:kraemerf at de.ibm.com
voice: +49-(0)171-3043699 / +4970342741078
IBM Germany
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/31aeae52/attachment.htm>

From daniel.kidger at uk.ibm.com  Thu Dec 21 12:21:59 2017
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Thu, 21 Dec 2017 12:21:59 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
Message-ID: <OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>

My suggestion is that it is better to not think of the performance coming from having more than 32 sub-blocks but instead that the performance comes from smaller sub-blocks. The fact that there are now more of them in say a 4MB blocksize filesytem is just a side effect.

Daniel


 

 
 	
Dr Daniel Kidger 
IBM Technical Sales Specialist
Software Defined Solution Sales

+ 44-(0)7818 522 266 
daniel.kidger at uk.ibm.com

> On 19 Dec 2017, at 21:32, Aaron Knister <aaron.s.knister at nasa.gov> wrote:
> 
> Thanks, Sven. Understood!
> 
>> On 12/19/17 3:20 PM, Sven Oehme wrote:
>> Hi,
>> 
>> the zero padding was never promoted into a GA stream, it was an
>> experiment to proof we are on the right track when we eliminate the
>> overhead from client to NSD Server, but also showed that alone is not
>> good enough. the work for the client is the same compared to the >32
>> subblocks, but the NSD Server has more work as it can't pack as many
>> subblocks and therefore files into larger blocks, so you need to do more
>> writes to store the same number of files. 
>> thats why there is the additional substantial improvement  when we then
>> went to >32 subblocks. 
>> 
>> sven
>> 
>> On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
>> SCIENCE CORP] <aaron.s.knister at nasa.gov
>> <mailto:aaron.s.knister at nasa.gov>> wrote:
>> 
>>    Thanks Sven! That makes sense to me and is what I thought was the
>>    case which is why I was confused when I saw the reply to the thread
>>    that said the >32 subblocks code had no performance impact. 
>> 
>>    A couple more question for you? in your presentation there?s a
>>    benchmark that shows the file create performance without the zero
>>    padding. Since you mention this is done for security reasons was
>>    that feature ever promoted to a GA Scale release? I?m also wondering
>>    if you could explain the performance difference between the no zero
>>    padding code and the > 32 subblock code since given your the example
>>    of 32K files and 16MB block size I figure both cases ought to write
>>    the same amount to disk. 
>> 
>>    Thanks!
>> 
>>    -Aaron
>> 
>> 
>> 
>> 
>> 
>>    On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com
>>    <mailto:oehmes at gmail.com>> wrote:
>>>    i thought i answered that already, but maybe i just thought about
>>>    answering it and then forgot about it :-D
>>> 
>>>    so yes more than 32 subblocks per block significant increase the
>>>    performance of filesystems with small files, for the sake of the
>>>    argument let's say 32k in a large block filesystem again for sake
>>>    of argument say 16MB. 
>>> 
>>>    you probably ask why ? 
>>> 
>>>    if you create a file and write 32k into it in a pre 5.0.0 Version
>>>    16 MB filesystem your client actually doesn't write 32k to the NSD
>>>    Server, it writes 512k, because thats the subblock size and we
>>>    need to write the full subblock (for security reasons). so first
>>>    you waste significant memory on the client to cache that zero
>>>    padding, you waste network bandwidth and you waste NSD Server
>>>    cache because you store it there too. this means you overrun the
>>>    cache more quickly, means you start doing read/modify writes
>>>    earlier on all your nice large raid tracks... i guess you get the
>>>    story by now. 
>>> 
>>>    in fact,  if you have a good raid code that can drive really a lot
>>>    of bandwidth out of individual drives like a GNR system you get
>>>    more performance for small file writes as larger your blocksize
>>>    is, because we can 'pack' more files into larger i/os and
>>>    therefore turn a small file create workload into a bandwidth
>>>    workload, essentially exactly what we did and i demonstrated in
>>>    the CORAL presentation . 
>>> 
>>>    hope that makes this crystal clear now . 
>>> 
>>>    sven
>>> 
>>> 
>>> 
>>>    On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister
>>>    <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
>>> 
>>>        Thanks, Alex. I'm all too familiar with the trade offs between
>>>        large
>>>        blocks and small files and we do use pretty robust SSD storage
>>>        for our
>>>        metadata. We support a wide range of workloads and we have
>>>        some folks
>>>        with many small (<1M) files and other folks with many large
>>>        (>256MB) files.
>>> 
>>>        My point in this thread is that IBM has said over and over
>>>        again in
>>>        presentations that there is a significant performance gain
>>>        with the >32
>>>        subblocks code on filesystems with large block sizes (although
>>>        to your
>>>        point I'm not clear on exactly what large means since I didn't
>>>        define
>>>        large in this context). Therefore given that the >32 subblock
>>>        code gives
>>>        a significant performance gain one could reasonably assume
>>>        that having a
>>>        filesystem with >32 subblocks is required to see this gain
>>>        (rather than
>>>        just running the >32 subblocks code on an fs w/o > 32 subblocks).
>>> 
>>>        This lead me to ask about a migration tool because in my mind
>>>        if there's
>>>        a performance gain from having >32 subblocks on the FS I'd
>>>        like that
>>>        feature and having to manually copy 10's of PB to new hardware
>>>        to get
>>>        this performance boost is unacceptable. However, IBM can't
>>>        seem to make
>>>        up their mind about whether or not the >32 subblocks code
>>>        *actually*
>>>        provides a performance increase. This seems like a pretty
>>>        straightforward question.
>>> 
>>>        -Aaron
>>> 
>>>>        On 12/15/17 3:48 PM, Alex Chekholko wrote:
>>>> Hey Aaron,
>>>> 
>>>> Can you define your sizes for "large blocks" and "small
>>>        files"?  If you
>>>> dial one up and the other down, your performance will be
>>>        worse.  And in
>>>> any case it's a pathological corner case so it shouldn't
>>>        matter much for
>>>> your workflow, unless you've designed your system with the
>>>        wrong values.
>>>> 
>>>> For example, for bioinformatics workloads, I prefer to use 256KB
>>>> filesystem block size, and I'd consider 4MB+ to be "large
>>>        block size",
>>>> which would make the filesystem obviously unsuitable for
>>>        processing
>>>> millions of 8KB files.
>>>> 
>>>> You can make a histogram of file sizes in your existing
>>>        filesystems and
>>>> then make your subblock size (1/32 of block size) on the
>>>        smaller end of
>>>> that.   Also definitely use the "small file in inode"
>>>        feature and put
>>>> your metadata on SSD.
>>>> 
>>>> Regards,
>>>> Alex
>>>> 
>>>> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
>>>> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>
>>>        <mailto:aaron.s.knister at nasa.gov
>>>        <mailto:aaron.s.knister at nasa.gov>>> wrote:
>>>> 
>>>>      Thanks, Bill.
>>>> 
>>>>      I still don't feel like I've got an clear answer from
>>>        IBM and frankly
>>>>      the core issue of a lack of migration tool was totally
>>>        dodged.
>>>> 
>>>>      Again in Sven's presentation from SSUG @ SC17
>>>>    
>>>         (https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=>)
>>>>      he mentions "It has a significant performance penalty
>>>        for small files in
>>>>      large block size filesystems" and the demonstrates that
>>>        with several
>>>>      mdtest runs (which show the effect with and without the >32
>>>>      subblocks code):
>>>> 
>>>> 
>>>>      4.2.1 base code - SUMMARY: (of 3 iterations)
>>>>      File creation : Mean = 2237.644
>>>> 
>>>>      zero-end-of-file-padding (4.2.2 + ifdef for zero
>>>        padding):  SUMMARY: (of
>>>>      3 iterations)
>>>>      File creation : Mean = 12866.842
>>>> 
>>>>      more sub blocks per block (4.2.2 + morethan32subblock code):
>>>>      File creation : Mean = 40316.721
>>>> 
>>>>      Can someone (ideally Sven) give me a straight answer as
>>>        to whether or
>>>>      not the > 32 subblock code actually makes a performance
>>>        difference for
>>>>      small files in large block filesystems? And if not, help
>>>        me understand
>>>>      why his slides and provided benchmark data have
>>>        consistently indicated
>>>>      it does?
>>>> 
>>>>      -Aaron
>>>> 
>>>>      On 12/1/17 11:44 AM, Bill Hartner wrote:
>>>>      > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported
>>>        some of the
>>>>      > results @ user group meeting.
>>>>      >
>>>>      > -Bill
>>>>      >
>>>>      > Bill Hartner
>>>>      > IBM Systems
>>>>      > Scalable I/O Development
>>>>      > Austin, Texas
>>>>      > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
>>>        <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>
>>>>      > home office 512-784-0980 <tel:(512)%20784-0980>
>>>        <tel:512-784-0980 <tel:(512)%20784-0980>>
>>>>      >
>>>>      >
>>>>      > Inactive hide details for Jan-Frode Myklebust
>>>        ---12/01/2017 06:53:44
>>>>      > AM---Bill, could you say something about what the
>>>        metadataJan-Frode
>>>>      > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you
>>>        say something
>>>>      > about what the metadata-storage here was?
>>>        ESS/NL-SAS/3way replication?
>>>>      >
>>>>      > From: Jan-Frode Myklebust <janfrode at tanso.net
>>>        <mailto:janfrode at tanso.net> <mailto:janfrode at tanso.net
>>>        <mailto:janfrode at tanso.net>>>
>>>>      > To: gpfsug main discussion list
>>>        <gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>>>
>>>>      > Date: 12/01/2017 06:53 AM
>>>>      > Subject: Re: [gpfsug-discuss] Online data migration tool
>>>>      > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>>>>      >
>>>>      >
>>>>    
>>>         ------------------------------------------------------------------------
>>>>      >
>>>>      >
>>>>      >
>>>>      > Bill, could you say something about what the
>>>        metadata-storage here was?
>>>>      > ESS/NL-SAS/3way replication?
>>>>      >
>>>>      > I just asked about this in the internal slack channel
>>>        #scale-help today..
>>>>      >
>>>>      >
>>>>      >
>>>>      > -jf
>>>>      >
>>>>      > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
>>>        <_bhartner at us.ibm.com_
>>>>      > <mailto:bhartner at us.ibm.com
>>>        <mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com
>>>        <mailto:bhartner at us.ibm.com>>>>:
>>>>      >
>>>>      >     > "It has a significant performance penalty for
>>>        small files in
>>>>      large
>>>>      >     > block size filesystems"
>>>>      >
>>>>      >     Aaron,
>>>>      >
>>>>      >     Below are mdtest results for a test we ran for
>>>        CORAL - file
>>>>      size was
>>>>      >     32k.
>>>>      >
>>>>      >     We have not gone back and ran the test on a file
>>>        system formatted
>>>>      >     without > 32 subblocks. We'll do that at some point...
>>>>      >
>>>>      >     -Bill
>>>>      >
>>>>      >     -- started at 10/28/2017 17:51:38 --
>>>>      >
>>>>      >     mdtest-1.9.3 was launched with 228 total task(s)
>>>        on 12 node(s)
>>>>      >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>>>>      >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w
>>>        32768 -C -F -r
>>>>      -p 360
>>>>      >     -u -y
>>>>      >     Path: /ibm/fs2-16m-10
>>>>      >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used
>>>        Inodes: 0.0%
>>>>      >
>>>>      >     228 tasks, 67239936 files
>>>>      >
>>>>      >     SUMMARY: (of 3 iterations)
>>>>      >     Operation Max Min Mean Std Dev
>>>>      >     --------- --- --- ---- -------
>>>>      >     File creation : 51953.498 50558.517 51423.221 616.643
>>>>      >     File stat : 0.000 0.000 0.000 0.000
>>>>      >     File read : 0.000 0.000 0.000 0.000
>>>>      >     File removal : 96746.376 92149.535 94658.774 1900.187
>>>>      >     Tree creation : 1.588 0.070 0.599 0.700
>>>>      >     Tree removal : 0.213 0.034 0.097 0.082
>>>>      >
>>>>      >     -- finished at 10/28/2017 19:51:54 --
>>>>      >
>>>>      >     Bill Hartner
>>>>      >     IBM Systems
>>>>      >     Scalable I/O Development
>>>>      >     Austin, Texas_
>>>>      >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
>>>        <mailto:bhartner at us.ibm.com>
>>>>      <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>
>>>>      >     home office 512-784-0980 <tel:(512)%20784-0980>
>>>        <tel:512-784-0980 <tel:(512)%20784-0980>>
>>>>      >
>>>>      >     _
>>>>      >     __gpfsug-discuss-bounces at spectrumscale.org_
>>>>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>>> wrote on
>>>>      >     11/29/2017 04:41:48 PM:
>>>>      >
>>>>      >     > From: Aaron Knister <_aaron.knister at gmail.com_
>>>>      >     <mailto:aaron.knister at gmail.com
>>>        <mailto:aaron.knister at gmail.com>
>>>        <mailto:aaron.knister at gmail.com
>>>        <mailto:aaron.knister at gmail.com>>>>
>>>>      >
>>>>      >
>>>>      >     > To: gpfsug main discussion list
>>>>      >     <_gpfsug-discuss at spectrumscale.org_
>>>>      >     <mailto:gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>>>>
>>>>      >
>>>>      >     > Date: 11/29/2017 04:42 PM
>>>>      >
>>>>      >
>>>>      >     > Subject: Re: [gpfsug-discuss] Online data
>>>        migration tool
>>>>      >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>>>>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>>>
>>>>      >
>>>>      >     >
>>>>      >
>>>>      >     > Thanks, Nikhil. Most of that was consistent with
>>>        my understnading,
>>>>      >     > however I was under the impression that the >32
>>>        subblocks code is
>>>>      >     > required to achieve the touted 50k file
>>>        creates/second that Sven has
>>>>      >     > talked about a bunch of times:
>>>>      >     >
>>>>      >     >
>>>>      >   
>>>>    
>>>          _https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>>>>      >     >
>>>>      >   
>>>>    
>>>          _https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=>
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>>>>      >     > _Sven_Oehme_-_News_from_Research.pdf
>>>>      >     >
>>>        _https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>>>>      >     >
>>>        _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>>>>      >
>>>>      >
>>>>      >     > from those presentations regarding 32 subblocks:
>>>>      >     >
>>>>      >     > "It has a significant performance penalty for
>>>        small files in large
>>>>      >     > block size filesystems"
>>>>      >
>>>>      >     > although I'm not clear on the specific
>>>        definition of "large". Many
>>>>      >     > filesystems I encounter only have a 1M block
>>>        size so it may not
>>>>      >     > matter there, although that same presentation
>>>        clearly shows the
>>>>      >     > benefit of larger block sizes which is yet
>>>        *another* thing for which
>>>>      >     > a migration tool would be helpful.
>>>>      >
>>>>      >     > -Aaron
>>>>      >     >
>>>>      >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>>>>      >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
>>>        <mailto:nikhilk at us.ibm.com>
>>>>      <mailto:nikhilk at us.ibm.com
>>>        <mailto:nikhilk at us.ibm.com>>>> wrote:
>>>>      >
>>>>      >     > Hi,
>>>>      >     >
>>>>      >     > I would like to clarify migration path to 5.0.0
>>>        from 4.X.X
>>>>      clusters.
>>>>      >     > For all Spectrum Scale clusters that are
>>>        currently at 4.X.X,
>>>>      it is
>>>>      >     > possible to migrate to 5.0.0 with no offline
>>>        data migration
>>>>      and no
>>>>      >     > need to move data. Once these clusters are at
>>>        5.0.0, they will
>>>>      >     > benefit from the performance improvements, new
>>>        features (such as
>>>>      >     > file audit logging), and various enhancements
>>>        that are
>>>>      included in
>>>>      >     5.0.0.
>>>>      >     >
>>>>      >     > That being said, there is one enhancement that
>>>        will not be
>>>>      applied
>>>>      >     > to these clusters, and that is the increased
>>>        number of
>>>>      sub-blocks
>>>>      >     > per block for small file allocation. This means
>>>        that for file
>>>>      >     > systems with a large block size and a lot of
>>>        small files, the
>>>>      >     > overall space utilization will be the same it
>>>        currently is
>>>>      in 4.X.X.
>>>>      >     > Since file systems created at 4.X.X and earlier
>>>        used a block
>>>>      size
>>>>      >     > that kept this allocation in mind, there should
>>>        be very little
>>>>      >     > impact on existing file systems.
>>>>      >     >
>>>>      >     > Outside of that one particular function, the
>>>        remainder of the
>>>>      >     > performance improvements, metadata improvements,
>>>        updated
>>>>      >     > compatibility, new functionality, and all of the
>>>        other
>>>>      enhancements
>>>>      >     > will be immediately available to you once you
>>>        complete the
>>>>      upgrade
>>>>      >     > to 5.0.0 -- with no need to reformat, move data,
>>>        or take
>>>>      your data
>>>>      >     offline.
>>>>      >     >
>>>>      >     > I hope that clarifies things a little and makes
>>>        the upgrade path
>>>>      >     > more accessible.
>>>>      >     >
>>>>      >     > Please let me know if there are any other
>>>        questions or concerns.
>>>>      >     >
>>>>      >     > Thank you,
>>>>      >     > Nikhil Khandelwal
>>>>      >     > Spectrum Scale Development
>>>>      >     > Client Adoption
>>>>      >     >
>>>>      >     > _______________________________________________
>>>>      >     > gpfsug-discuss mailing list
>>>>      >     > gpfsug-discuss at _spectrumscale.org_
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>>>      >     > _https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>>>>      >
>>>>      >     > _______________________________________________
>>>>      >     > gpfsug-discuss mailing list
>>>>      >     > gpfsug-discuss at _spectrumscale.org_
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>>>      >
>>>>      >     > _https://urldefense.proofpoint.com/v2/url?_
>>>>      <https://urldefense.proofpoint.com/v2/url?_>
>>>>      >     >
>>>>      >   
>>>         u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>>>>      >     >
>>>>      >   
>>>         siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>>>>      >
>>>>      >     _______________________________________________
>>>>      >     gpfsug-discuss mailing list
>>>>      >     gpfsug-discuss at _spectrumscale.org_
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>>>>      >     __https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>>>>      >     gpfsug-discuss mailing list
>>>>      >     gpfsug-discuss at spectrumscale.org
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>>      >   
>>>>    
>>>          https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>>>>      >
>>>>      >
>>>>      >
>>>>      >
>>>>      > _______________________________________________
>>>>      > gpfsug-discuss mailing list
>>>>      > gpfsug-discuss at spectrumscale.org
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>>      > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=>
>>>>      >
>>>> 
>>>>      --
>>>>      Aaron Knister
>>>>      NASA Center for Climate Simulation (Code 606.2)
>>>>      Goddard Space Flight Center
>>>>      (301) 286-2776 <tel:(301)%20286-2776>
>>>        <tel:%28301%29%20286-2776>
>>>>      _______________________________________________
>>>>      gpfsug-discuss mailing list
>>>>      gpfsug-discuss at spectrumscale.org
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>>      https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=>
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> gpfsug-discuss mailing list
>>>> gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>>> 
>>> 
>>>        --
>>>        Aaron Knister
>>>        NASA Center for Climate Simulation (Code 606.2)
>>>        Goddard Space Flight Center
>>>        (301) 286-2776 <tel:(301)%20286-2776>
>>>        _______________________________________________
>>>        gpfsug-discuss mailing list
>>>        gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>        https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>> 
>>    _______________________________________________
>>    gpfsug-discuss mailing list
>>    gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>    https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>> 
>> 
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>> 
> 
> -- 
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
> 
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/facaf6ea/attachment.htm>

From kh.atmane at gmail.com  Thu Dec 21 12:41:49 2017
From: kh.atmane at gmail.com (atmane)
Date: Thu, 21 Dec 2017 13:41:49 +0100
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 49
In-Reply-To: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
References: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
Message-ID: <op.zblj74uypgw25x@pc-atm>

Hi
This Is My Config

mmlsconfig | grep verbsRdma
verbsRdma enable
mmlsconfig | grep verbsPorts
verbsPorts mlx4_0/1 mlx4_0/2 mlx4_1/1 mlx4_1/2
ibstat | grep -E "CA|Port"

CA 'mlx4_0'
         CA type: MT4099
         Port 1:
                 Port GUID: 0x0002c903001a1571
         Port 2:
                 Port GUID: 0x0002c903001a1572
CA 'mlx4_1'
         CA type: MT4099
         Port 1:
                 Port GUID: 0x0002c903001a15c1
         Port 2:
                 Port GUID: 0x0002c903001a15c2
ethtool ib0 |grep "Link detected"
Link detected: yes
ethtool ib1 |grep "Link detected"
Link detected: yes
ethtool ib2 |grep "Link detected"
Link detected: yes
ethtool ib3 |grep "Link detected"
Link detected: yes
ifconfig ib0 |grep inet
inet 192.168.100.3 netmask 255.255.255.0  broadcast 192.168.100.255
ifconfig ib1 |grep inet
Infiniband hardware address can be incorrect! Please read BUGS section in  
ifconfig(8).
ifconfig ib2 |grep inet
Infiniband hardware address can be incorrect! Please read BUGS section in  
ifconfig(8).
ifconfig ib3 |grep inet
Infiniband hardware address can be incorrect! Please read BUGS section in  
ifconfig(8).
ip addr show ib1
8: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
1024
     link/infiniband  
a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:72 brd  
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
9: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
1024
     link/infiniband  
a0:00:02:20:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c1 brd  
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

10: ib3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP  
qlen 1024
     link/infiniband  
a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c2 brd  
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff






Le Thu, 21 Dec 2017 13:00:01 +0100,  
<gpfsug-discuss-request at spectrumscale.org> a ?crit:

> Send gpfsug-discuss mailing list submissions to
> 	gpfsug-discuss at spectrumscale.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
> 	gpfsug-discuss-request at spectrumscale.org
>
> You can reach the person managing the list at
> 	gpfsug-discuss-owner at spectrumscale.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: more than one mlx connectx-4 adapter in same	host
>       (Frank Kraemer)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 21 Dec 2017 08:07:24 +0100
> From: "Frank Kraemer" <kraemerf at de.ibm.com>
> To: gpfsug-discuss at spectrumscale.org
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in
> 	same	host
> Message-ID:
> 	<OFE99AF1B2.2CCA57A0-ONC12581FD.00272158-C12581FD.00272158 at notes.na.collabserv.com>
> 	
> Content-Type: text/plain; charset="us-ascii"
>
> David,
>
>> We're trying to get 40 gbe connection between Mellanox switches and  
>> Cisco
>> switches down at the other end of the machine room
>> The BiDi part seems to be the best given about 30m  run on multimode.
>> However Mellanox support says it's not supported.
>> Want to use this to get close to IB speeds for GPFS on nodes that aren't
>> on the IB fabric.
>> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
>> switches are different brands?
>
> maybe that's a good reason to get in contact with the team from  
> Interoptic.
> They claim a good expertise for these kind of problems - feedback is  
> good.
> http://packetpushers.net/podcast/podcasts/show-360-all-about-optics-interoptic-sponsored/
> https://interoptic.com/
>
> Frank Kraemer
> IBM Consulting IT Specialist  / Client Technical Architect
> Am Weiher 24, 65451 Kelsterbach
> mailto:kraemerf at de.ibm.com
> voice: +49-(0)171-3043699 / +4970342741078
> IBM Germany
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:  
> <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171221/31aeae52/attachment-0001.html>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 71, Issue 49
> **********************************************


-- 
Utilisant le logiciel de courrier d'Opera : http://www.opera.com/mail/


From a.khiredine at meteo.dz  Thu Dec 21 14:02:40 2017
From: a.khiredine at meteo.dz (atmane khiredine)
Date: Thu, 21 Dec 2017 14:02:40 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 49
In-Reply-To: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
References: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
Message-ID: <4B32CB5C696F2849BDEF7DF9EACE884B72ACDD56@SDEB-EXC02.meteo.dz>



Atmane Khiredine
HPC System Administrator | Office National de la M?t?orologie
T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz

________________________________________
De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org]
Envoy? : jeudi 21 d?cembre 2017 13:00
? : gpfsug-discuss at spectrumscale.org
Objet : gpfsug-discuss Digest, Vol 71, Issue 49

Send gpfsug-discuss mailing list submissions to
        gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
        gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
        gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Re: more than one mlx connectx-4 adapter in same  host
      (Frank Kraemer)

  This Is My Config
    mmlsconfig | grep verbsRdma
    verbsRdma enable
    mmlsconfig | grep verbsPorts
    verbsPorts mlx4_0/1 mlx4_0/2 mlx4_1/1 mlx4_1/2
    ibstat | grep -E "CA|Port"
    CA 'mlx4_0'
             CA type: MT4099
             Port 1:
                     Port GUID: 0x0002c903001a1571
             Port 2:
                     Port GUID: 0x0002c903001a1572
    CA 'mlx4_1'
             CA type: MT4099
             Port 1:
                     Port GUID: 0x0002c903001a15c1
             Port 2:
                     Port GUID: 0x0002c903001a15c2
    ethtool ib0 |grep "Link detected"
    Link detected: yes
    ethtool ib1 |grep "Link detected"
    Link detected: yes
 

    ethtool ib2 |grep "Link detected"
    Link detected: yes
    ethtool ib3 |grep "Link detected"
    Link detected: yes
    ifconfig ib0 |grep inet
    inet 192.168.100.3 netmask 255.255.255.0  broadcast 192.168.100.255
    ifconfig ib1 |grep inet
    Infiniband hardware address can be incorrect! Please read BUGS section in  
    ifconfig(8).
    ifconfig ib2 |grep inet
    Infiniband hardware address can be incorrect! Please read BUGS section in  
    ifconfig(8).
    ifconfig ib3 |grep inet
    Infiniband hardware address can be incorrect! Please read BUGS section in  
    ifconfig(8).
    ip addr show ib1
    8: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
    1024
         link/infiniband  
    a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:72 brd  
    00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    9: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
    1024
         link/infiniband  
    a0:00:02:20:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c1 brd  
    00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
 
    10: ib3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP  
    qlen 1024
         link/infiniband  
    a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c2 brd  
    00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

Message: 1
Date: Thu, 21 Dec 2017 08:07:24 +0100
From: "Frank Kraemer" <kraemerf at de.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in
        same    host
Message-ID:
        <OFE99AF1B2.2CCA57A0-ONC12581FD.00272158-C12581FD.00272158 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

David,

> We're trying to get 40 gbe connection between Mellanox switches and Cisco
> switches down at the other end of the machine room
> The BiDi part seems to be the best given about 30m  run on multimode.
> However Mellanox support says it's not supported.
> Want to use this to get close to IB speeds for GPFS on nodes that aren't
> on the IB fabric.
> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
> switches are different brands?

maybe that's a good reason to get in contact with the team from Interoptic.
They claim a good expertise for these kind of problems - feedback is good.
http://packetpushers.net/podcast/podcasts/show-360-all-about-optics-interoptic-sponsored/
https://interoptic.com/

Frank Kraemer
IBM Consulting IT Specialist  / Client Technical Architect
Am Weiher 24, 65451 Kelsterbach
mailto:kraemerf at de.ibm.com
voice: +49-(0)171-3043699 / +4970342741078
IBM Germany
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171221/31aeae52/attachment-0001.html>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 71, Issue 49
**********************************************


From oehmes at gmail.com  Thu Dec 21 16:38:27 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Thu, 21 Dec 2017 16:38:27 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
References: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
	<OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
Message-ID: <CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>

Daniel,

while this might be easier to think about it, its not true :-)
lets just use an example.  a disk drive can do 100 io's per second with
128kb random writes and 80 iops with 256kb writes . now lets do the math
with a 8+2p setup for each of the 2 cases. this means you can do 100 times
1mb writes (8*128k) or 80 times 2 mb writes so 100 MB/sec or 160 MB/sec
with the exact same drives. given you can fit 2 times as many subblocks
into the 2mb block you would gain 60% of speed by just going to this larger
size. so if you now go to a 16MB blocksize and you have just 50 iops @ 2MB
each you can write ~800 MB/sec with the exact same setup and same size
small writes, that's a factor of 8 .
so i/o size AND number of subblocks matter.

Sven



On Thu, Dec 21, 2017 at 12:22 PM Daniel Kidger <daniel.kidger at uk.ibm.com>
wrote:

> My suggestion is that it is better to not think of the performance coming
> from having more than 32 sub-blocks but instead that the performance comes
> from smaller sub-blocks. The fact that there are now more of them in say a
> 4MB blocksize filesytem is just a side effect.
>
> Daniel
> [image: /spectrum_storage-banne]
>
>
> [image: Spectrum Scale Logo]
>
>
> *Dr Daniel Kidger*
> IBM Technical Sales Specialist
> Software Defined Solution Sales
>
> + <+%2044-7818%20522%20266> 44-(0)7818 522 266 <+%2044-7818%20522%20266>
> daniel.kidger at uk.ibm.com
>
> On 19 Dec 2017, at 21:32, Aaron Knister <aaron.s.knister at nasa.gov> wrote:
>
> Thanks, Sven. Understood!
>
> On 12/19/17 3:20 PM, Sven Oehme wrote:
>
> Hi,
>
>
> the zero padding was never promoted into a GA stream, it was an
>
> experiment to proof we are on the right track when we eliminate the
>
> overhead from client to NSD Server, but also showed that alone is not
>
> good enough. the work for the client is the same compared to the >32
>
> subblocks, but the NSD Server has more work as it can't pack as many
>
> subblocks and therefore files into larger blocks, so you need to do more
>
> writes to store the same number of files.
>
> thats why there is the additional substantial improvement  when we then
>
> went to >32 subblocks.
>
>
> sven
>
>
> On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
>
> SCIENCE CORP] <aaron.s.knister at nasa.gov
>
> <mailto:aaron.s.knister at nasa.gov <aaron.s.knister at nasa.gov>>> wrote:
>
>
>    Thanks Sven! That makes sense to me and is what I thought was the
>
>    case which is why I was confused when I saw the reply to the thread
>
>    that said the >32 subblocks code had no performance impact.
>
>
>    A couple more question for you? in your presentation there?s a
>
>    benchmark that shows the file create performance without the zero
>
>    padding. Since you mention this is done for security reasons was
>
>    that feature ever promoted to a GA Scale release? I?m also wondering
>
>    if you could explain the performance difference between the no zero
>
>    padding code and the > 32 subblock code since given your the example
>
>    of 32K files and 16MB block size I figure both cases ought to write
>
>    the same amount to disk.
>
>
>    Thanks!
>
>
>    -Aaron
>
>
>
>
>
>
>    On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com
>
>    <mailto:oehmes at gmail.com <oehmes at gmail.com>>> wrote:
>
>    i thought i answered that already, but maybe i just thought about
>
>    answering it and then forgot about it :-D
>
>
>    so yes more than 32 subblocks per block significant increase the
>
>    performance of filesystems with small files, for the sake of the
>
>    argument let's say 32k in a large block filesystem again for sake
>
>    of argument say 16MB.
>
>
>    you probably ask why ?
>
>
>    if you create a file and write 32k into it in a pre 5.0.0 Version
>
>    16 MB filesystem your client actually doesn't write 32k to the NSD
>
>    Server, it writes 512k, because thats the subblock size and we
>
>    need to write the full subblock (for security reasons). so first
>
>    you waste significant memory on the client to cache that zero
>
>    padding, you waste network bandwidth and you waste NSD Server
>
>    cache because you store it there too. this means you overrun the
>
>    cache more quickly, means you start doing read/modify writes
>
>    earlier on all your nice large raid tracks... i guess you get the
>
>    story by now.
>
>
>    in fact,  if you have a good raid code that can drive really a lot
>
>    of bandwidth out of individual drives like a GNR system you get
>
>    more performance for small file writes as larger your blocksize
>
>    is, because we can 'pack' more files into larger i/os and
>
>    therefore turn a small file create workload into a bandwidth
>
>    workload, essentially exactly what we did and i demonstrated in
>
>    the CORAL presentation .
>
>
>    hope that makes this crystal clear now .
>
>
>    sven
>
>
>
>
>    On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister
>
>    <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov
> <aaron.s.knister at nasa.gov>>> wrote:
>
>
>        Thanks, Alex. I'm all too familiar with the trade offs between
>
>        large
>
>        blocks and small files and we do use pretty robust SSD storage
>
>        for our
>
>        metadata. We support a wide range of workloads and we have
>
>        some folks
>
>        with many small (<1M) files and other folks with many large
>
>        (>256MB) files.
>
>
>        My point in this thread is that IBM has said over and over
>
>        again in
>
>        presentations that there is a significant performance gain
>
>        with the >32
>
>        subblocks code on filesystems with large block sizes (although
>
>        to your
>
>        point I'm not clear on exactly what large means since I didn't
>
>        define
>
>        large in this context). Therefore given that the >32 subblock
>
>        code gives
>
>        a significant performance gain one could reasonably assume
>
>        that having a
>
>        filesystem with >32 subblocks is required to see this gain
>
>        (rather than
>
>        just running the >32 subblocks code on an fs w/o > 32 subblocks).
>
>
>        This lead me to ask about a migration tool because in my mind
>
>        if there's
>
>        a performance gain from having >32 subblocks on the FS I'd
>
>        like that
>
>        feature and having to manually copy 10's of PB to new hardware
>
>        to get
>
>        this performance boost is unacceptable. However, IBM can't
>
>        seem to make
>
>        up their mind about whether or not the >32 subblocks code
>
>        *actually*
>
>        provides a performance increase. This seems like a pretty
>
>        straightforward question.
>
>
>        -Aaron
>
>
>        On 12/15/17 3:48 PM, Alex Chekholko wrote:
>
> Hey Aaron,
>
>
> Can you define your sizes for "large blocks" and "small
>
>        files"?  If you
>
> dial one up and the other down, your performance will be
>
>        worse.  And in
>
> any case it's a pathological corner case so it shouldn't
>
>        matter much for
>
> your workflow, unless you've designed your system with the
>
>        wrong values.
>
>
> For example, for bioinformatics workloads, I prefer to use 256KB
>
> filesystem block size, and I'd consider 4MB+ to be "large
>
>        block size",
>
> which would make the filesystem obviously unsuitable for
>
>        processing
>
> millions of 8KB files.
>
>
> You can make a histogram of file sizes in your existing
>
>        filesystems and
>
> then make your subblock size (1/32 of block size) on the
>
>        smaller end of
>
> that.   Also definitely use the "small file in inode"
>
>        feature and put
>
> your metadata on SSD.
>
>
> Regards,
>
> Alex
>
>
> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
>
> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov
> <aaron.s.knister at nasa.gov>>
>
>        <mailto:aaron.s.knister at nasa.gov <aaron.s.knister at nasa.gov>
>
>        <mailto:aaron.s.knister at nasa.gov <aaron.s.knister at nasa.gov>>>>
> wrote:
>
>
>      Thanks, Bill.
>
>
>      I still don't feel like I've got an clear answer from
>
>        IBM and frankly
>
>      the core issue of a lack of migration tool was totally
>
>        dodged.
>
>
>      Again in Sven's presentation from SSUG @ SC17
>
>
>
>         (
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=
>
>        <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=
> >)
>
>      he mentions "It has a significant performance penalty
>
>        for small files in
>
>      large block size filesystems" and the demonstrates that
>
>        with several
>
>      mdtest runs (which show the effect with and without the >32
>
>      subblocks code):
>
>
>
>      4.2.1 base code - SUMMARY: (of 3 iterations)
>
>      File creation : Mean = 2237.644
>
>
>      zero-end-of-file-padding (4.2.2 + ifdef for zero
>
>        padding):  SUMMARY: (of
>
>      3 iterations)
>
>      File creation : Mean = 12866.842
>
>
>      more sub blocks per block (4.2.2 + morethan32subblock code):
>
>      File creation : Mean = 40316.721
>
>
>      Can someone (ideally Sven) give me a straight answer as
>
>        to whether or
>
>      not the > 32 subblock code actually makes a performance
>
>        difference for
>
>      small files in large block filesystems? And if not, help
>
>        me understand
>
>      why his slides and provided benchmark data have
>
>        consistently indicated
>
>      it does?
>
>
>      -Aaron
>
>
>      On 12/1/17 11:44 AM, Bill Hartner wrote:
>
>      > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported
>
>        some of the
>
>      > results @ user group meeting.
>
>      >
>
>      > -Bill
>
>      >
>
>      > Bill Hartner
>
>      > IBM Systems
>
>      > Scalable I/O Development
>
>      > Austin, Texas
>
>      > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com
> <bhartner at us.ibm.com>>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com> <
> mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>>
>
>      > home office 512-784-0980 <(512)%20784-0980> <tel:(512)%20784-0980>
>
>        <tel:512-784-0980 <512-784-0980> <tel:(512)%20784-0980>>
>
>      >
>
>      >
>
>      > Inactive hide details for Jan-Frode Myklebust
>
>        ---12/01/2017 06:53:44
>
>      > AM---Bill, could you say something about what the
>
>        metadataJan-Frode
>
>      > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you
>
>        say something
>
>      > about what the metadata-storage here was?
>
>        ESS/NL-SAS/3way replication?
>
>      >
>
>      > From: Jan-Frode Myklebust <janfrode at tanso.net
>
>        <mailto:janfrode at tanso.net <janfrode at tanso.net>> <
> mailto:janfrode at tanso.net <janfrode at tanso.net>
>
>        <mailto:janfrode at tanso.net <janfrode at tanso.net>>>>
>
>      > To: gpfsug main discussion list
>
>        <gpfsug-discuss at spectrumscale.org
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>>>
>
>      > Date: 12/01/2017 06:53 AM
>
>      > Subject: Re: [gpfsug-discuss] Online data migration tool
>
>      > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>>
>
>      >
>
>      >
>
>
>
>
>         ------------------------------------------------------------------------
>
>      >
>
>      >
>
>      >
>
>      > Bill, could you say something about what the
>
>        metadata-storage here was?
>
>      > ESS/NL-SAS/3way replication?
>
>      >
>
>      > I just asked about this in the internal slack channel
>
>        #scale-help today..
>
>      >
>
>      >
>
>      >
>
>      > -jf
>
>      >
>
>      > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
>
>        <_bhartner at us.ibm.com_
>
>      > <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>> <
> mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>>>>:
>
>      >
>
>      >     > "It has a significant performance penalty for
>
>        small files in
>
>      large
>
>      >     > block size filesystems"
>
>      >
>
>      >     Aaron,
>
>      >
>
>      >     Below are mdtest results for a test we ran for
>
>        CORAL - file
>
>      size was
>
>      >     32k.
>
>      >
>
>      >     We have not gone back and ran the test on a file
>
>        system formatted
>
>      >     without > 32 subblocks. We'll do that at some point...
>
>      >
>
>      >     -Bill
>
>      >
>
>      >     -- started at 10/28/2017 17:51:38 --
>
>      >
>
>      >     mdtest-1.9.3 was launched with 228 total task(s)
>
>        on 12 node(s)
>
>      >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>
>      >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w
>
>        32768 -C -F -r
>
>      -p 360
>
>      >     -u -y
>
>      >     Path: /ibm/fs2-16m-10
>
>      >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used
>
>        Inodes: 0.0%
>
>      >
>
>      >     228 tasks, 67239936 files
>
>      >
>
>      >     SUMMARY: (of 3 iterations)
>
>      >     Operation Max Min Mean Std Dev
>
>      >     --------- --- --- ---- -------
>
>      >     File creation : 51953.498 50558.517 51423.221 616.643
>
>      >     File stat : 0.000 0.000 0.000 0.000
>
>      >     File read : 0.000 0.000 0.000 0.000
>
>      >     File removal : 96746.376 92149.535 94658.774 1900.187
>
>      >     Tree creation : 1.588 0.070 0.599 0.700
>
>      >     Tree removal : 0.213 0.034 0.097 0.082
>
>      >
>
>      >     -- finished at 10/28/2017 19:51:54 --
>
>      >
>
>      >     Bill Hartner
>
>      >     IBM Systems
>
>      >     Scalable I/O Development
>
>      >     Austin, Texas_
>
>      >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
> <bhartner at us.ibm.com>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>
>
>      <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com> <
> mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>>>
>
>      >     home office 512-784-0980 <(512)%20784-0980>
> <tel:(512)%20784-0980>
>
>        <tel:512-784-0980 <512-784-0980> <tel:(512)%20784-0980>>
>
>      >
>
>      >     _
>
>      >     __gpfsug-discuss-bounces at spectrumscale.org_
>
>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>>> wrote on
>
>      >     11/29/2017 04:41:48 PM:
>
>      >
>
>      >     > From: Aaron Knister <_aaron.knister at gmail.com_
>
>      >     <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>
>
>        <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>>
>
>        <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>
>
>        <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>>>>>
>
>      >
>
>      >
>
>      >     > To: gpfsug main discussion list
>
>      >     <_gpfsug-discuss at spectrumscale.org_
>
>      >     <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>>>>
>
>      >
>
>      >     > Date: 11/29/2017 04:42 PM
>
>      >
>
>      >
>
>      >     > Subject: Re: [gpfsug-discuss] Online data
>
>        migration tool
>
>      >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>
>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>>>
>
>      >
>
>      >     >
>
>      >
>
>      >     > Thanks, Nikhil. Most of that was consistent with
>
>        my understnading,
>
>      >     > however I was under the impression that the >32
>
>        subblocks code is
>
>      >     > required to achieve the touted 50k file
>
>        creates/second that Sven has
>
>      >     > talked about a bunch of times:
>
>      >     >
>
>      >     >
>
>      >
>
>
>
>          _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=
> >
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >>
>
>      >     >
>
>      >
>
>
>
>          _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=
> >
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >>
>
>      >     > _Sven_Oehme_-_News_from_Research.pdf
>
>      >     >
>
>        _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=
>
>      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=
> >
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >>
>
>      >     >
>
>        _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>
>      >
>
>      >
>
>      >     > from those presentations regarding 32 subblocks:
>
>      >     >
>
>      >     > "It has a significant performance penalty for
>
>        small files in large
>
>      >     > block size filesystems"
>
>      >
>
>      >     > although I'm not clear on the specific
>
>        definition of "large". Many
>
>      >     > filesystems I encounter only have a 1M block
>
>        size so it may not
>
>      >     > matter there, although that same presentation
>
>        clearly shows the
>
>      >     > benefit of larger block sizes which is yet
>
>        *another* thing for which
>
>      >     > a migration tool would be helpful.
>
>      >
>
>      >     > -Aaron
>
>      >     >
>
>      >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>
>      >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
> <nikhilk at us.ibm.com>
>
>        <mailto:nikhilk at us.ibm.com <nikhilk at us.ibm.com>>
>
>      <mailto:nikhilk at us.ibm.com <nikhilk at us.ibm.com>
>
>        <mailto:nikhilk at us.ibm.com <nikhilk at us.ibm.com>>>>> wrote:
>
>      >
>
>      >     > Hi,
>
>      >     >
>
>      >     > I would like to clarify migration path to 5.0.0
>
>        from 4.X.X
>
>      clusters.
>
>      >     > For all Spectrum Scale clusters that are
>
>        currently at 4.X.X,
>
>      it is
>
>      >     > possible to migrate to 5.0.0 with no offline
>
>        data migration
>
>      and no
>
>      >     > need to move data. Once these clusters are at
>
>        5.0.0, they will
>
>      >     > benefit from the performance improvements, new
>
>        features (such as
>
>      >     > file audit logging), and various enhancements
>
>        that are
>
>      included in
>
>      >     5.0.0.
>
>      >     >
>
>      >     > That being said, there is one enhancement that
>
>        will not be
>
>      applied
>
>      >     > to these clusters, and that is the increased
>
>        number of
>
>      sub-blocks
>
>      >     > per block for small file allocation. This means
>
>        that for file
>
>      >     > systems with a large block size and a lot of
>
>        small files, the
>
>      >     > overall space utilization will be the same it
>
>        currently is
>
>      in 4.X.X.
>
>      >     > Since file systems created at 4.X.X and earlier
>
>        used a block
>
>      size
>
>      >     > that kept this allocation in mind, there should
>
>        be very little
>
>      >     > impact on existing file systems.
>
>      >     >
>
>      >     > Outside of that one particular function, the
>
>        remainder of the
>
>      >     > performance improvements, metadata improvements,
>
>        updated
>
>      >     > compatibility, new functionality, and all of the
>
>        other
>
>      enhancements
>
>      >     > will be immediately available to you once you
>
>        complete the
>
>      upgrade
>
>      >     > to 5.0.0 -- with no need to reformat, move data,
>
>        or take
>
>      your data
>
>      >     offline.
>
>      >     >
>
>      >     > I hope that clarifies things a little and makes
>
>        the upgrade path
>
>      >     > more accessible.
>
>      >     >
>
>      >     > Please let me know if there are any other
>
>        questions or concerns.
>
>      >     >
>
>      >     > Thank you,
>
>      >     > Nikhil Khandelwal
>
>      >     > Spectrum Scale Development
>
>      >     > Client Adoption
>
>      >     >
>
>      >     > _______________________________________________
>
>      >     > gpfsug-discuss mailing list
>
>      >     > gpfsug-discuss at _spectrumscale.org_
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
>
>      >     > _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
>
>      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
> >
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>
>
>      >
>
>      >     > _______________________________________________
>
>      >     > gpfsug-discuss mailing list
>
>      >     > gpfsug-discuss at _spectrumscale.org_
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/6513c936/attachment.htm>

From valdis.kletnieks at vt.edu  Thu Dec 21 20:35:26 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Thu, 21 Dec 2017 15:35:26 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>
References: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
	<OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
	<CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>
Message-ID: <37655.1513888526@turing-police.cc.vt.edu>

On Thu, 21 Dec 2017 16:38:27 +0000, Sven Oehme said:

> size. so if you now go to a 16MB blocksize and you have just 50 iops @ 2MB
> each you can write ~800 MB/sec with the exact same setup and same size
> small writes, that's a factor of 8 .

That's assuming your metadata storage is able to handle open/read/write/close
on enough small files per second to push 800MB/sec.  If you're talking 128K subblocks,
you're going to need some 6,400 small files per second to fill that pipe...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/385aeb17/attachment.sig>

From oehmes at gmail.com  Fri Dec 22 00:02:43 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Fri, 22 Dec 2017 00:02:43 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <37655.1513888526@turing-police.cc.vt.edu>
References: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
	<OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
	<CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>
	<37655.1513888526@turing-police.cc.vt.edu>
Message-ID: <CALssuR0Pk=ck9rT-OTvp3SEUroDTDj2bcxFGDpHzRX1wztSjHQ@mail.gmail.com>

thats not how GPFS aehm Scale works :-)
each client has pre-allocated inodes in memory and creating files is a
matter of spooling records. yes, eventually you need to destage this to the
disk, but that happens only every few seconds and given this i/os are
usually very colocated so good storage cache technology can reduce i/os to
physical media significant.

to proof the point look at this numbers :
-- started at 10/17/2017 14:29:13 --

mdtest-1.9.3 was launched with 110 total task(s) on 11 node(s)
Command line used: /ghome/oehmes/mpi/bin/mdtest-pcmpi9131-existingdir -d
/ibm/fs2-16m-09/shared/mdtest-ec -i 1 -n 10000 -F -w 0 -Z -p 8 -N 11 -u
Path: /ibm/fs2-16m-09/shared
FS: 128.1 TiB   Used FS: 0.2%   Inodes: 476.8 Mi   Used Inodes: 0.0%

110 tasks, 1100000 files

SUMMARY: (of 1 iterations)
   Operation                      Max            Min           Mean
Std Dev
   ---------                      ---            ---           ----
-------
   File creation     :     444221.343     444221.343     444221.343
  0.000
   File stat         :    6704498.841    6704498.841    6704498.841
  0.000
   File read         :    3859105.596    3859105.596    3859105.596
  0.000
   File removal      :     409336.606     409336.606     409336.606
  0.000
   Tree creation     :          5.344          5.344          5.344
  0.000
   Tree removal      :          1.145          1.145          1.145
  0.000

-- finished at 10/17/2017 14:29:27 --

this is a run against a 16mb blocksize filesystem with only spinning disks
(just one GL6 ESS) , not a single SSD and as you can see , this system on
11 nodes produces 444k creates / second something far above and beyond of
what drives can do.

and yes i know this stuff is all very complicated and not easy to explain
:-)

sven


On Thu, Dec 21, 2017 at 8:35 PM <valdis.kletnieks at vt.edu> wrote:

> On Thu, 21 Dec 2017 16:38:27 +0000, Sven Oehme said:
>
> > size. so if you now go to a 16MB blocksize and you have just 50 iops @
> 2MB
> > each you can write ~800 MB/sec with the exact same setup and same size
> > small writes, that's a factor of 8 .
>
> That's assuming your metadata storage is able to handle
> open/read/write/close
> on enough small files per second to push 800MB/sec.  If you're talking
> 128K subblocks,
> you're going to need some 6,400 small files per second to fill that pipe...
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/a66aca84/attachment.htm>

From damir.krstic at gmail.com  Fri Dec 22 17:44:50 2017
From: damir.krstic at gmail.com (Damir Krstic)
Date: Fri, 22 Dec 2017 17:44:50 +0000
Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group without
	takeover
Message-ID: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>

It's been a very frustrating couple of months with our 2 ESS systems. IBM
tells us we had blueflame bug and they came on site and updated our ESS to
the latest version back in middle of November. Wednesday night one of the
NSD servers in one of our ESS building blocks kernel panicked. No idea why
and none of the logs are insightful. We have a PMR open with IBM. I am not
very confident we will get to the bottom of what's causing kernel panics on
our IO servers. The system has gone down over 4 times now in 2 months.

When we tried brining it back up, it rejoined the recovery group and the IO
on the entire cluster locked up until we were able to find couple of
compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on
those nodes resolved the issue of the filesystem locking up.

So far we have never been successful in brining back an IO server and not
having a filesystem lock up until we find a node with pending state with
tscomm. Anyway, the system was stable for few minutes until the same IO
server that went down on Wednesday night went into an arbitrating mode. It
never recovered. We stopped gpfs on that server and IO recovered again. We
left gpfs down and cluster seems to be OK.

My question is, is there a way of brining back the IO server into the mix
without the recoverygroup takeover happening? Could I just start a gpfs and
have it back in the mix as a backup server for the recoverygroup and if so,
how do you do that. Right now that server is designated as primary server
for the recovery group. I would like to have both IO servers in the mix for
redundancy purposes.

This ESS situation is beyond frustrating and I don't see end in sight.

Any help is appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/4d155e13/attachment.htm>

From janfrode at tanso.net  Fri Dec 22 17:55:39 2017
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 22 Dec 2017 17:55:39 +0000
Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group
	without takeover
In-Reply-To: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
References: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
Message-ID: <CAHwPatgUFx4GHitKsmiSWZJC=20FCVBoMgmaDnTK-8zUbj4J6g@mail.gmail.com>

Can?t you just reverse the mmchrecoverygroup --servers order, before
starting the io-server?


-jf
fre. 22. des. 2017 kl. 18:45 skrev Damir Krstic <damir.krstic at gmail.com>:

> It's been a very frustrating couple of months with our 2 ESS systems. IBM
> tells us we had blueflame bug and they came on site and updated our ESS to
> the latest version back in middle of November. Wednesday night one of the
> NSD servers in one of our ESS building blocks kernel panicked. No idea why
> and none of the logs are insightful. We have a PMR open with IBM. I am not
> very confident we will get to the bottom of what's causing kernel panics on
> our IO servers. The system has gone down over 4 times now in 2 months.
>
> When we tried brining it back up, it rejoined the recovery group and the
> IO on the entire cluster locked up until we were able to find couple of
> compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on
> those nodes resolved the issue of the filesystem locking up.
>
> So far we have never been successful in brining back an IO server and not
> having a filesystem lock up until we find a node with pending state with
> tscomm. Anyway, the system was stable for few minutes until the same IO
> server that went down on Wednesday night went into an arbitrating mode. It
> never recovered. We stopped gpfs on that server and IO recovered again. We
> left gpfs down and cluster seems to be OK.
>
> My question is, is there a way of brining back the IO server into the mix
> without the recoverygroup takeover happening? Could I just start a gpfs and
> have it back in the mix as a backup server for the recoverygroup and if so,
> how do you do that. Right now that server is designated as primary server
> for the recovery group. I would like to have both IO servers in the mix for
> redundancy purposes.
>
> This ESS situation is beyond frustrating and I don't see end in sight.
>
> Any help is appreciated.
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/d7a7cb53/attachment.htm>

From alex at calicolabs.com  Fri Dec 22 18:01:58 2017
From: alex at calicolabs.com (Alex Chekholko)
Date: Fri, 22 Dec 2017 10:01:58 -0800
Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group
	without takeover
In-Reply-To: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
References: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
Message-ID: <CANcy_Pb8JnZG+nvhuDGFb_X+aKPerzwu74iM7VmVwvFckak9zQ@mail.gmail.com>

Hi Damir,

I'm not sure whether this applies to you, but this was my experience.

GPFS absolutely depends on a reliable network interconnect.  If anything
goes wrong on the network layer, GPFS may not be able to recover.

Do you have visibility and monitoring on all the low-level network counters
on all the relevant network interfaces?

e.g. if one of your clients is connected to a switch port that is flaky in
some way and some GPFS message goes unacked, you can get unusual client
state and then the whole cluster hangs...

In my case years ago we ended up replacing some flaky HP switches and all
our GPFS troubles went away!  And we were able to show those switches had
some issues when doing non-GPFS testing with iperf, etc.

Regards,
Alex

On Fri, Dec 22, 2017 at 9:44 AM, Damir Krstic <damir.krstic at gmail.com>
wrote:

> It's been a very frustrating couple of months with our 2 ESS systems. IBM
> tells us we had blueflame bug and they came on site and updated our ESS to
> the latest version back in middle of November. Wednesday night one of the
> NSD servers in one of our ESS building blocks kernel panicked. No idea why
> and none of the logs are insightful. We have a PMR open with IBM. I am not
> very confident we will get to the bottom of what's causing kernel panics on
> our IO servers. The system has gone down over 4 times now in 2 months.
>
> When we tried brining it back up, it rejoined the recovery group and the
> IO on the entire cluster locked up until we were able to find couple of
> compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on
> those nodes resolved the issue of the filesystem locking up.
>
> So far we have never been successful in brining back an IO server and not
> having a filesystem lock up until we find a node with pending state with
> tscomm. Anyway, the system was stable for few minutes until the same IO
> server that went down on Wednesday night went into an arbitrating mode. It
> never recovered. We stopped gpfs on that server and IO recovered again. We
> left gpfs down and cluster seems to be OK.
>
> My question is, is there a way of brining back the IO server into the mix
> without the recoverygroup takeover happening? Could I just start a gpfs and
> have it back in the mix as a backup server for the recoverygroup and if so,
> how do you do that. Right now that server is designated as primary server
> for the recovery group. I would like to have both IO servers in the mix for
> redundancy purposes.
>
> This ESS situation is beyond frustrating and I don't see end in sight.
>
> Any help is appreciated.
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/bded79d3/attachment.htm>

From makaplan at us.ibm.com  Fri Dec  1 04:25:30 2017
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Thu, 30 Nov 2017 23:25:30 -0500
Subject: [gpfsug-discuss] FIle system vs Database
In-Reply-To: <OFDAB5F371.63BBCEDD-ON002581E8.00776AE7@LocalDomain>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com><OF1A4C52AF.364AA2DD-ON002581E8.00592509@LocalDomain><OF0CB3DA65.211AD6C3-ON852581E8.006590C3-852581E8.0065FFAE@notes.na.collabserv.com><OF777157A3.FE35AFDD-ON002581E8.0066BB88@LocalDomain><OFC357735B.03D413FB-ON852581E8.0072148B-852581E8.0072A3C9@notes.na.collabserv.com>
	<OFDAB5F371.63BBCEDD-ON002581E8.00776AE7@LocalDomain>
Message-ID: <OF4ADF66F6.43003461-ON852581E9.0017BDD1-852581E9.00184F30@notes.na.collabserv.com>

Indeed, for a very large directory you might get some speedup using

samples/ilm/mmfind directory -ls -maxdepth 1

There are some caveats, the same as those for the command upon which 
mmfind rests, mmapplypolicy.




From:   Skylar Thompson <skylar2 at u.washington.edu>
To:     gpfsug-discuss at spectrumscale.org
Date:   11/30/2017 04:44 PM
Subject:        Re: [gpfsug-discuss] FIle system vs Database
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Interesting, thanks for the information Marc. Could there be an 
improvement
for something like "ls -l some-dir" using the API, though? Instead of 
getdents +
stat for every file (entering and leaving kernel mode many times), could 
it be done
in one operation with one context switch?




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171130/4e738bf8/attachment-0001.htm>

From chris.schlipalius at pawsey.org.au  Fri Dec  1 05:32:28 2017
From: chris.schlipalius at pawsey.org.au (Chris Schlipalius)
Date: Fri, 01 Dec 2017 13:32:28 +0800
Subject: [gpfsug-discuss] 2018 March 26th Singapore Spectrum Scale User
 Group event announced - Call for user stories
In-Reply-To: <A675FAE6-BAE2-4E17-AC2B-7EF1607CC7C9@pawsey.org.au>
References: <A675FAE6-BAE2-4E17-AC2B-7EF1607CC7C9@pawsey.org.au>
Message-ID: <391567D9-DC51-46B9-8E0E-D8EE13F6B814@pawsey.org.au>

Hello,
This is an announcement for the event held just before SCA18 and  general call out for user use cases or presentations that you would like to present, at the inaugural Spectrum Scale Usergroup Singapore on the Monday 26th March 2018, Sentosa, Singapore.
This is being held in conjunction with SCA18  https://sc-asia.org/
     
All current Singapore Spectrum Scale User Group event details can be found here: http://goo.gl/dXtqvS
Feel free to circulate this event link to all that may need it.
Please reserve your tickets now, and if you would like to speak, please email me the title and duration of the talk and speakers name details so I can add this to the agenda on Eventbrite and promote this in this discussion list and spectrumscale.org website.
     
Accommodation.
Once numbers of attendees are known I can obtain delegate accommodation custom rate for guests staying at the resort for this event. So please reserve a ticket in the Eventbrite link above ASAP.
We are looking forwards to a great new Usergroup next year in a fabulous venue.
Thanks again to NSCC and IBM for helping to arrange the venue and event booking.
     
Regards,
Chris Schlipalius
     
Team Lead, Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO)
12 Burvill Court
Kensington  WA  6151
Australia
     
Tel  +61 8 6436 8815  
Email  chris.schlipalius at pawsey.org.au

Regards,
Chris Schlipalius
 
Team Lead, Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO)
12 Burvill Court
Kensington  WA  6151
Australia
 
Tel  +61 8 6436 8815  
Email  chris.schlipalius at pawsey.org.au
Web  www.pawsey.org.au <http://www.pawsey.org.au/>
 
 
 

 
 
Web  www.pawsey.org.au <http://www.pawsey.org.au/>
     
     
    




From bhartner at us.ibm.com  Fri Dec  1 12:44:38 2017
From: bhartner at us.ibm.com (Bill Hartner)
Date: Fri, 1 Dec 2017 06:44:38 -0600
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
Message-ID: <OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>

> "It has a significant performance penalty for small files in large
> block size filesystems"

Aaron,

Below are mdtest results for a test we ran for CORAL - file size was 32k.

We have not gone back and ran the test on a file system formatted without >
32 subblocks.  We'll do that at some point...

-Bill

-- started at 10/28/2017 17:51:38 --

mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
Command line used: /tmp/mdtest-binary-dir/mdtest
-d /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360 -u
-y
Path: /ibm/fs2-16m-10
FS: 128.1 TiB   Used FS: 0.3%   Inodes: 476.8 Mi   Used Inodes: 0.0%

228 tasks, 67239936 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean
Std Dev
   ---------                      ---            ---           ----
-------
   File creation     :      51953.498      50558.517      51423.221
616.643
   File stat         :          0.000          0.000          0.000
0.000
   File read         :          0.000          0.000          0.000
0.000
   File removal      :      96746.376      92149.535      94658.774
1900.187
   Tree creation     :          1.588          0.070          0.599
0.700
   Tree removal      :          0.213          0.034          0.097
0.082

-- finished at 10/28/2017 19:51:54 --

Bill Hartner
IBM Systems
Scalable I/O Development
Austin, Texas
bhartner at us.ibm.com
home office 512-784-0980


gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:

> From: Aaron Knister <aaron.knister at gmail.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 11/29/2017 04:42 PM
> Subject: Re: [gpfsug-discuss] Online data migration tool
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
> Thanks, Nikhil. Most of that was consistent with my understnading,
> however I was under the impression that the >32 subblocks code is
> required to achieve the touted 50k file creates/second that Sven has
> talked about a bunch of times:
>
>
http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf

> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
> _Sven_Oehme_-_News_from_Research.pdf
> http://files.gpfsug.org/presentations/2016/SC16/12_-
> _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf

> from those presentations regarding 32 subblocks:
>
> "It has a significant performance penalty for small files in large
> block size filesystems"

> although I'm not clear on the specific definition of "large". Many
> filesystems I encounter only have a 1M block size so it may not
> matter there, although that same presentation clearly shows the
> benefit of larger block sizes which is yet *another* thing for which
> a migration tool would be helpful.

> -Aaron
>
> On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com>
wrote:
> Hi,
>
> I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
> For all Spectrum Scale clusters that are currently at 4.X.X, it is
> possible to migrate to 5.0.0 with no offline data migration and no
> need to move data. Once these clusters are at 5.0.0, they will
> benefit from the performance improvements, new features (such as
> file audit logging), and various enhancements that are included in 5.0.0.
>
> That being said, there is one enhancement that will not be applied
> to these clusters, and that is the increased number of sub-blocks
> per block for small file allocation. This means that for file
> systems with a large block size and a lot of small files, the
> overall space utilization will be the same it currently is in 4.X.X.
> Since file systems created at 4.X.X and earlier used a block size
> that kept this allocation in mind, there should be very little
> impact on existing file systems.
>
> Outside of that one particular function, the remainder of the
> performance improvements, metadata improvements, updated
> compatibility, new functionality, and all of the other enhancements
> will be immediately available to you once you complete the upgrade
> to 5.0.0 -- with no need to reformat, move data, or take your data
offline.
>
> I hope that clarifies things a little and makes the upgrade path
> more accessible.
>
> Please let me know if there are any other questions or concerns.
>
> Thank you,
> Nikhil Khandelwal
> Spectrum Scale Development
> Client Adoption
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-

>
siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/18d266d7/attachment-0001.htm>

From janfrode at tanso.net  Fri Dec  1 12:53:21 2017
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 01 Dec 2017 12:53:21 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
Message-ID: <CAHwPatj2=BUa1zHO453BBtAWvmwhxXoBqSByoEDYKg_PvZEa-A@mail.gmail.com>

Bill, could you say something about what the metadata-storage here was?
ESS/NL-SAS/3way replication?

I just asked about this in the internal slack channel #scale-help today..



-jf

fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <bhartner at us.ibm.com>:

> > "It has a significant performance penalty for small files in large
> > block size filesystems"
>
> Aaron,
>
> Below are mdtest results for a test we ran for CORAL - file size was 32k.
>
> We have not gone back and ran the test on a file system formatted without
> > 32 subblocks. We'll do that at some point...
>
> -Bill
>
> -- started at 10/28/2017 17:51:38 --
>
> mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> Command line used: /tmp/mdtest-binary-dir/mdtest -d
> /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360 -u -y
> Path: /ibm/fs2-16m-10
> FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>
> 228 tasks, 67239936 files
>
> SUMMARY: (of 3 iterations)
> Operation Max Min Mean Std Dev
> --------- --- --- ---- -------
> File creation : 51953.498 50558.517 51423.221 616.643
> File stat : 0.000 0.000 0.000 0.000
> File read : 0.000 0.000 0.000 0.000
> File removal : 96746.376 92149.535 94658.774 1900.187
> Tree creation : 1.588 0.070 0.599 0.700
> Tree removal : 0.213 0.034 0.097 0.082
>
> -- finished at 10/28/2017 19:51:54 --
>
> Bill Hartner
> IBM Systems
> Scalable I/O Development
> Austin, Texas
> bhartner at us.ibm.com
> home office 512-784-0980
>
>
> gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:
>
> > From: Aaron Knister <aaron.knister at gmail.com>
>
>
> > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>
> > Date: 11/29/2017 04:42 PM
>
>
> > Subject: Re: [gpfsug-discuss] Online data migration tool
> > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
> >
>
> > Thanks, Nikhil. Most of that was consistent with my understnading,
> > however I was under the impression that the >32 subblocks code is
> > required to achieve the touted 50k file creates/second that Sven has
> > talked about a bunch of times:
> >
> >
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf
> > http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
> > _Sven_Oehme_-_News_from_Research.pdf
> > http://files.gpfsug.org/presentations/2016/SC16/12_-
> > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>
>
> > from those presentations regarding 32 subblocks:
> >
> > "It has a significant performance penalty for small files in large
> > block size filesystems"
>
> > although I'm not clear on the specific definition of "large". Many
> > filesystems I encounter only have a 1M block size so it may not
> > matter there, although that same presentation clearly shows the
> > benefit of larger block sizes which is yet *another* thing for which
> > a migration tool would be helpful.
>
> > -Aaron
> >
> > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com>
> wrote:
>
> > Hi,
> >
> > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
> > For all Spectrum Scale clusters that are currently at 4.X.X, it is
> > possible to migrate to 5.0.0 with no offline data migration and no
> > need to move data. Once these clusters are at 5.0.0, they will
> > benefit from the performance improvements, new features (such as
> > file audit logging), and various enhancements that are included in 5.0.0.
> >
> > That being said, there is one enhancement that will not be applied
> > to these clusters, and that is the increased number of sub-blocks
> > per block for small file allocation. This means that for file
> > systems with a large block size and a lot of small files, the
> > overall space utilization will be the same it currently is in 4.X.X.
> > Since file systems created at 4.X.X and earlier used a block size
> > that kept this allocation in mind, there should be very little
> > impact on existing file systems.
> >
> > Outside of that one particular function, the remainder of the
> > performance improvements, metadata improvements, updated
> > compatibility, new functionality, and all of the other enhancements
> > will be immediately available to you once you complete the upgrade
> > to 5.0.0 -- with no need to reformat, move data, or take your data
> offline.
> >
> > I hope that clarifies things a little and makes the upgrade path
> > more accessible.
> >
> > Please let me know if there are any other questions or concerns.
> >
> > Thank you,
> > Nikhil Khandelwal
> > Spectrum Scale Development
> > Client Adoption
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
>
> > https://urldefense.proofpoint.com/v2/url?
> >
> u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
> >
> siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/a6a697b3/attachment-0001.htm>

From luis.bolinches at fi.ibm.com  Fri Dec  1 14:00:19 2017
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Fri, 1 Dec 2017 14:00:19 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF9BFF17D3.E70240F6-ON002581E9.0046DD04@LocalDomain>
Message-ID: <OFECC4DC82.CE3786C2-ON002581E9.004CEEB3-1512136819057@notes.na.collabserv.com>

For Hana 2.0 only SP1 and 2 are supported

--
Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations
Luis Bolinches
Consultant IT Specialist
Mobile Phone: +358503112585
https://www.youracclaim.com/user/luis-bolinches

"If you always give you will always have" --  Anonymous

> On 1 Dec 2017, at 14.53, Jan-Frode Myklebust <janfrode at tanso.net> wrote:
> 
> Bill, could you say something about what the metadata-storage here was? ESS/NL-SAS/3way replication?
> 
> I just asked about this in the internal slack channel #scale-help today..
> 
> 
> 
> -jf
> 
>> fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <bhartner at us.ibm.com>:
>> > "It has a significant performance penalty for small files in large 
>> > block size filesystems"
>> 
>> 
>> Aaron,
>> 
>> Below are mdtest results for a test we ran for CORAL - file size was 32k.
>> 
>> We have not gone back and ran the test on a file system formatted without > 32 subblocks. We'll do that at some point...
>> 
>> -Bill
>> 
>> -- started at 10/28/2017 17:51:38 --
>> 
>> mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>> Command line used: /tmp/mdtest-binary-dir/mdtest -d /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360 -u -y
>> Path: /ibm/fs2-16m-10
>> FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>> 
>> 228 tasks, 67239936 files
>> 
>> SUMMARY: (of 3 iterations)
>> Operation Max Min Mean Std Dev
>> --------- --- --- ---- -------
>> File creation : 51953.498 50558.517 51423.221 616.643
>> File stat : 0.000 0.000 0.000 0.000
>> File read : 0.000 0.000 0.000 0.000
>> File removal : 96746.376 92149.535 94658.774 1900.187
>> Tree creation : 1.588 0.070 0.599 0.700
>> Tree removal : 0.213 0.034 0.097 0.082
>> 
>> -- finished at 10/28/2017 19:51:54 --
>> 
>> Bill Hartner
>> IBM Systems
>> Scalable I/O Development
>> Austin, Texas
>> bhartner at us.ibm.com
>> home office 512-784-0980
>> 
>> 
>> gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:
>> 
>> > From: Aaron Knister <aaron.knister at gmail.com>
>> 
>> 
>> > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> 
>> > Date: 11/29/2017 04:42 PM
>> 
>> 
>> > Subject: Re: [gpfsug-discuss] Online data migration tool
>> > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> 
>> > 
>> 
>> > Thanks, Nikhil. Most of that was consistent with my understnading, 
>> > however I was under the impression that the >32 subblocks code is 
>> > required to achieve the touted 50k file creates/second that Sven has
>> > talked about a bunch of times:
>> > 
>> > http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf
>> > http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
>> > _Sven_Oehme_-_News_from_Research.pdf
>> > http://files.gpfsug.org/presentations/2016/SC16/12_-
>> > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>> 
>> 
>> > from those presentations regarding 32 subblocks:
>> > 
>> > "It has a significant performance penalty for small files in large 
>> > block size filesystems"
>> 
>> > although I'm not clear on the specific definition of "large". Many 
>> > filesystems I encounter only have a 1M block size so it may not 
>> > matter there, although that same presentation clearly shows the 
>> > benefit of larger block sizes which is yet *another* thing for which
>> > a migration tool would be helpful.
>> 
>> > -Aaron
>> > 
>> > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com> wrote:
>> 
>> > Hi,
>> > 
>> > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
>> > For all Spectrum Scale clusters that are currently at 4.X.X, it is 
>> > possible to migrate to 5.0.0 with no offline data migration and no 
>> > need to move data. Once these clusters are at 5.0.0, they will 
>> > benefit from the performance improvements, new features (such as 
>> > file audit logging), and various enhancements that are included in 5.0.0.
>> > 
>> > That being said, there is one enhancement that will not be applied 
>> > to these clusters, and that is the increased number of sub-blocks 
>> > per block for small file allocation. This means that for file 
>> > systems with a large block size and a lot of small files, the 
>> > overall space utilization will be the same it currently is in 4.X.X.
>> > Since file systems created at 4.X.X and earlier used a block size 
>> > that kept this allocation in mind, there should be very little 
>> > impact on existing file systems.
>> > 
>> > Outside of that one particular function, the remainder of the 
>> > performance improvements, metadata improvements, updated 
>> > compatibility, new functionality, and all of the other enhancements 
>> > will be immediately available to you once you complete the upgrade 
>> > to 5.0.0 -- with no need to reformat, move data, or take your data offline.
>> > 
>> > I hope that clarifies things a little and makes the upgrade path 
>> > more accessible.
>> > 
>> > Please let me know if there are any other questions or concerns.
>> > 
>> > Thank you,
>> > Nikhil Khandelwal
>> > Spectrum Scale Development
>> > Client Adoption
>> > 
>> > _______________________________________________
>> > gpfsug-discuss mailing list
>> > gpfsug-discuss at spectrumscale.org
>> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> 
>> > _______________________________________________
>> > gpfsug-discuss mailing list
>> > gpfsug-discuss at spectrumscale.org
>> 
>> > https://urldefense.proofpoint.com/v2/url?
>> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>> > siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3 
Registered in Finland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/e09df482/attachment-0001.htm>

From bhartner at us.ibm.com  Fri Dec  1 16:44:23 2017
From: bhartner at us.ibm.com (Bill Hartner)
Date: Fri, 1 Dec 2017 10:44:23 -0600
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com><OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain><OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
Message-ID: <OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>


ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the results @
user group meeting.

-Bill

Bill Hartner
IBM Systems
Scalable I/O Development
Austin, Texas
bhartner at us.ibm.com
home office 512-784-0980




From:	Jan-Frode Myklebust <janfrode at tanso.net>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	12/01/2017 06:53 AM
Subject:	Re: [gpfsug-discuss] Online data migration tool
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Bill, could you say something about what the metadata-storage here was?
ESS/NL-SAS/3way replication?

I just asked about this in the internal slack channel #scale-help today..



-jf

fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <bhartner at us.ibm.com>:
  > "It has a significant performance penalty for small files in large
  > block size filesystems"



  Aaron,

  Below are mdtest results for a test we ran for CORAL - file size was 32k.

  We have not gone back and ran the test on a file system formatted without
  > 32 subblocks. We'll do that at some point...

  -Bill

  -- started at 10/28/2017 17:51:38 --

  mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
  Command line used: /tmp/mdtest-binary-dir/mdtest
  -d /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360
  -u -y
  Path: /ibm/fs2-16m-10
  FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%

  228 tasks, 67239936 files

  SUMMARY: (of 3 iterations)
  Operation Max Min Mean Std Dev
  --------- --- --- ---- -------
  File creation : 51953.498 50558.517 51423.221 616.643
  File stat : 0.000 0.000 0.000 0.000
  File read : 0.000 0.000 0.000 0.000
  File removal : 96746.376 92149.535 94658.774 1900.187
  Tree creation : 1.588 0.070 0.599 0.700
  Tree removal : 0.213 0.034 0.097 0.082

  -- finished at 10/28/2017 19:51:54 --

  Bill Hartner
  IBM Systems
  Scalable I/O Development
  Austin, Texas
  bhartner at us.ibm.com
  home office 512-784-0980


  gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:

  > From: Aaron Knister <aaron.knister at gmail.com>



  > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>


  > Date: 11/29/2017 04:42 PM



  > Subject: Re: [gpfsug-discuss] Online data migration tool
  > Sent by: gpfsug-discuss-bounces at spectrumscale.org


  >


  > Thanks, Nikhil. Most of that was consistent with my understnading,
  > however I was under the impression that the >32 subblocks code is
  > required to achieve the touted 50k file creates/second that Sven has
  > talked about a bunch of times:
  >
  >
  http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf

  > http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
  > _Sven_Oehme_-_News_from_Research.pdf
  > http://files.gpfsug.org/presentations/2016/SC16/12_-
  > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf



  > from those presentations regarding 32 subblocks:
  >
  > "It has a significant performance penalty for small files in large
  > block size filesystems"

  > although I'm not clear on the specific definition of "large". Many
  > filesystems I encounter only have a 1M block size so it may not
  > matter there, although that same presentation clearly shows the
  > benefit of larger block sizes which is yet *another* thing for which
  > a migration tool would be helpful.

  > -Aaron
  >
  > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com>
  wrote:


  > Hi,
  >
  > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
  > For all Spectrum Scale clusters that are currently at 4.X.X, it is
  > possible to migrate to 5.0.0 with no offline data migration and no
  > need to move data. Once these clusters are at 5.0.0, they will
  > benefit from the performance improvements, new features (such as
  > file audit logging), and various enhancements that are included in
  5.0.0.
  >
  > That being said, there is one enhancement that will not be applied
  > to these clusters, and that is the increased number of sub-blocks
  > per block for small file allocation. This means that for file
  > systems with a large block size and a lot of small files, the
  > overall space utilization will be the same it currently is in 4.X.X.
  > Since file systems created at 4.X.X and earlier used a block size
  > that kept this allocation in mind, there should be very little
  > impact on existing file systems.
  >
  > Outside of that one particular function, the remainder of the
  > performance improvements, metadata improvements, updated
  > compatibility, new functionality, and all of the other enhancements
  > will be immediately available to you once you complete the upgrade
  > to 5.0.0 -- with no need to reformat, move data, or take your data
  offline.
  >
  > I hope that clarifies things a little and makes the upgrade path
  > more accessible.
  >
  > Please let me know if there are any other questions or concerns.
  >
  > Thank you,
  > Nikhil Khandelwal
  > Spectrum Scale Development
  > Client Adoption
  >
  > _______________________________________________
  > gpfsug-discuss mailing list
  > gpfsug-discuss at spectrumscale.org
  > http://gpfsug.org/mailman/listinfo/gpfsug-discuss



  > _______________________________________________
  > gpfsug-discuss mailing list
  > gpfsug-discuss at spectrumscale.org


  > https://urldefense.proofpoint.com/v2/url?
  >
  u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-

  >
  siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=


  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at spectrumscale.org
  http://gpfsug.org/mailman/listinfo/gpfsug-discuss
  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at spectrumscale.org
  https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/60b7fce0/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/60b7fce0/attachment-0001.gif>

From r.sobey at imperial.ac.uk  Fri Dec  1 16:57:46 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Fri, 1 Dec 2017 16:57:46 +0000
Subject: [gpfsug-discuss] Policy rule to list files based on access time
Message-ID: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Hi all,

I'm getting some strange results using a policy to list all the files (well, I'm only interested in the total size that GPFS chooses) that have not been accessed for more than 1/2/3 etc years.

The policy rule I'm using is:

RULE 'list_old' LIST 'atime_gt_730'
WHERE DAYS(CURRENT_TIMESTAMP) -DAYS(ACCESS_TIME) > 730

..for anything over 2 years. My filesystem is 3.5 years old and we first started migrating to it about three years ago, yet although GPFS is choosing ~400000 files that match, the total size is just north of 11000KB:

[I] Summary of Rule Applicability and File Choices:
Rule#      Hit_Cnt          KB_Hit          Chosen       KB_Chosen          KB_Ill     Rule
     0       453765           11776          453765           11776               0     RULE 'list_old' LIST 'atime_gt_730' WHERE(.)

I'm having a hard time believing half a million files total 11MB especially given the very random spread of usage the system sees.

The mmapplypolicy command I'm running is:

/usr/lpp/mmfs/bin/mmapplypolicy /gpfs/path -f /gpfs/policies/ -I defer --scope filesystem -P ./list_old_allfs.txt

The directory 'path' above is the root of the main CES share which encompasses all my filesets.

This is my first real foray into using the policy engine so I might be doing something silly, or misinterpreting the results, or there is simply a far easier way to achieve my goal.

Can anyone help?

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/69ada640/attachment-0001.htm>

From alex at calicolabs.com  Fri Dec  1 17:18:57 2017
From: alex at calicolabs.com (Alex Chekholko)
Date: Fri, 1 Dec 2017 09:18:57 -0800
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <CANcy_PbUS93Y7TMxL2f+LcH7_4fJk0SkpJED54n3h=2Ju5Lzfg@mail.gmail.com>

Hi Richard,

Either add something like "-L 2" to your mmapplypolicy command or add an
'EXEC ...' to your policy file to have it print out the actual files that
matched. There is an example exec script somewhere in the samples folder

I have found that in the past when I doubted the policy results, there was
something I misunderstood. So just go ahead and examine the metadata on the
actual files that match.

Perhaps at some point in the past, you had some kind of application scan
through the filesystem and thus all the access times got updated to newer
than you expect.

For the file count/size numbers, I don't find that unusual; there are all
kinds of user applications that have pathological patterns like 450k files
totaling 11MB.  But worth double-checking.  You can run some example
policies like the one you ran, but list only >10TB files or list only files
of 0 size; you never know what you might find.

Regards,
Alex


On Fri, Dec 1, 2017 at 8:57 AM, Sobey, Richard A <r.sobey at imperial.ac.uk>
wrote:

> Hi all,
>
>
>
> I?m getting some strange results using a policy to list all the files
> (well, I?m only interested in the total size that GPFS chooses) that have
> not been accessed for more than 1/2/3 etc years.
>
>
>
> The policy rule I?m using is:
>
>
>
> RULE 'list_old' LIST 'atime_gt_730'
>
> WHERE DAYS(CURRENT_TIMESTAMP) -DAYS(ACCESS_TIME) > 730
>
>
>
> ..for anything over 2 years. My filesystem is 3.5 years old and we first
> started migrating to it about three years ago, yet although GPFS is
> choosing ~400000 files that match, the total size is just north of 11000KB:
>
>
>
> [I] Summary of Rule Applicability and File Choices:
>
> Rule#      Hit_Cnt          KB_Hit          Chosen
> KB_Chosen          KB_Ill     Rule
>
>      0       453765           11776          453765
> 11776               0     RULE 'list_old' LIST 'atime_gt_730' WHERE(.)
>
>
>
> I?m having a hard time believing half a million files total 11MB
> especially given the very random spread of usage the system sees.
>
>
>
> The mmapplypolicy command I?m running is:
>
>
>
> /usr/lpp/mmfs/bin/mmapplypolicy /gpfs/path -f /gpfs/policies/ -I defer
> --scope filesystem -P ./list_old_allfs.txt
>
>
>
> The directory ?path? above is the root of the main CES share which
> encompasses all my filesets.
>
>
>
> This is my first real foray into using the policy engine so I might be
> doing something silly, or misinterpreting the results, or there is simply a
> far easier way to achieve my goal.
>
>
>
> Can anyone help?
>
>
>
> Thanks
>
> Richard
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/1395f933/attachment-0001.htm>

From ewahl at osc.edu  Fri Dec  1 17:59:21 2017
From: ewahl at osc.edu (Edward Wahl)
Date: Fri, 1 Dec 2017 12:59:21 -0500
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <20171201125921.507043a3@osc.edu>


You don't specify a choice algorithm?  Might want to change your script with a
quick thing like this to view all the candidates.  I use a similar 120day and
I've looked at atime/ctimes of them before and was happy with the output.
You can then quickly size the output with a quick sed/awk/whatever.

SHOW (varchar(file_size) || '  ' || varchar(USER_ID))
WHERE DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME) >= 180 

and add "--choice-algorithm exact" to the command line to see what files match. 

I can't recall what the default is. Might already be exact?  Oops, man pages
says yes.  Might then run all 3 and see if the output is much different. 

Ed

On Fri, 1 Dec 2017 16:57:46 +0000
"Sobey, Richard A" <r.sobey at imperial.ac.uk> wrote:

> Hi all,
> 
> I'm getting some strange results using a policy to list all the files (well,
> I'm only interested in the total size that GPFS chooses) that have not been
> accessed for more than 1/2/3 etc years.
> 
> The policy rule I'm using is:
> 
> RULE 'list_old' LIST 'atime_gt_730'
> WHERE DAYS(CURRENT_TIMESTAMP) -DAYS(ACCESS_TIME) > 730
> 
> ..for anything over 2 years. My filesystem is 3.5 years old and we first
> started migrating to it about three years ago, yet although GPFS is choosing
> ~400000 files that match, the total size is just north of 11000KB:
> 
> [I] Summary of Rule Applicability and File Choices:
> Rule#      Hit_Cnt          KB_Hit          Chosen       KB_Chosen
> KB_Ill     Rule 0       453765           11776          453765
> 11776               0     RULE 'list_old' LIST 'atime_gt_730' WHERE(.)
> 
> I'm having a hard time believing half a million files total 11MB especially
> given the very random spread of usage the system sees.
> 
> The mmapplypolicy command I'm running is:
> 
> /usr/lpp/mmfs/bin/mmapplypolicy /gpfs/path -f /gpfs/policies/ -I defer
> --scope filesystem -P ./list_old_allfs.txt
> 
> The directory 'path' above is the root of the main CES share which
> encompasses all my filesets.
> 
> This is my first real foray into using the policy engine so I might be doing
> something silly, or misinterpreting the results, or there is simply a far
> easier way to achieve my goal.
> 
> Can anyone help?
> 
> Thanks
> Richard



-- 

Ed Wahl
Ohio Supercomputer Center
614-292-9302


From makaplan at us.ibm.com  Sat Dec  2 18:03:25 2017
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Sat, 2 Dec 2017 13:03:25 -0500
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>
Message-ID: <OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>

You might want to look at FILE_SIZE.  KB_ALLOCATED will be 0 if the file 
data fits into the inode.

You might also want to use SIZE(FILE_SIZE) in the policy LIST rule, this 
will cause the KB_HIT and KB_CHOSEN numbers to be the sum of FILE_SIZEs 
instead of the default SIZE(KB_ALLOCATED).

--marc of GPFS

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171202/1703340b/attachment-0001.htm>

From r.sobey at imperial.ac.uk  Sat Dec  2 18:53:14 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Sat, 2 Dec 2017 18:53:14 +0000
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>,
	<OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>
Message-ID: <DB6PR0601MB210125AFF8783FFFAB2BF3E4DF3E0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Thank you all for your replies. I will take a look at them on Monday. And reply individually where I need to..

Richard

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Marc A Kaplan <makaplan at us.ibm.com>
Sent: Saturday, December 2, 2017 6:03:25 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Policy rule to list files based on access time

You might want to look at FILE_SIZE.  KB_ALLOCATED will be 0 if the file data fits into the inode.

You might also want to use SIZE(FILE_SIZE) in the policy LIST rule, this will cause the KB_HIT and KB_CHOSEN numbers to be the sum of FILE_SIZEs instead of the default SIZE(KB_ALLOCATED).

--marc of GPFS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171202/815f8412/attachment-0001.htm>

From bipcuds at gmail.com  Mon Dec  4 02:49:06 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Sun, 3 Dec 2017 21:49:06 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file quota/limit
	possible to set?
Message-ID: <CAAxuGpEFH7mi80Aa2u_LbaHP+3JU1y4qKQPFOB+HohCD76XZ2Q@mail.gmail.com>

From: Keith Ball <kball at redlineperf.com>
To: gpfsug-discuss at spectrumscale.org
Cc:
Bcc:
Date: Sun, 3 Dec 2017 16:13:13 -0500
Subject: Smallest block quota/limit and file quota/limit possible to set?
HI All,

We have a system where all users have their own private group as well.
However, for a given fileset (we are using --perfileset-quota), we would
like to ONLY allow users who also belong to just a few central groups to be
able to write to the fileset.

That is, user "userA" has its own "groupA", but we only want the user to be
able to write to the fileset if:
 - userA belongs to one of the groups (e.g. group1, group2, group3) that
have explicitly set quotas
 - The group(s) in question are within quota/limits.

In general, we do not want any users that do NOT belong to one of the three
groups with enabled quotas to be able to write anything at all to the
fileset.

Is there a way to set a ZERO quota for block/file in GPFS, that means what
it actually should mean? i.e. "Your limit is 0 file = you cannot create
files in this fileset". Creating some kind of "supergroup" owner of the
fileset (with entitled users as members of the group) could work, but that
will only work for *one* group.

If we cannot set the block and file limits to zero, what *are* the smallest
block and fie limits? In GPFS 3.5, they seem to be 1760MB for block. Is
there a smallest quota for files? (blocksize is 16MB, which will be reduced
to 4MB probably, in a subsequent cluster).

Many Thanks,
  Keith



-- 
Keith D. Ball, PhD
RedLine Performance Solutions, LLC
web:  http://www.redlineperf.com/
email: kball at redlineperf.com <aqualkenbush at redlineperf.com>
cell: 540-557-7851 <(540)%20557-7851>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171203/8c867510/attachment-0001.htm>

From scale at us.ibm.com  Mon Dec  4 03:49:38 2017
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Mon, 4 Dec 2017 09:19:38 +0530
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit	possible to set?
In-Reply-To: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
Message-ID: <OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>

Hi Keith,

You can use ACLs for fine grained permissions. A quota limit of 0 in GPFS 
implies no limits.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   Keith Ball <bipcuds at gmail.com>
To:     gpfsug-discuss at spectrumscale.org
Date:   12/04/2017 08:19 AM
Subject:        [gpfsug-discuss] Smallest block quota/limit and file 
quota/limit     possible to set?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



From: Keith Ball <kball at redlineperf.com>
To: gpfsug-discuss at spectrumscale.org
Cc: 
Bcc: 
Date: Sun, 3 Dec 2017 16:13:13 -0500
Subject: Smallest block quota/limit and file quota/limit possible to set?
HI All,

We have a system where all users have their own private group as well. 
However, for a given fileset (we are using --perfileset-quota), we would 
like to ONLY allow users who also belong to just a few central groups to 
be able to write to the fileset. 

That is, user "userA" has its own "groupA", but we only want the user to 
be able to write to the fileset if:
 - userA belongs to one of the groups (e.g. group1, group2, group3) that 
have explicitly set quotas
 - The group(s) in question are within quota/limits.

In general, we do not want any users that do NOT belong to one of the 
three groups with enabled quotas to be able to write anything at all to 
the fileset.

Is there a way to set a ZERO quota for block/file in GPFS, that means what 
it actually should mean? i.e. "Your limit is 0 file = you cannot create 
files in this fileset". Creating some kind of "supergroup" owner of the 
fileset (with entitled users as members of the group) could work, but that 
will only work for *one* group. 

If we cannot set the block and file limits to zero, what *are* the 
smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB for 
block. Is there a smallest quota for files? (blocksize is 16MB, which will 
be reduced to 4MB probably, in a subsequent cluster).

Many Thanks,
  Keith 



-- 
Keith D. Ball, PhD
RedLine Performance Solutions, LLC
web:  http://www.redlineperf.com/
email: kball at redlineperf.com
cell: 540-557-7851_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=N0b8UJpG1zFuuQjvCE1SaaIG2EUDv4v783wDc3TbnyM&s=vuNQN4VJonV9nmcTEOKKr-hL2LAWSOUV7HKAQBKWcK4&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/0c8ffd87/attachment-0001.htm>

From z.han at imperial.ac.uk  Mon Dec  4 09:52:31 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Mon, 4 Dec 2017 09:52:31 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or above?
Message-ID: <alpine.LFD.2.20.1712040941230.2395@laptop>

Hi All,

Any one is using a Linux kernel 3.12.x or above
to run gpfs 4.2.3-4.2? I mean you've compiled
your own kernel without paying for a professional
service.

We're stuck by CentOS/RHEL's distributed kernel
as the PCI passthrough is required for VMs. Your
comments or suggestions are much appreciated.

Kind regards,

Zong-Pei



From S.J.Thompson at bham.ac.uk  Mon Dec  4 10:29:27 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Mon, 4 Dec 2017 10:29:27 +0000
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712040941230.2395@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
Message-ID: <B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>

The FAQ at:

https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux

Lists support with (e.g. Ubutu 16.04.2) with kernel 4.4.0-62, so likely it would work with a build your own kernel, but that doesn?t mean it is **supported**

Simon

On 04/12/2017, 09:52, "gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk" <gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk> wrote:

    Hi All,
    
    Any one is using a Linux kernel 3.12.x or above
    to run gpfs 4.2.3-4.2? I mean you've compiled
    your own kernel without paying for a professional
    service.
    
    We're stuck by CentOS/RHEL's distributed kernel
    as the PCI passthrough is required for VMs. Your
    comments or suggestions are much appreciated.
    
    Kind regards,
    
    Zong-Pei
    
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
    


From z.han at imperial.ac.uk  Mon Dec  4 10:37:39 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Mon, 4 Dec 2017 10:37:39 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
	<B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>
Message-ID: <alpine.LFD.2.20.1712041036340.2395@laptop>

Thanks, Simon.

Only on clients (oh so-called clients!) ;)


On Mon, 4 Dec 2017, Simon Thompson (IT Research Support) wrote:
> Date: Mon, 4 Dec 2017 10:29:27 +0000
> From: "Simon Thompson (IT Research Support)" <S.J.Thompson at bham.ac.uk>
> Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
>     above?
> 
> The FAQ at:
>
> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux
>
> Lists support with (e.g. Ubutu 16.04.2) with kernel 4.4.0-62, so likely it would work with a build your own kernel, but that doesn?t mean it is **supported**
>
> Simon
>
> On 04/12/2017, 09:52, "gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk" <gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk> wrote:
>
>    Hi All,
>
>    Any one is using a Linux kernel 3.12.x or above
>    to run gpfs 4.2.3-4.2? I mean you've compiled
>    your own kernel without paying for a professional
>    service.
>
>    We're stuck by CentOS/RHEL's distributed kernel
>    as the PCI passthrough is required for VMs. Your
>    comments or suggestions are much appreciated.
>
>    Kind regards,
>
>    Zong-Pei
>
>    _______________________________________________
>    gpfsug-discuss mailing list
>    gpfsug-discuss at spectrumscale.org
>    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From ulmer at ulmer.org  Mon Dec  4 13:46:31 2017
From: ulmer at ulmer.org (Stephen Ulmer)
Date: Mon, 4 Dec 2017 08:46:31 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit	possible to set?
In-Reply-To: <OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
Message-ID: <8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>

I don?t understand why not having permission(s) doesn?t prevent the user from writing into the fileset...

As described, your case is about not wanting userA to be able to write to a fileset if userA isn?t in some groups. Don?t put them in those groups. That?s not even Spectrum Scale specific, it?s about generic *nix permissions.

What am I missing? I don?t understand why you would want to use quota to enforce permissions. (There could be a legitimate reason here, but I don?t understand it.)

Liberty,

-- 
Stephen Ulmer

Sent from a mobile device; please excuse autocorrect silliness.

> On Dec 3, 2017, at 10:49 PM, IBM Spectrum Scale <scale at us.ibm.com> wrote:
> 
> Hi Keith,
> 
> You can use ACLs for fine grained permissions. A quota limit of 0 in GPFS implies no limits.
> 
> Regards, The Spectrum Scale (GPFS) team
> 
> ------------------------------------------------------------------------------------------------------------------
> If you feel that your question can benefit other users of  Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. 
> 
> If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact  1-800-237-5511 in the United States or your local IBM Service Center in other countries. 
> 
> The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.
> 
> 
> 
> From:        Keith Ball <bipcuds at gmail.com>
> To:        gpfsug-discuss at spectrumscale.org
> Date:        12/04/2017 08:19 AM
> Subject:        [gpfsug-discuss] Smallest block quota/limit and file quota/limit        possible to set?
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> 
> 
> 
> From: Keith Ball <kball at redlineperf.com>
> To: gpfsug-discuss at spectrumscale.org
> Cc: 
> Bcc: 
> Date: Sun, 3 Dec 2017 16:13:13 -0500
> Subject: Smallest block quota/limit and file quota/limit possible to set?
> HI All,
> 
> We have a system where all users have their own private group as well. However, for a given fileset (we are using --perfileset-quota), we would like to ONLY allow users who also belong to just a few central groups to be able to write to the fileset. 
> 
> That is, user "userA" has its own "groupA", but we only want the user to be able to write to the fileset if:
>  - userA belongs to one of the groups (e.g. group1, group2, group3) that have explicitly set quotas
>  - The group(s) in question are within quota/limits.
> 
> In general, we do not want any users that do NOT belong to one of the three groups with enabled quotas to be able to write anything at all to the fileset.
> 
> Is there a way to set a ZERO quota for block/file in GPFS, that means what it actually should mean? i.e. "Your limit is 0 file = you cannot create files in this fileset". Creating some kind of "supergroup" owner of the fileset (with entitled users as members of the group) could work, but that will only work for *one* group. 
> 
> If we cannot set the block and file limits to zero, what *are* the smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB for block. Is there a smallest quota for files? (blocksize is 16MB, which will be reduced to 4MB probably, in a subsequent cluster).
> 
> Many Thanks,
>   Keith 
> 
> 
> 
> -- 
> Keith D. Ball, PhD
> RedLine Performance Solutions, LLC
> web:  http://www.redlineperf.com/
> email: kball at redlineperf.com
> cell: 540-557-7851_______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=N0b8UJpG1zFuuQjvCE1SaaIG2EUDv4v783wDc3TbnyM&s=vuNQN4VJonV9nmcTEOKKr-hL2LAWSOUV7HKAQBKWcK4&e=
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/9b42e1fc/attachment-0001.htm>

From valdis.kletnieks at vt.edu  Mon Dec  4 16:27:38 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Mon, 04 Dec 2017 11:27:38 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit possible to set?
In-Reply-To: <8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
Message-ID: <19755.1512404858@turing-police.cc.vt.edu>

On Mon, 04 Dec 2017 08:46:31 -0500, Stephen Ulmer said:

> As described, your case is about not wanting userA to be able to write to a
> fileset if userA isn???t in some groups. Don???t put them in those groups. That???s
> not even Spectrum Scale specific, it???s about generic *nix permissions.

The problem is that doesn't stop userA from scribbling into a world-writable directory,
and they may have some of those because more than one group needs to write in
a specific directory and ACLs are problematic for other reasons....
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/18b3756a/attachment-0001.sig>

From stuartb at 4gh.net  Mon Dec  4 16:33:17 2017
From: stuartb at 4gh.net (Stuart Barkley)
Date: Mon, 4 Dec 2017 11:33:17 -0500 (EST)
Subject: [gpfsug-discuss] Smallest block quota/limit and file
 quota/limit possible to set?
In-Reply-To: <8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
Message-ID: <alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>

We have a /projects filesystem where individual projects can "buy" a
specific amount of disk space.  We enforce this purchase limit by
creating a specific group for the allocation, adding designated users
to the group and setting a group quota.

This works fine as long as the users properly use setgid directories
and keep proper group ownership of the files and directories.

However, for various reasons our users keep creating files and
directories with incorrect group ownership.  In most cases this is
accidental and eventually causes problems when other group members
need to access the files.  In abusive cases (not yet seen) people
could use this to exceed project disk space allocations.

To address this problem we have default quotas set to about 2GB (the
smallest we seem to be able to set).  This prevents users for
consuming too much unpurchased disk space.  However, this continues to
allow users to create files and directories with incorrect group
ownership and it takes users a while to discover their error.  User
education and cleanup becomes a problem long after the user thinks
things are working.

We would like to have groups without quota definitions to not be able
to create any files.  This would prevent accidental file creation at
the first attempt.

Stuart Barkley

On Mon, 4 Dec 2017 at 08:46 -0000, Stephen Ulmer wrote:

> I don?t understand why not having permission(s) doesn?t prevent the
> user from writing into the fileset...
>
> As described, your case is about not wanting userA to be able to
> write to a fileset if userA isn?t in some groups. Don?t put them in
> those groups. That?s not even Spectrum Scale specific, it?s about
> generic *nix permissions.
>
> What am I missing? I don?t understand why you would want to use
> quota to enforce permissions. (There could be a legitimate reason
> here, but I don?t understand it.)
>
> Liberty,
>
> --
> Stephen Ulmer
>
> Sent from a mobile device; please excuse autocorrect silliness.
>
> > On Dec 3, 2017, at 10:49 PM, IBM Spectrum Scale <scale at us.ibm.com> wrote:
> >
> > Hi Keith,
> >
> > You can use ACLs for fine grained permissions. A quota limit of 0
> > in GPFS implies no limits.
> >
> > Regards, The Spectrum Scale (GPFS) team
> >
> > From:        Keith Ball <bipcuds at gmail.com>
> > To:        gpfsug-discuss at spectrumscale.org
> > Date:        12/04/2017 08:19 AM
> > Subject:        [gpfsug-discuss] Smallest block quota/limit and file quota/limit        possible to set?
> > Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> >
> > HI All,
> >
> > We have a system where all users have their own private group as
> > well. However, for a given fileset (we are using
> > --perfileset-quota), we would like to ONLY allow users who also
> > belong to just a few central groups to be able to write to the
> > fileset.
> >
> > That is, user "userA" has its own "groupA", but we only want the
> > user to be able to write to the fileset if:
> >  - userA belongs to one of the groups (e.g. group1, group2,
> >  group3) that have explicitly set quotas
> >  - The group(s) in question are within quota/limits.
> >
> > In general, we do not want any users that do NOT belong to one of
> > the three groups with enabled quotas to be able to write anything
> > at all to the fileset.
> >
> > Is there a way to set a ZERO quota for block/file in GPFS, that
> > means what it actually should mean? i.e. "Your limit is 0 file =
> > you cannot create files in this fileset". Creating some kind of
> > "supergroup" owner of the fileset (with entitled users as members
> > of the group) could work, but that will only work for *one* group.
> >
> > If we cannot set the block and file limits to zero, what *are* the
> > smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB
> > for block. Is there a smallest quota for files? (blocksize is
> > 16MB, which will be reduced to 4MB probably, in a subsequent
> > cluster).
> >
> > Many Thanks,
> >   Keith


From S.J.Thompson at bham.ac.uk  Mon Dec  4 17:08:19 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Mon, 4 Dec 2017 17:08:19 +0000
Subject: [gpfsug-discuss] Smallest block quota/limit and file
 quota/limit possible to set?
In-Reply-To: <alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>,
	<alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>

Stuart,

Have you looked at using filesets instead an using fileset quotas to achieve this?

This is what we do and the max number of filesets (currently) isn't an issue for us.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of stuartb at 4gh.net [stuartb at 4gh.net]
Sent: 04 December 2017 16:33
To: gpfsug main discussion list
Cc: gpfsug-discuss-bounces at spectrumscale.org
Subject: Re: [gpfsug-discuss] Smallest block quota/limit and file quota/limit possible to set?

We have a /projects filesystem where individual projects can "buy" a
specific amount of disk space.  We enforce this purchase limit by
creating a specific group for the allocation, adding designated users
to the group and setting a group quota.

This works fine as long as the users properly use setgid directories
and keep proper group ownership of the files and directories.

However, for various reasons our users keep creating files and
directories with incorrect group ownership.  In most cases this is
accidental and eventually causes problems when other group members
need to access the files.  In abusive cases (not yet seen) people
could use this to exceed project disk space allocations.

To address this problem we have default quotas set to about 2GB (the
smallest we seem to be able to set).  This prevents users for
consuming too much unpurchased disk space.  However, this continues to
allow users to create files and directories with incorrect group
ownership and it takes users a while to discover their error.  User
education and cleanup becomes a problem long after the user thinks
things are working.

We would like to have groups without quota definitions to not be able
to create any files.  This would prevent accidental file creation at
the first attempt.

Stuart Barkley

On Mon, 4 Dec 2017 at 08:46 -0000, Stephen Ulmer wrote:

> I don?t understand why not having permission(s) doesn?t prevent the
> user from writing into the fileset...
>
> As described, your case is about not wanting userA to be able to
> write to a fileset if userA isn?t in some groups. Don?t put them in
> those groups. That?s not even Spectrum Scale specific, it?s about
> generic *nix permissions.
>
> What am I missing? I don?t understand why you would want to use
> quota to enforce permissions. (There could be a legitimate reason
> here, but I don?t understand it.)
>
> Liberty,
>
> --
> Stephen Ulmer
>
> Sent from a mobile device; please excuse autocorrect silliness.
>
> > On Dec 3, 2017, at 10:49 PM, IBM Spectrum Scale <scale at us.ibm.com> wrote:
> >
> > Hi Keith,
> >
> > You can use ACLs for fine grained permissions. A quota limit of 0
> > in GPFS implies no limits.
> >
> > Regards, The Spectrum Scale (GPFS) team
> >
> > From:        Keith Ball <bipcuds at gmail.com>
> > To:        gpfsug-discuss at spectrumscale.org
> > Date:        12/04/2017 08:19 AM
> > Subject:        [gpfsug-discuss] Smallest block quota/limit and file quota/limit        possible to set?
> > Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> >
> > HI All,
> >
> > We have a system where all users have their own private group as
> > well. However, for a given fileset (we are using
> > --perfileset-quota), we would like to ONLY allow users who also
> > belong to just a few central groups to be able to write to the
> > fileset.
> >
> > That is, user "userA" has its own "groupA", but we only want the
> > user to be able to write to the fileset if:
> >  - userA belongs to one of the groups (e.g. group1, group2,
> >  group3) that have explicitly set quotas
> >  - The group(s) in question are within quota/limits.
> >
> > In general, we do not want any users that do NOT belong to one of
> > the three groups with enabled quotas to be able to write anything
> > at all to the fileset.
> >
> > Is there a way to set a ZERO quota for block/file in GPFS, that
> > means what it actually should mean? i.e. "Your limit is 0 file =
> > you cannot create files in this fileset". Creating some kind of
> > "supergroup" owner of the fileset (with entitled users as members
> > of the group) could work, but that will only work for *one* group.
> >
> > If we cannot set the block and file limits to zero, what *are* the
> > smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB
> > for block. Is there a smallest quota for files? (blocksize is
> > 16MB, which will be reduced to 4MB probably, in a subsequent
> > cluster).
> >
> > Many Thanks,
> >   Keith
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From stuartb at 4gh.net  Mon Dec  4 20:24:50 2017
From: stuartb at 4gh.net (Stuart Barkley)
Date: Mon, 4 Dec 2017 15:24:50 -0500 (EST)
Subject: [gpfsug-discuss] Smallest block quota/limit and file
 quota/limit possible to set?
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>,
	<alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
Message-ID: <alpine.BSF.2.21.1712041521500.34093@freeman.4gh.net>

On Mon, 4 Dec 2017 at 12:08 -0000, Simon Thompson (IT Research Support) wrote:

> Have you looked at using filesets instead an using fileset quotas to
> achieve this?
>
> This is what we do and the max number of filesets (currently) isn't
> an issue for us.

This is a possibility.  We will be moving to a new infrastructure in a
couple of months and I'll have our administrator and contractor
discuss this possibility.

Thanks,
Stuart
-- 
I've never been lost; I was once bewildered for three days, but never lost!
                                        --  Daniel Boone


From valdis.kletnieks at vt.edu  Mon Dec  4 20:44:34 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Mon, 04 Dec 2017 15:44:34 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit possible to set?
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>,
	<alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
Message-ID: <50130.1512420274@turing-police.cc.vt.edu>

On Mon, 04 Dec 2017 17:08:19 +0000, "Simon Thompson (IT Research Support)" said:
> Have you looked at using filesets instead an using fileset quotas to achieve this?

Note that fileset quotas aren't able to represent "No Storage Allowed" either....
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/c92bcc95/attachment-0001.sig>

From Greg.Lehmann at csiro.au  Mon Dec  4 23:28:05 2017
From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au)
Date: Mon, 4 Dec 2017 23:28:05 +0000
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712041036340.2395@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
	<B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>
	<alpine.LFD.2.20.1712041036340.2395@laptop>
Message-ID: <e65b855db0924e62abed07a24be62251@exch1-cdc.nexus.csiro.au>

We run GPFS client SW on SLES 12 SP2 which has a 4.4 kernel. It is only at 4.2.3-1 at present.

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of z.han at imperial.ac.uk
Sent: Monday, 4 December 2017 8:38 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or above?

Thanks, Simon.

Only on clients (oh so-called clients!) ;)


On Mon, 4 Dec 2017, Simon Thompson (IT Research Support) wrote:
> Date: Mon, 4 Dec 2017 10:29:27 +0000
> From: "Simon Thompson (IT Research Support)" <S.J.Thompson at bham.ac.uk>
> Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
>     above?
> 
> The FAQ at:
>
> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux
>
> Lists support with (e.g. Ubutu 16.04.2) with kernel 4.4.0-62, so likely it would work with a build your own kernel, but that doesn?t mean it is **supported**
>
> Simon
>
> On 04/12/2017, 09:52, "gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk" <gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk> wrote:
>
>    Hi All,
>
>    Any one is using a Linux kernel 3.12.x or above
>    to run gpfs 4.2.3-4.2? I mean you've compiled
>    your own kernel without paying for a professional
>    service.
>
>    We're stuck by CentOS/RHEL's distributed kernel
>    as the PCI passthrough is required for VMs. Your
>    comments or suggestions are much appreciated.
>
>    Kind regards,
>
>    Zong-Pei
>
>    _______________________________________________
>    gpfsug-discuss mailing list
>    gpfsug-discuss at spectrumscale.org
>    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From r.sobey at imperial.ac.uk  Tue Dec  5 10:38:37 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 5 Dec 2017 10:38:37 +0000
Subject: [gpfsug-discuss] Hidden fileset
Message-ID: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Quick question: is there a way to make a fileset hidden so it doesn't appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

I want to migrate a load of files in but I don't want the customer to access them prematurely.

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/ab0283f4/attachment-0001.htm>

From robert.horton at icr.ac.uk  Tue Dec  5 10:47:51 2017
From: robert.horton at icr.ac.uk (Robert Horton)
Date: Tue, 5 Dec 2017 10:47:51 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <1512470871.4350.27.camel@icr.ac.uk>

On Tue, 2017-12-05 at 10:38 +0000, Sobey, Richard A wrote:
Quick question: is there a way to make a fileset hidden so it doesn?t appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

Can you link it somewhere obscure, put the data in, then re-link where you actually want it?

Rob

--

Robert Horton | Research Data Storage Lead
The Institute of Cancer Research | 237 Fulham Road | London | SW3 6JB
T +44 (0)20 7153 5350 | E robert.horton at icr.ac.uk<mailto:dugan.witherick at icr.ac.uk> | W www.icr.ac.uk | Twitter @ICR_London
Facebook: www.facebook.com/theinstituteofcancerresearch

Making the discoveries that defeat cancer

The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.

This e-mail message is confidential and for use by the addressee only.  If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/d2edfd94/attachment-0001.htm>

From jonathan.buzzard at strath.ac.uk  Tue Dec  5 10:56:19 2017
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Tue, 05 Dec 2017 10:56:19 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <1512470871.4350.27.camel@icr.ac.uk>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<1512470871.4350.27.camel@icr.ac.uk>
Message-ID: <1512471379.18554.162.camel@strath.ac.uk>

On Tue, 2017-12-05 at 10:47 +0000, Robert Horton wrote:
> On Tue, 2017-12-05 at 10:38 +0000, Sobey, Richard A wrote:
> > Quick question: is there a way to make a fileset hidden so it
> > doesn?t appear when browsing the filesystem through a CES SMB
> > export? Could I do something clever with the directory mask?
> 
> Can you link it somewhere obscure, put the data in, then re-link
> where you actually want it?
> 

Would not setting the DOS hidden attribute on the directory do the
trick? That's assuming the CES SMB is doing everything correctly, no
experience with that but if it is that will do the trick.

Obviously if a user has Explorer set to view hidden files and folders
they will see it.

The other option would be to do something like

chown root:root /path/to/fileset
chmod 700 /path/to/fileset

Then from memory Samba won't show it because you don't have permissions
to access it.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG



From chair at spectrumscale.org  Tue Dec  5 10:58:33 2017
From: chair at spectrumscale.org (Simon Thompson)
Date: Tue, 05 Dec 2017 10:58:33 +0000
Subject: [gpfsug-discuss] SSUG @ CIUK next week [Tuesday 12th December 2017]
Message-ID: <00CFF55D-4D04-4F8D-90D9-F26AF7DA3B1D@spectrumscale.org>

Hi All,

 

Just a reminder that the last #SSUG of 2017 is happening next week as part of Computing Insights UK (CIUK) in Manchester. The agenda for the session is:

 

2pm Introductions and welcome, Simon Thompson & Claire O?Toole (SSUG Committee) 
2:10pm Spectrum Scale 5.0, Daniel Kidger (IBM) 
2:40pm Deep Dive: Blueprints, Ulf Troppens (IBM) 
3:40pm User Talk: Institute of Cancer Research, Robert Horton (ICR) 
4:00pm User Talk: Red Bull Technology, John Brookes (Red Bull Technology) 
4:20pm Questions and discussion 
4:50pm Close

 

To attend, you must be registered as a delegate of CIUK:

https://eventbooking.stfc.ac.uk/news-events/ciuk-2017

 

Simon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/22096afa/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Tue Dec  5 11:04:40 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Tue, 5 Dec 2017 11:04:40 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A3A3@EX13.adf.bham.ac.uk>

Don't give them permissions on the fileset and use the smb option:

"hideunreadable=yes"

https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htmhttps://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htm

This enables access based enumeration.

Simon
________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sobey, Richard A [r.sobey at imperial.ac.uk]
Sent: 05 December 2017 10:38
To: 'gpfsug-discuss at spectrumscale.org'
Subject: ?spam? [gpfsug-discuss] Hidden fileset

Quick question: is there a way to make a fileset hidden so it doesn?t appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

I want to migrate a load of files in but I don?t want the customer to access them prematurely.

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/4ae420b2/attachment-0001.htm>

From r.sobey at imperial.ac.uk  Tue Dec  5 11:14:03 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 5 Dec 2017 11:14:03 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A3A3@EX13.adf.bham.ac.uk>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A3A3@EX13.adf.bham.ac.uk>
Message-ID: <DB6PR0601MB2101298220A37AA14C2DA71EDF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Holy.. why did I not know about that option ?

Will look into that.

And Robert ? yes linking it somewhere else is my current option but it just makes TSM back it all up again when it gets linked to the correct path. Next on the agenda is how to exclude certain directories from the backup.

Thanks
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support)
Sent: 05 December 2017 11:05
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Hidden fileset

Don't give them permissions on the fileset and use the smb option:

"hideunreadable=yes"

https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htmhttps://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htm

This enables access based enumeration.

Simon
________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sobey, Richard A [r.sobey at imperial.ac.uk]
Sent: 05 December 2017 10:38
To: 'gpfsug-discuss at spectrumscale.org'
Subject: ?spam? [gpfsug-discuss] Hidden fileset
Quick question: is there a way to make a fileset hidden so it doesn?t appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

I want to migrate a load of files in but I don?t want the customer to access them prematurely.

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/135ccbd2/attachment-0001.htm>

From andreas.mattsson at maxiv.lu.se  Tue Dec  5 13:15:38 2017
From: andreas.mattsson at maxiv.lu.se (Andreas Mattsson)
Date: Tue, 5 Dec 2017 13:15:38 +0000
Subject: [gpfsug-discuss] Infiniband connection rejected,
	ibv_create_qp err 13
Message-ID: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>

Hi.

Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message "ibv_create_qp err 13"?
I'm having issues with this on a IBM ESS system.

The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to.
The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs.
MPI-jobs on the client nodes can communicate over the infiniband fabric without issues.
Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue.
When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted interfaces.

I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this.

Any help or suggestions is appreciated.
Regards,
Andreas Mattsson
_____________________________________________
[cid:part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer

MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/0e987953/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5610 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/0e987953/attachment-0001.png>

From aaron.s.knister at nasa.gov  Tue Dec  5 13:23:43 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Tue, 5 Dec 2017 13:23:43 +0000
Subject: [gpfsug-discuss] Infiniband connection rejected,
 ibv_create_qp err 13
In-Reply-To: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>
References: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>
Message-ID: <957FBC22-DDAF-4BEF-95DA-1DB7B9AB2CCA@nasa.gov>



Looks like 13 is EPERM which means apparently permissions didn?t exist to create the QP of the desired type which is odd since mmfsd runs as root. Is there any remote chance SELinux is enabled (e.g. sestatus)? Although I?d think mmfsd would run unconfined in the default policy, but maybe it didn?t transition correctly.

On December 5, 2017 at 08:16:49 EST, Andreas Mattsson <andreas.mattsson at maxiv.lu.se> wrote:

Hi.



Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message ?ibv_create_qp err 13??

I?m having issues with this on a IBM ESS system.



The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to.

The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs.

MPI-jobs on the client nodes can communicate over the infiniband fabric without issues.

Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue.

When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted interfaces.



I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this.



Any help or suggestions is appreciated.

Regards,

Andreas Mattsson

_____________________________________________

[mid:d8d07f7e01ec4fcca5ae124f40c2d457 at maxiv.lu.se/part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer



MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/1d5fad85/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5610 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/1d5fad85/attachment-0001.png>

From andreas.mattsson at maxiv.lu.se  Tue Dec  5 14:22:11 2017
From: andreas.mattsson at maxiv.lu.se (Andreas Mattsson)
Date: Tue, 5 Dec 2017 14:22:11 +0000
Subject: [gpfsug-discuss] Infiniband connection rejected,
 ibv_create_qp err 13
In-Reply-To: <957FBC22-DDAF-4BEF-95DA-1DB7B9AB2CCA@nasa.gov>
References: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>
	<957FBC22-DDAF-4BEF-95DA-1DB7B9AB2CCA@nasa.gov>
Message-ID: <869901bdbfd347969d21d619204a9b87@maxiv.lu.se>

Thanks!
I?ve double checked the selinux state, and it is disabled on both all the ESS nodes and on all the client nodes.
mmfsd is running as root on all nodes as well.
It seem a bit strange that this would be an issue of permissions though. I forgot to state this in my original question, but the issue comes and goes, and can affect some clients while not affecting others at the same time, and which clients are affected at any given time changes as well.
Just a thought; How does infiniband queue pairs react to time skew between nodes?

For future reference, where did you find the specification of ibv_create_qp error 13?
I must have been looking in all the wrong places, because I?ve been unable to find the meaning of this error.

Regards,
Andreas
_____________________________________________
[cid:part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer

MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>

Fr?n: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] F?r Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
Skickat: den 5 december 2017 14:24
Till: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
?mne: Re: [gpfsug-discuss] Infiniband connection rejected, ibv_create_qp err 13



Looks like 13 is EPERM which means apparently permissions didn?t exist to create the QP of the desired type which is odd since mmfsd runs as root. Is there any remote chance SELinux is enabled (e.g. sestatus)? Although I?d think mmfsd would run unconfined in the default policy, but maybe it didn?t transition correctly.
On December 5, 2017 at 08:16:49 EST, Andreas Mattsson <andreas.mattsson at maxiv.lu.se<mailto:andreas.mattsson at maxiv.lu.se>> wrote:

Hi.



Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message ?ibv_create_qp err 13??

I?m having issues with this on a IBM ESS system.



The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to.

The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs.

MPI-jobs on the client nodes can communicate over the infiniband fabric without issues.

Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue.

When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted interfaces.



I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this.



Any help or suggestions is appreciated.

Regards,

Andreas Mattsson

_____________________________________________

[mid:d8d07f7e01ec4fcca5ae124f40c2d457 at maxiv.lu.se/part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer



MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/9909e177/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5610 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/9909e177/attachment-0001.png>

From r.sobey at imperial.ac.uk  Tue Dec  5 14:40:33 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 5 Dec 2017 14:40:33 +0000
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <DB6PR0601MB210125AFF8783FFFAB2BF3E4DF3E0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>,
	<OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>
	<DB6PR0601MB210125AFF8783FFFAB2BF3E4DF3E0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <DB6PR0601MB210100429C099C1A2BFBD016DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Thanks all, managed to rerun the policy with the additional list options (kb_allocated etc) and verified that indeed there are at least 60000 files with the same name that are 0 bytes, and more besides.

Now my paranoia is setting in that somehow these files are all corrupted ?

Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 02 December 2017 18:53
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Policy rule to list files based on access time

Thank you all for your replies. I will take a look at them on Monday. And reply individually where I need to..
Richard
Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Marc A Kaplan <makaplan at us.ibm.com>
Sent: Saturday, December 2, 2017 6:03:25 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Policy rule to list files based on access time

You might want to look at FILE_SIZE.  KB_ALLOCATED will be 0 if the file data fits into the inode.

You might also want to use SIZE(FILE_SIZE) in the policy LIST rule, this will cause the KB_HIT and KB_CHOSEN numbers to be the sum of FILE_SIZEs instead of the default SIZE(KB_ALLOCATED).

--marc of GPFS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/9593f4f0/attachment-0001.htm>

From Mark.Bush at siriuscom.com  Tue Dec  5 22:43:35 2017
From: Mark.Bush at siriuscom.com (Mark Bush)
Date: Tue, 5 Dec 2017 22:43:35 +0000
Subject: [gpfsug-discuss] CES and local workstations
Message-ID: <DM5PR05MB35774E1C84500C5EBAA9A5E0E43D0@DM5PR05MB3577.namprd05.prod.outlook.com>

I have a windows 10 machine that is part of my local domain.  I have a separate SpecScale test cluster that has local (not part of my AD domain) ldap and CES (NFS/SMB) running.  I cannot get my local workstation to connect to a SMB share at all.  When I get the logon prompt I'm using IBM NAS\Administrator and the password for that user but it never lets me in.  Is this just defaulting to regular linux SAMBA authentication mechanisms at this point?  Of is the SAMBA code in the gpfs packages unique somehow in this?  I've confirmed that my mmuserauth service appears correct and that ldap is working properly. Any clues?

Frustrated.



Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/72015aba/attachment-0001.htm>

From daniel.kidger at uk.ibm.com  Tue Dec  5 22:44:31 2017
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Tue, 5 Dec 2017 22:44:31 +0000
Subject: [gpfsug-discuss] mmauth/mmremotecluster wonkyness?
In-Reply-To: <OF7BE5A1DA.D530374F-ON002581E8.005B1ECF@LocalDomain>
Message-ID: <OFE0EA91FB.30381E41-ON002581ED.007CED24-1512513871966@notes.na.collabserv.com>

If you think about it, if a CES node opens a file for an NFS client and then a client in a remote MC mounted cluster opens the same file then that MC client node needs to talk to that CES node to negotiate the lock.

Daniel


 

 
 	
Dr Daniel Kidger 
IBM Technical Sales Specialist
Software Defined Solution Sales

+ 44-(0)7818 522 266 
daniel.kidger at uk.ibm.com

> On 30 Nov 2017, at 17:35, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:
> 
> It?s my understanding and experience that all member nodes of two clusters that are multi-clustered must be able to (and will eventually given enough time/activity) make connections to any and all nodes in both clusters. Even if you don?t designate the 2 protocol nodes as contact nodes I would expect to see connections from remote clusters to the protocol nodes just because of the nature of the beast. If you don?t want remote nodes to make connections to the protocol nodes then I believe you would need to put the protocol nodes in their own cluster. CES/CNFS hasn?t always supported this but I think it is now supported, at least with NFS. 
> 
> 
> 
> 
> 
>> On November 30, 2017 at 11:28:03 EST, valdis.kletnieks at vt.edu wrote:
>> We have a 10-node cluster running gpfs 4.2.2.3, where 8 nodes are GPFS contact 
>> nodes for 2 filesystems, and 2 are protocol nodes doingNFS exports of the 
>> filesystems. 
>> 
>> But we see some nodes in remote clusters trying to GPFS connect to 
>> the 2 protocol nodes anyhow. 
>> 
>> My reading of the manpages is that the remote cluster is responsible 
>> for setting '-n contactNodes' when they do the 'mmremotecluster add', 
>> and there's no way to sanity check or enforce that at the local end, and 
>> fail/flag connections to unintended non-contact nodes if the remote 
>> admin forgets/botches the -n. 
>> 
>> Is that actually correct? If so, is it time for an RFE? 
>> _______________________________________________ 
>> gpfsug-discuss mailing list 
>> gpfsug-discuss at spectrumscale.org 
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/da71f602/attachment-0001.htm>

From z.han at imperial.ac.uk  Wed Dec  6 14:04:00 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Wed, 6 Dec 2017 14:04:00 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712040941230.2395@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
Message-ID: <alpine.LFD.2.20.1712061358300.10433@laptop>

I'm pleased to report that kernel 4.4.103 has been
successfully compiled and works with gpfs 4.2.3-4.2,
after a few failed attempts for other 3/4 kernels!

The only minor issue is the iDRAC virtual console
stops working, but fortunately the serial connection
is fine. Any expert advice will be warmly received.

Kind regards,

Zong-Pei


From z.han at imperial.ac.uk  Wed Dec  6 14:05:46 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Wed, 6 Dec 2017 14:05:46 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712061358300.10433@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
	<alpine.LFD.2.20.1712061358300.10433@laptop>
Message-ID: <alpine.LFD.2.20.1712061404430.10433@laptop>

PS

On CentOS Linux release 7.4.1708 (Core)


On Wed, 6 Dec 2017, z.han at imperial.ac.uk wrote:
> Date: Wed, 6 Dec 2017 14:04:00 +0000
> From: z.han at imperial.ac.uk
> Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
>     above?
> 
> I'm pleased to report that kernel 4.4.103 has been
> successfully compiled and works with gpfs 4.2.3-4.2,
> after a few failed attempts for other 3/4 kernels!
>
> The only minor issue is the iDRAC virtual console
> stops working, but fortunately the serial connection
> is fine. Any expert advice will be warmly received.
>
> Kind regards,
>
> Zong-Pei
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>


From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Dec  6 21:56:33 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 6 Dec 2017 21:56:33 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
Message-ID: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>

Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/886bd1a4/attachment-0001.htm>

From jdkwedar at gmail.com  Wed Dec  6 22:10:11 2017
From: jdkwedar at gmail.com (Joshua Kwedar (froz1))
Date: Wed, 6 Dec 2017 17:10:11 -0500
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
Message-ID: <2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)

chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]

Josh K

> On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu> wrote:
> 
> Hi All,
> 
> So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(
> 
> Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.
> 
> Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)
> 
> Thanks?
> 
> ?
> Kevin Buterbaugh - Senior System Administrator
> Vanderbilt University - Advanced Computing Center for Research and Education
> Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/c23c3d52/attachment-0001.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Dec  6 22:13:46 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 6 Dec 2017 22:13:46 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
Message-ID: <26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID  {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]


Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:

Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/28c88bcc/attachment-0001.htm>

From jdkwedar at gmail.com  Wed Dec  6 22:32:18 2017
From: jdkwedar at gmail.com (Joshua Kwedar)
Date: Wed, 06 Dec 2017 17:32:18 -0500
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
Message-ID: <D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>

Hmm.. odd.? 

 

Here?s what the lsuser output should look like.? 

 

# /usr/lpp/mmfs/gui/cli/lsuser

Name? Long name Password status Group names???????????????? Failed login attempts

admin?????????? active????????? Administrator,SecurityAdmin 0

EFSSG1000I The command completed successfully.

 

Can you try something like?

 

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin

 

 

 

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

 

All, 

 

Sorry - should?ve mentioned that:

 

/root

root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231

EFSSG0001C Cannot validate option: login

/root

root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y

lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:

/root

root at testnsd1# 

 

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

 

Kevin



On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com> wrote:

 

The GUI password can be changed via command line using chuser.

 

/usr/lpp/mmfs/gui/cli/chuser



Usage is as follows (where userID = admin)



chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]


Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu> wrote:

Hi All, 

 

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

 

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

 

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

 

Thanks?

 

?

Kevin Buterbaugh - Senior System Administrator

Vanderbilt University - Advanced Computing Center for Research and Education

Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633

 

 

 

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

 

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/aec01adf/attachment-0001.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Dec  6 22:41:12 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 6 Dec 2017 22:41:12 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
Message-ID: <D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin

On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin


On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser


Usage is as follows (where userID = admin)



chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/531a9a8d/attachment-0001.htm>

From r.sobey at imperial.ac.uk  Thu Dec  7 11:57:07 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 7 Dec 2017 11:57:07 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>,
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
Message-ID: <DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin

On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin


On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser


Usage is as follows (where userID = admin)



chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171207/c3bdf293/attachment-0001.htm>

From r.sobey at imperial.ac.uk  Thu Dec  7 12:10:52 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 7 Dec 2017 12:10:52 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>,
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
	<DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Sorry I need to learn to read... didn't see the "object 'Administrator' does not exist" error.

That said, my workaround for the problem of forgetting the password was to create a new "admin2" user and use that to reset the password on admin itself.

[root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin
EFSSG0019I The user admin2 has been successfully created.
EFSSG1000I The command completed successfully.


Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 07 December 2017 11:57
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.
Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I'll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it's subsequently been updated to GPFS 4.2.3.5 ... not sure that's relevant but wanted to mention it just in case.

Thanks!

Kevin


On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here's what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like...

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should've mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it's like the login doesn't exist ... but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I've forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM's, re-installing, and starting over from scratch?  If that's what I have to do, it's no big deal as this is just our little 6-node test cluster, but I thought I'd ask before going down that route.

Oh, and if someone has a way to accomplish this that they'd rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won't tell if you won't tell (and hopefully Michael Flynn won't tell either!).... ;-)

Thanks...

-
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171207/9e93a421/attachment-0001.htm>

From bipcuds at gmail.com  Thu Dec  7 22:48:49 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Thu, 7 Dec 2017 17:48:49 -0500
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <CAAxuGpHj+4z6cLff69Z25CqfUaTWFjoV9vNGWhHSEUqWgWOZ9w@mail.gmail.com>

Hi All,

In working with per-fileset quotas (not user/group/fileset quotas at the
filesystem level), I have the follwing issues/questions.

1.) Setting grace periods. I notice that some of the group quotas in a
specific fileset have a grace period (or remaining grace period) of X days,
while others report "none":

# mmrepquota -g --block-size G hpc-fs:fileset01
                         Block Limits
|                     File Limits
Name       fileset    type             GB      quota      limit
in_doubt    grace |    files   quota    limit in_doubt    grace
groupa      fileset01 GRP            2257          2          2
0   4 days |      143     100      100        0   4 days
root       fileset01 GRP             710          0          0
0     none |    15578       0        0        0     none
groupb   fileset01 GRP            2106        400        400          0   4
days |        1 1048576  1048576        0     none
...

How can I set a grace period of "none" on group quotas? mmsetquota does not
appear (from the man pages) to provide any way to set grace periods for
per-fileset quotas:

mmsetquota Device ??grace {user | group | fileset}
           {[??block GracePeriod] [??files GracePeriod]}

How can I set to "none" or "0days"? (i.e. no grace period given if over
quota). Or, for that matter, set grace periods for any duration at all?


2.) How to remove any explicit quota settings for (not just deactivating
default quota settings) at the per-fileset level. The mmdefquotaoff docs
seem to suggest that the ?-d? option will not remove explicit per-fileset
quota settings if they are non-zero (so really, what use is the -d option
then?)

Many Thanks!
  Keith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171207/b790fd92/attachment-0001.htm>

From Robert.Oesterlin at nuance.com  Fri Dec  8 12:39:58 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Fri, 8 Dec 2017 12:39:58 +0000
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <D7F39BE9-B69E-4AB4-8E6E-7B8FA022EE9D@nuance.com>

1) One way you can set the grace period to none by setting the soft and hard quotas to the same value.

2) By removing any quota setting, do you mean unlimited quota? Which is just setting it (quota) to zero. (meaning unlimited)

Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Keith Ball <bipcuds at gmail.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Thursday, December 7, 2017 at 4:49 PM
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] [gpfsug-discuss] Working with per-fileset quotas

How can I set a grace period of "none" on group quotas? mmsetquota does not appear (from the man pages) to provide any way to set grace periods for per-fileset quotas:

mmsetquota Device ??grace {user | group | fileset}
           {[??block GracePeriod] [??files GracePeriod]}
How can I set to "none" or "0days"? (i.e. no grace period given if over quota). Or, for that matter, set grace periods for any duration at all?


2.) How to remove any explicit quota settings for (not just deactivating default quota settings) at the per-fileset level. The mmdefquotaoff docs seem to suggest that the ?-d? option will not remove explicit per-fileset quota settings if they are non-zero (so really, what use is the -d option then?)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/c006d850/attachment-0001.htm>

From truongv at us.ibm.com  Fri Dec  8 15:41:04 2017
From: truongv at us.ibm.com (Truong Vu)
Date: Fri, 8 Dec 2017 10:41:04 -0500
Subject: [gpfsug-discuss] Working with per-fileset quotas
In-Reply-To: <OF2C4BFBB9.F213F6C2-ON002581F0.0041F153@LocalDomain>
References: <OF2C4BFBB9.F213F6C2-ON002581F0.0041F153@LocalDomain>
Message-ID: <OF95F138D7.BB03BC26-ON852581F0.00560D00-852581F0.00562863@notes.na.collabserv.com>


1) That is correct.  The grace period can't be set for per-fileset
quota.  As you pointed out, you can only change the grace period for
user, group or fileset.

If you want a particular fileset to have no grace period, you can
set the hard limit to be the same as the soft limit.

When the grace column shows "none", this means the soft limit has
not been reached.  Once the soft limit is reached, the grace period
is start counting.

2) To remove explicit quota settings, you need to set the limit to 0.




From:	gpfsug-discuss-request at spectrumscale.org
To:	gpfsug-discuss at spectrumscale.org
Date:	12/08/2017 07:00 AM
Subject:	gpfsug-discuss Digest, Vol 71, Issue 19
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Send gpfsug-discuss mailing list submissions to
		 gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit

https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=5OySIaqfU0j1miWKKp6aydLjiGbE8z5pDz5JGveRRlQ&s=HKXwjIgPAsNTzNSL3-FrvHAXVyvZdzGYugbZgJ3FvMI&e=

or, via email, send a message with subject or body 'help' to
		 gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
		 gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Working with per-fileset quotas (Keith Ball)


----------------------------------------------------------------------

Message: 1
Date: Thu, 7 Dec 2017 17:48:49 -0500
From: Keith Ball <bipcuds at gmail.com>
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID:
		 <CAAxuGpHj
+4z6cLff69Z25CqfUaTWFjoV9vNGWhHSEUqWgWOZ9w at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi All,

In working with per-fileset quotas (not user/group/fileset quotas at the
filesystem level), I have the follwing issues/questions.

1.) Setting grace periods. I notice that some of the group quotas in a
specific fileset have a grace period (or remaining grace period) of X days,
while others report "none":

# mmrepquota -g --block-size G hpc-fs:fileset01
                         Block Limits
|                     File Limits
Name       fileset    type             GB      quota      limit
in_doubt    grace |    files   quota    limit in_doubt    grace
groupa      fileset01 GRP            2257          2          2
0   4 days |      143     100      100        0   4 days
root       fileset01 GRP             710          0          0
0     none |    15578       0        0        0     none
groupb   fileset01 GRP            2106        400        400          0   4
days |        1 1048576  1048576        0     none
...

How can I set a grace period of "none" on group quotas? mmsetquota does not
appear (from the man pages) to provide any way to set grace periods for
per-fileset quotas:

mmsetquota Device ??grace {user | group | fileset}
           {[??block GracePeriod] [??files GracePeriod]}

How can I set to "none" or "0days"? (i.e. no grace period given if over
quota). Or, for that matter, set grace periods for any duration at all?


2.) How to remove any explicit quota settings for (not just deactivating
default quota settings) at the per-fileset level. The mmdefquotaoff docs
seem to suggest that the ?-d? option will not remove explicit per-fileset
quota settings if they are non-zero (so really, what use is the -d option
then?)

Many Thanks!
  Keith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20171207_b790fd92_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=5OySIaqfU0j1miWKKp6aydLjiGbE8z5pDz5JGveRRlQ&s=-DNcYGwFlUrOQZsZ9yQAYBdVv46u1xxeA4wJ2-VxQ_A&e=
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=5OySIaqfU0j1miWKKp6aydLjiGbE8z5pDz5JGveRRlQ&s=HKXwjIgPAsNTzNSL3-FrvHAXVyvZdzGYugbZgJ3FvMI&e=



End of gpfsug-discuss Digest, Vol 71, Issue 19
**********************************************



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/790f55b2/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/790f55b2/attachment-0001.gif>

From eric.wonderley at vt.edu  Fri Dec  8 16:10:16 2017
From: eric.wonderley at vt.edu (J. Eric Wonderley)
Date: Fri, 8 Dec 2017 11:10:16 -0500
Subject: [gpfsug-discuss] mm'add|del'node with ccr enabled
Message-ID: <CABOSGQf2JTQdp3E+8dkFXY07x5u9SnePh0yhAgyD6YMty2WhHA@mail.gmail.com>

Hello:

If I recall correctly this does not work...correct?  I think the last time
I attempted this was gpfs version <=4.1.  I think I attempted to add a
quorum node.

The process was that I remember doing was mmshutdown -a, mmchcluster
--ccr-disable, mmaddnode yadayada, mmchcluster --ccr-enable, mmstartup.

I think with ccr disabled mmaddnode can be run with gpfs up.  We would like
to run with ccr enabled but it does make adding/removing nodes unpleasant.

Would this be required of a non-quorum node?

Any changes concerning this with gpfs version >=4.2?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/850462fc/attachment-0001.htm>

From Robert.Oesterlin at nuance.com  Fri Dec  8 16:17:13 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Fri, 8 Dec 2017 16:17:13 +0000
Subject: [gpfsug-discuss] mm'add|del'node with ccr enabled
Message-ID: <A1F99FA0-3794-4989-A1A2-D81A038765B5@nuance.com>

Yes, there were issues with this in pre 4.2? versions of GPFS. My clusters are at 4.2.2 with CCR (working my way to 4.2.3-5) and I?ve been using this process on my quorum nodes when I need to update them:

mmchnode ?quorum -N new-node (designate a new quorum node, going from 3 to 4)
mmchnode ?nonquorum -N old-node (remove existing node to be updated)

I can then shutdown ?old-node? and then when I bring it back up, I reverse the process. I?ve had no issues using this process.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "J. Eric Wonderley" <eric.wonderley at vt.edu>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Friday, December 8, 2017 at 10:11 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] [gpfsug-discuss] mm'add|del'node with ccr enabled

Hello:

If I recall correctly this does not work...correct?  I think the last time I attempted this was gpfs version <=4.1.  I think I attempted to add a quorum node.

The process was that I remember doing was mmshutdown -a, mmchcluster --ccr-disable, mmaddnode yadayada, mmchcluster --ccr-enable, mmstartup.

I think with ccr disabled mmaddnode can be run with gpfs up.  We would like to run with ccr enabled but it does make adding/removing nodes unpleasant.

Would this be required of a non-quorum node?

Any changes concerning this with gpfs version >=4.2?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/da944b42/attachment-0001.htm>

From TROPPENS at de.ibm.com  Sat Dec  9 00:11:27 2017
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Sat, 9 Dec 2017 01:11:27 +0100
Subject: [gpfsug-discuss] Genomics Blueprint - SSUG @ CIUK next week
 [Tuesday 12th December 2017]
In-Reply-To: <OF0DB03636.FFEC975E-ON002581ED.003C507F@LocalDomain>
References: <OF0DB03636.FFEC975E-ON002581ED.003C507F@LocalDomain>
Message-ID: <OF3EFB1F2F.FBD960FA-ONC12581F1.00006B84-C12581F1.00010C76@notes.na.collabserv.com>

We just published the first blueprint:

IBM Spectrum Scale Best Practices for Genomics Medicine Workloads
http://www.redbooks.ibm.com/abstracts/redp5479.html

This blueprint is optimized for genomics workloads but most recommendations
apply to other workloads.

Looking forward to receive first-hand feedback at CIUK.


--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294




From:	Simon Thompson <chair at spectrumscale.org>
To:	"gpfsug-discuss at spectrumscale.org"
            <gpfsug-discuss at spectrumscale.org>
Date:	05.12.2017 11:58
Subject:	[gpfsug-discuss] SSUG @ CIUK next week [Tuesday 12th December
            2017]
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hi All,

Just a reminder that the last #SSUG of 2017 is happening next week as part
of Computing Insights UK (CIUK) in Manchester. The agenda for the session
is:

2pm Introductions and welcome, Simon Thompson & Claire O?Toole (SSUG
Committee)
2:10pm Spectrum Scale 5.0, Daniel Kidger (IBM)
2:40pm Deep Dive: Blueprints, Ulf Troppens (IBM)
3:40pm User Talk: Institute of Cancer Research, Robert Horton (ICR)
4:00pm User Talk: Red Bull Technology, John Brookes (Red Bull Technology)
4:20pm Questions and discussion
4:50pm Close

To attend, you must be registered as a delegate of CIUK:
https://eventbooking.stfc.ac.uk/news-events/ciuk-2017

Simon_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/9d16277d/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/9d16277d/attachment-0001.gif>

From Renar.Grunenberg at huk-coburg.de  Sat Dec  9 10:58:52 2017
From: Renar.Grunenberg at huk-coburg.de (Grunenberg, Renar)
Date: Sat, 9 Dec 2017 10:58:52 +0000
Subject: [gpfsug-discuss] mm'add|del'node with ccr enabled
In-Reply-To: <CABOSGQf2JTQdp3E+8dkFXY07x5u9SnePh0yhAgyD6YMty2WhHA@mail.gmail.com>
References: <CABOSGQf2JTQdp3E+8dkFXY07x5u9SnePh0yhAgyD6YMty2WhHA@mail.gmail.com>
Message-ID: <f926f5c352f146ceb49f5e14fb1e76fd@SMXRF105.msg.hukrf.de>

Hallo Eric,
our experiences are add and delete new/old nodes is working only if this node is no quorum node in an ccr cluster, no problem. There are no mmshutdown steps necessary. We are on 4.2.3.6. I think this is already available since >4.2. If you want to add a new quorum node, than you must put this node first as a client and after that you can change this node to a quorum node.


Renar Grunenberg
Abteilung Informatik ? Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon:        09561 96-44110
Telefax:        09561 96-44104
E-Mail: Renar.Grunenberg at huk-coburg.de
Internet:       www.huk.de
________________________________
HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Her?y, Dr. J?rg Rheinl?nder (stv.), Sarah R?ssler, Daniel Thomas (stv.).
________________________________
Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.

This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this information in error) please notify the
sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.
________________________________
Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von J. Eric Wonderley
Gesendet: Freitag, 8. Dezember 2017 17:10
An: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Betreff: [gpfsug-discuss] mm'add|del'node with ccr enabled

Hello:

If I recall correctly this does not work...correct?  I think the last time I attempted this was gpfs version <=4.1.  I think I attempted to add a quorum node.

The process was that I remember doing was mmshutdown -a, mmchcluster --ccr-disable, mmaddnode yadayada, mmchcluster --ccr-enable, mmstartup.

I think with ccr disabled mmaddnode can be run with gpfs up.  We would like to run with ccr enabled but it does make adding/removing nodes unpleasant.

Would this be required of a non-quorum node?

Any changes concerning this with gpfs version >=4.2?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/16c21559/attachment-0001.htm>

From bipcuds at gmail.com  Sat Dec  9 21:49:54 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Sat, 9 Dec 2017 16:49:54 -0500
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <CAAxuGpHH6e2K=mBTZQ=tA7wc-UMYmg7_8F9hvrh8Kn7Gh78_rA@mail.gmail.com>

I meant more specifically, remove any indication that quotas have been set,
for anyone (or everyone). That way, I could have a script to clean out old
quota definitions, then set both default and explicit quotas, without any
old settings lurking around. Is there a way to get rid of (e.g. zero out)
all existing explicit PER-FILESET quota definitions, and start over
(without having to look at mmrepquota and zero them out one by one)?


So I see your point about soft=lard limit => no grace period. I guess
what's odd is that I see any indication that the grace period is not
"none"; what grace period would be assigned to a per-fileset quota if I
cannot even view or set it?

Thanks,
   Keith


From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Working with per-fileset quotas
>
> 2) By removing any quota setting, do you mean unlimited quota? Which is
> just setting it (quota) to zero. (meaning unlimited)
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 8 Dec 2017 10:41:04 -0500
> From: "Truong Vu" <truongv at us.ibm.com>
> To: gpfsug-discuss at spectrumscale.org
> Subject: Re: [gpfsug-discuss] Working with per-fileset quotas
> Message-ID:
>         <OF95F138D7.BB03BC26-ON852581F0.00560D00-852581F0.00562863@
> notes.na.collabserv.com>
>
> Content-Type: text/plain; charset="us-ascii"
>
>
> 1) That is correct.  The grace period can't be set for per-fileset
> quota.  As you pointed out, you can only change the grace period for
> user, group or fileset.
>
> If you want a particular fileset to have no grace period, you can
> set the hard limit to be the same as the soft limit.
>
> When the grace column shows "none", this means the soft limit has
> not been reached.  Once the soft limit is reached, the grace period
> is start counting.
>
> 2) To remove explicit quota settings, you need to set the limit to 0.
>
>
>
>
> From:   gpfsug-discuss-request at spectrumscale.org
> To:     gpfsug-discuss at spectrumscale.org
> Date:   12/08/2017 07:00 AM
> Subject:        gpfsug-discuss Digest, Vol 71, Issue 19
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
>
>
>    1. Working with per-fileset quotas (Keith Ball)
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 7 Dec 2017 17:48:49 -0500
> From: Keith Ball <bipcuds at gmail.com>
> To: gpfsug-discuss at spectrumscale.org
> Subject: [gpfsug-discuss] Working with per-fileset quotas
> Message-ID:
>                  <CAAxuGpHj
> +4z6cLff69Z25CqfUaTWFjoV9vNGWhHSEUqWgWOZ9w at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi All,
>
> In working with per-fileset quotas (not user/group/fileset quotas at the
> filesystem level), I have the follwing issues/questions.
>
> 1.) Setting grace periods. I notice that some of the group quotas in a
> specific fileset have a grace period (or remaining grace period) of X days,
> while others report "none":
>
> # mmrepquota -g --block-size G hpc-fs:fileset01
>                          Block Limits
> |                     File Limits
> Name       fileset    type             GB      quota      limit
> in_doubt    grace |    files   quota    limit in_doubt    grace
> groupa      fileset01 GRP            2257          2          2
> 0   4 days |      143     100      100        0   4 days
> root       fileset01 GRP             710          0          0
> 0     none |    15578       0        0        0     none
> groupb   fileset01 GRP            2106        400        400          0   4
> days |        1 1048576  1048576        0     none
> ...
>
> How can I set a grace period of "none" on group quotas? mmsetquota does not
> appear (from the man pages) to provide any way to set grace periods for
> per-fileset quotas:
>
> mmsetquota Device ??grace {user | group | fileset}
>            {[??block GracePeriod] [??files GracePeriod]}
>
> How can I set to "none" or "0days"? (i.e. no grace period given if over
> quota). Or, for that matter, set grace periods for any duration at all?
>
>
> 2.) How to remove any explicit quota settings for (not just deactivating
> default quota settings) at the per-fileset level. The mmdefquotaoff docs
> seem to suggest that the ?-d? option will not remove explicit per-fileset
> quota settings if they are non-zero (so really, what use is the -d option
> then?)
>
> Many Thanks!
>   Keith
>
>

-- 
Keith D. Ball, PhD
RedLine Performance Solutions, LLC
web:  http://www.redlineperf.com/
email: kball at redlineperf.com <aqualkenbush at redlineperf.com>
cell: 540-557-7851 <(540)%20557-7851>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/203f7a33/attachment-0001.htm>

From Robert.Oesterlin at nuance.com  Sun Dec 10 01:27:13 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Sun, 10 Dec 2017 01:27:13 +0000
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <C1EFE45D-20E7-4689-87A6-7C5B2CE2ED5F@nuance.com>

Looking at the default case in a new file system/fileset, if you did an ?mmrepquota? you?d see everyone have a default quota of zero. Meaning ? any time you set a user/fileset/group quota back to zero, you are removing any trace of the previous quota. On you your specific question - yes, one by one is it, I?m not aware of any other way to do it.

Hard quota set, no soft quota: no grace period
Hard Soft quota set and equal: no grace period
Hard quota set or unset, soft quota set: grace period

The default behavior is no grace period unless soft quota is set, maybe that?s why it?s no displayed?


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Keith Ball <bipcuds at gmail.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Saturday, December 9, 2017 at 3:50 PM
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Working with per-fileset quotas

I meant more specifically, remove any indication that quotas have been set, for anyone (or everyone). That way, I could have a script to clean out old quota definitions, then set both default and explicit quotas, without any old settings lurking around. Is there a way to get rid of (e.g. zero out) all existing explicit PER-FILESET quota definitions, and start over (without having to look at mmrepquota and zero them out one by one)?


So I see your point about soft=lard limit => no grace period. I guess what's odd is that I see any indication that the grace period is not "none"; what grace period would be assigned to a per-fileset quota if I cannot even view or set it?


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171210/5b19346b/attachment-0001.htm>

From john.hearns at asml.com  Wed Dec 13 12:18:01 2017
From: john.hearns at asml.com (John Hearns)
Date: Wed, 13 Dec 2017 12:18:01 +0000
Subject: [gpfsug-discuss] Am I a gateway or not?
Message-ID: <HE1PR02MB14506B5498911ABC8CE08D4F88350@HE1PR02MB1450.eurprd02.prod.outlook.com>

Firstly, I hope everyone at the Manchester meeting is enjoying it and that the snow did not hamper too many travel plans.
I like Manchester - I shall tell the tale of blowing the 100Amp fuse in the University server room some other time.
(Yes - a fuse. Not a circuit breaker).

I am writing some monitoring scripts for AFM - actually to interface with Bright.
I would like the script to detect if the node which is running it is a Gateway node or has the AFM component.
If not, the script silently exits.
This is not a Callback script - I can write these and indeed have one for the AFM fileset demount event.

In order to determine which components are on a given node I could:

  1.  Run "mmhealth node show" and grep for AFM
  2.  Run mmlscluster and grep for $HOSTNAME

Both methods seem slightly inelegant. IS there a handy-dandy command I can run to see the components on this node?

John Hearns
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/22341ece/attachment-0001.htm>

From scale at us.ibm.com  Wed Dec 13 14:45:52 2017
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Wed, 13 Dec 2017 09:45:52 -0500
Subject: [gpfsug-discuss] Am I a gateway or not?
In-Reply-To: <OF9C6D99B0.DF71D62B-ON002581F5.0043B3C0@LocalDomain>
References: <OF9C6D99B0.DF71D62B-ON002581F5.0043B3C0@LocalDomain>
Message-ID: <OF1EF7D0AE.0DD977EE-ON852581F5.00510567-852581F5.00511AAB@notes.na.collabserv.com>

Hi Abhishek,

Can you please help answer this query.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   John Hearns <john.hearns at asml.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   12/13/2017 07:19 AM
Subject:        [gpfsug-discuss] Am I a gateway or not?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Firstly, I hope everyone at the Manchester meeting is enjoying it and that 
the snow did not hamper too many travel plans.
I like Manchester ? I shall tell the tale of blowing the 100Amp fuse in 
the University server room some other time.
(Yes ? a fuse. Not a circuit breaker).
 
I am writing some monitoring scripts for AFM ? actually to interface with 
Bright.
I would like the script to detect if the node which is running it is a 
Gateway node or has the AFM component.
If not, the script silently exits.
This is not a Callback script ? I can write these and indeed have one for 
the AFM fileset demount event.
 
In order to determine which components are on a given node I could:
a.      Run ?mmhealth node show? and grep for AFM
b.      Run mmlscluster and grep for $HOSTNAME
 
Both methods seem slightly inelegant. IS there a handy-dandy command I can 
run to see the components on this node?
 
John Hearns
-- The information contained in this communication and any attachments is 
confidential and may be privileged, and is for the sole use of the 
intended recipient(s). Any unauthorized review, use, disclosure or 
distribution is prohibited. Unless explicitly stated otherwise in the body 
of this communication or the attachment thereto (if any), the information 
is provided on an AS-IS basis without any express or implied warranties or 
liabilities. To the extent you are relying on this information, you are 
doing so at your own risk. If you are not the intended recipient, please 
notify the sender immediately by replying to this message and destroy all 
copies of this message and any attachments. Neither the sender nor the 
company/group of companies he or she represents shall be liable for the 
proper and complete transmission of the information contained in this 
communication, or for any delay in its receipt. 
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=iZJyYX6s6-fBYfRNYStIrfVhph3NLzwXJNqTZEfKOyA&s=HoPYEIPpBLIFV-JCq69MS026ZLBRxqqvqNZYGH8UmlY&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/92f3f864/attachment-0001.htm>

From abhisdav at in.ibm.com  Wed Dec 13 16:58:39 2017
From: abhisdav at in.ibm.com (Abhishek Dave)
Date: Wed, 13 Dec 2017 22:28:39 +0530
Subject: [gpfsug-discuss] Am I a gateway or not?
In-Reply-To: <OF1EF7D0AE.0DD977EE-ON852581F5.00510567-852581F5.00511998@LocalDomain>
References: <OF9C6D99B0.DF71D62B-ON002581F5.0043B3C0@LocalDomain>
	<OF1EF7D0AE.0DD977EE-ON852581F5.00510567-852581F5.00511998@LocalDomain>
Message-ID: <OF90FA1155.FA40F211-ON652581F5.005C72A9-652581F5.005D42EF@notes.na.collabserv.com>

Hi John,

I would recommend running 'mmlscluster' command on any cluster node to
check if any AFM Gateway Node(s) is available.

This will list hostname(s) which are running as AFM Gateway.

/usr/lpp/mmfs/bin/mmlscluster | grep -w gateway | awk '{print $4}'

e.g.

~bash# /usr/lpp/mmfs/bin/mmlscluster | grep -w gateway | awk '{print $4}'
node1
node2

Thanks,
Abhishek, Dave




From:	IBM Spectrum Scale/Poughkeepsie/IBM
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc:	Abhishek Dave/India/IBM at IBMIN, IBM Spectrum
            Scale/Poughkeepsie/IBM at IBMUS
Date:	12/13/2017 08:16 PM
Subject:	Re: [gpfsug-discuss] Am I a gateway or not?
Sent by:	Haizhu Liu


Hi Abhishek,

Can you please help answer this query.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------

If you feel that your question can benefit other users of  Spectrum Scale
(GPFS), then please post it to the public IBM developerWroks Forum at
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
.

If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact
1-800-237-5511 in the United States or your local IBM Service Center in
other countries.

The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.




From:	John Hearns <john.hearns at asml.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	12/13/2017 07:19 AM
Subject:	[gpfsug-discuss] Am I a gateway or not?
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Firstly, I hope everyone at the Manchester meeting is enjoying it and that
the snow did not hamper too many travel plans.
I like Manchester ? I shall tell the tale of blowing the 100Amp fuse in the
University server room some other time.
(Yes ? a fuse. Not a circuit breaker).

I am writing some monitoring scripts for AFM ? actually to interface with
Bright.
I would like the script to detect if the node which is running it is a
Gateway node or has the AFM component.
If not, the script silently exits.
This is not a Callback script ? I can write these and indeed have one for
the AFM fileset demount event.

In order to determine which components are on a given node I could:
   a.	Run ?mmhealth node show? and grep for AFM
   b.	Run mmlscluster and grep for $HOSTNAME

Both methods seem slightly inelegant. IS there a handy-dandy command I can
run to see the components on this node?

John Hearns


-- The information contained in this communication and any attachments is
confidential and may be privileged, and is for the sole use of the intended
recipient(s). Any unauthorized review, use, disclosure or distribution is
prohibited. Unless explicitly stated otherwise in the body of this
communication or the attachment thereto (if any), the information is
provided on an AS-IS basis without any express or implied warranties or
liabilities. To the extent you are relying on this information, you are
doing so at your own risk. If you are not the intended recipient, please
notify the sender immediately by replying to this message and destroy all
copies of this message and any attachments. Neither the sender nor the
company/group of companies he or she represents shall be liable for the
proper and complete transmission of the information contained in this
communication, or for any delay in its receipt.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=iZJyYX6s6-fBYfRNYStIrfVhph3NLzwXJNqTZEfKOyA&s=HoPYEIPpBLIFV-JCq69MS026ZLBRxqqvqNZYGH8UmlY&e=










-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/3b793bb7/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/3b793bb7/attachment-0001.gif>

From chair at spectrumscale.org  Thu Dec 14 11:16:05 2017
From: chair at spectrumscale.org (Simon Thompson)
Date: Thu, 14 Dec 2017 11:16:05 +0000
Subject: [gpfsug-discuss] SSUG17 wraps
Message-ID: <F96F7F48-5592-45D9-9A1D-16B9D8F8D507@spectrumscale.org>

Well, our last event of 2017 took place this year at CIUK in Manchester on 12th December. Claire pointed out that it was exactly 7 years since the user group was first formed and we?ve grown significantly since then. There?s now hundreds of subscribers to the mailing list and regular events taking place across the globe.

 

For those interested in the presentations this year, all the slide decks we have or that have been approved are published on the user group website.

 

Thanks to all the speakers from both IBM and the user community for speaking at our events!

 

And finally, we?ve updated the look of our website and changed to something we hope will be a bit easier for use to manage. We?ve made some changes to the navigation to make things easier to find, so please do check out the website and let myself of Claire know directly if you have any comments on this.

 

We?re also interested in hearing from you if you?d like to write a feature or article for the user group website, perhaps comment on a scale feature you use, or something you?ve struggled with and fixed. Please get in touch if you?d like to do this.

 

We?re already planning for 2018 events and hope we?ll have a successful year next year.

 

Simon Thompson

UK Group Chair

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171214/53dbbc98/attachment-0001.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Thu Dec 14 19:38:52 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Thu, 14 Dec 2017 19:38:52 +0000
Subject: [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5 upgrade
Message-ID: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>

Hi All,

         26 mmbackupDors-20171023.log
         26 mmbackupDors-20171024.log
         26 mmbackupDors-20171025.log
         26 mmbackupDors-20171026.log
    2922752 mmbackupDors-20171027.log
    1375555 mmbackupDors-20171028.log
      59328 mmbackupDors-20171029.log
    2748095 mmbackupDors-20171030.log
     124953 mmbackupDors-20171031.log

That?s ?wc -l? output ? and the difference in size occurred with the GPFS 4.2.3.5 upgrade.  I?m not technically ?responsible? for mmbackup here, so I?m not at all familiar with it.  However, we?ve asked a certain vendor (not IBM) about it and they don?t know either, so I don?t feel too awfully bad.

And we have looked at the man page and didn?t see any obvious options to decrease the verbosity.  We did not make any changes to the backup script itself, so the mmbackup invocation is the same.  Any ideas?  Thanks?

Kevin
?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171214/23f59fce/attachment-0001.htm>

From scale at us.ibm.com  Fri Dec 15 14:32:25 2017
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Fri, 15 Dec 2017 09:32:25 -0500
Subject: [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5
	upgrade
In-Reply-To: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>
References: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>
Message-ID: <OFEF3AE524.5EDB2C57-ON852581F7.004FCC00-852581F7.004FDF6C@notes.na.collabserv.com>

Tru,

Can you please help with this query or forward to the right person. 
Thanks. 

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   12/14/2017 02:39 PM
Subject:        [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5 
upgrade
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi All, 

         26 mmbackupDors-20171023.log
         26 mmbackupDors-20171024.log
         26 mmbackupDors-20171025.log
         26 mmbackupDors-20171026.log
    2922752 mmbackupDors-20171027.log
    1375555 mmbackupDors-20171028.log
      59328 mmbackupDors-20171029.log
    2748095 mmbackupDors-20171030.log
     124953 mmbackupDors-20171031.log

That?s ?wc -l? output ? and the difference in size occurred with the GPFS 
4.2.3.5 upgrade.  I?m not technically ?responsible? for mmbackup here, so 
I?m not at all familiar with it.  However, we?ve asked a certain vendor 
(not IBM) about it and they don?t know either, so I don?t feel too awfully 
bad.

And we have looked at the man page and didn?t see any obvious options to 
decrease the verbosity.  We did not make any changes to the backup script 
itself, so the mmbackup invocation is the same.  Any ideas?  Thanks?

Kevin
?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and 
Education
Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=ZFlCTb2hD9WKda9YJ94wM4Kb-ZTkTg6yuH2H0DabbRs&s=4TRCVgkMd1Ly9WvaBPVWr4ciur_ZYZswzqrlFHl_6EI&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/9212a7f7/attachment-0001.htm>

From makaplan at us.ibm.com  Fri Dec 15 15:14:40 2017
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Fri, 15 Dec 2017 10:14:40 -0500
Subject: [gpfsug-discuss] mmbackup log file size after GPFS
	4.2.3.5upgrade
In-Reply-To: <OFEF3AE524.5EDB2C57-ON852581F7.004FCC00-852581F7.004FDF6C@notes.na.collabserv.com>
References: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>
	<OFEF3AE524.5EDB2C57-ON852581F7.004FCC00-852581F7.004FDF6C@notes.na.collabserv.com>
Message-ID: <OF94C2F295.6B1FFCC4-ON852581F7.005351E8-852581F7.0053BE2A@notes.na.collabserv.com>

It's not clear that this is a problem or malfunction.
Customer should contact IBM support and be ready to transmit copies of the 
cited log files and other mmbackup command output (stdout and stderr 
messages) for analysis.
Also mmsnap output.



From:   "IBM Spectrum Scale" <scale at us.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc:     Truong Vu <truongv at us.ibm.com>
Date:   12/15/2017 09:32 AM
Subject:        Re: [gpfsug-discuss] mmbackup log file size after GPFS 
4.2.3.5 upgrade
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Tru,

Can you please help with this query or forward to the right person. 
Thanks.  

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        12/14/2017 02:39 PM
Subject:        [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5 
upgrade
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi All, 

         26 mmbackupDors-20171023.log
         26 mmbackupDors-20171024.log
         26 mmbackupDors-20171025.log
         26 mmbackupDors-20171026.log
    2922752 mmbackupDors-20171027.log
    1375555 mmbackupDors-20171028.log
      59328 mmbackupDors-20171029.log
    2748095 mmbackupDors-20171030.log
     124953 mmbackupDors-20171031.log

That?s ?wc -l? output ? and the difference in size occurred with the GPFS 
4.2.3.5 upgrade.  I?m not technically ?responsible? for mmbackup here, so 
I?m not at all familiar with it.  However, we?ve asked a certain vendor 
(not IBM) about it and they don?t know either, so I don?t feel too awfully 
bad.

And we have looked at the man page and didn?t see any obvious options to 
decrease the verbosity.  We did not make any changes to the backup script 
itself, so the mmbackup invocation is the same.  Any ideas?  Thanks?

Kevin
?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and 
Education
Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=ZFlCTb2hD9WKda9YJ94wM4Kb-ZTkTg6yuH2H0DabbRs&s=4TRCVgkMd1Ly9WvaBPVWr4ciur_ZYZswzqrlFHl_6EI&e=



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=n3w55C-SUpJhWZEaM249Jx7hx44Q8KsZi0-S2WaisJc&s=SFkl9xPj2c7TR7paHtU3Nby1vonHbBBt1Vmv7QdLH9Q&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/f4679d08/attachment-0001.htm>

From Robert.Oesterlin at nuance.com  Fri Dec 15 18:40:28 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Fri, 15 Dec 2017 18:40:28 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0.0 Available on Passport
	Advantage
Message-ID: <985A9D83-4B5E-4234-9884-FFA5AE295063@nuance.com>

Scale 5.0 was released today and is available for download. Time to construct a test cluster!


Bob Oesterlin
Sr Principal Storage Engineer, Nuance
507-269-0413

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/b7a70d41/attachment-0001.htm>

From aaron.s.knister at nasa.gov  Fri Dec 15 19:49:59 2017
From: aaron.s.knister at nasa.gov (Aaron Knister)
Date: Fri, 15 Dec 2017 14:49:59 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
Message-ID: <fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>

Thanks, Bill.

I still don't feel like I've got an clear answer from IBM and frankly
the core issue of a lack of migration tool was totally dodged.

Again in Sven's presentation from SSUG @ SC17
(http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf)
he mentions "It has a significant performance penalty for small files in
large block size filesystems" and the demonstrates that with several
mdtest runs (which show the effect with and without the >32 subblocks code):


4.2.1 base code - SUMMARY: (of 3 iterations)
File creation : Mean = 2237.644

zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY: (of
3 iterations)
File creation : Mean = 12866.842

more sub blocks per block (4.2.2 + morethan32subblock code):
File creation : Mean = 40316.721

Can someone (ideally Sven) give me a straight answer as to whether or
not the > 32 subblock code actually makes a performance difference for
small files in large block filesystems? And if not, help me understand
why his slides and provided benchmark data have consistently indicated
it does?

-Aaron

On 12/1/17 11:44 AM, Bill Hartner wrote:
> ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> results @ user group meeting.
> 
> -Bill
> 
> Bill Hartner
> IBM Systems
> Scalable I/O Development
> Austin, Texas
> bhartner at us.ibm.com
> home office 512-784-0980
> 
> 
> Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
> AM---Bill, could you say something about what the metadataJan-Frode
> Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> about what the metadata-storage here was? ESS/NL-SAS/3way replication?
> 
> From: Jan-Frode Myklebust <janfrode at tanso.net>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 12/01/2017 06:53 AM
> Subject: Re: [gpfsug-discuss] Online data migration tool
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> 
> ------------------------------------------------------------------------
> 
> 
> 
> Bill, could you say something about what the metadata-storage here was?
> ESS/NL-SAS/3way replication?
> 
> I just asked about this in the internal slack channel #scale-help today..
> 
> 
> 
> -jf
> 
> fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
> <mailto:bhartner at us.ibm.com>>:
> 
>     > "It has a significant performance penalty for small files in large
>     > block size filesystems"
> 
>     Aaron,
> 
>     Below are mdtest results for a test we ran for CORAL - file size was
>     32k.
> 
>     We have not gone back and ran the test on a file system formatted
>     without > 32 subblocks. We'll do that at some point...
> 
>     -Bill
> 
>     -- started at 10/28/2017 17:51:38 --
> 
>     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360
>     -u -y
>     Path: /ibm/fs2-16m-10
>     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> 
>     228 tasks, 67239936 files
> 
>     SUMMARY: (of 3 iterations)
>     Operation Max Min Mean Std Dev
>     --------- --- --- ---- -------
>     File creation : 51953.498 50558.517 51423.221 616.643
>     File stat : 0.000 0.000 0.000 0.000
>     File read : 0.000 0.000 0.000 0.000
>     File removal : 96746.376 92149.535 94658.774 1900.187
>     Tree creation : 1.588 0.070 0.599 0.700
>     Tree removal : 0.213 0.034 0.097 0.082
> 
>     -- finished at 10/28/2017 19:51:54 --
> 
>     Bill Hartner
>     IBM Systems
>     Scalable I/O Development
>     Austin, Texas_
>     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com>
>     home office 512-784-0980
> 
>     _
>     __gpfsug-discuss-bounces at spectrumscale.org_
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>?wrote on
>     11/29/2017 04:41:48 PM:
> 
>     > From: Aaron Knister <_aaron.knister at gmail.com_
>     <mailto:aaron.knister at gmail.com>>
> 
> 
>     > To: gpfsug main discussion list
>     <_gpfsug-discuss at spectrumscale.org_
>     <mailto:gpfsug-discuss at spectrumscale.org>>
> 
>     > Date: 11/29/2017 04:42 PM
> 
> 
>     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> 
>     >
> 
>     > Thanks, Nikhil. Most of that was consistent with my understnading,
>     > however I was under the impression that the >32 subblocks code is
>     > required to achieve the touted 50k file creates/second that Sven has
>     > talked about a bunch of times:
>     >
>     >
>     _http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>
>     >
>     _http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_ <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>
>     > _Sven_Oehme_-_News_from_Research.pdf
>     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>
>     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> 
> 
>     > from those presentations regarding 32 subblocks:
>     >
>     > "It has a significant performance penalty for small files in large
>     > block size filesystems"
> 
>     > although I'm not clear on the specific definition of "large". Many
>     > filesystems I encounter only have a 1M block size so it may not
>     > matter there, although that same presentation clearly shows the
>     > benefit of larger block sizes which is yet *another* thing for which
>     > a migration tool would be helpful.
> 
>     > -Aaron
>     >
>     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com>> wrote:
> 
>     > Hi,
>     >
>     > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
>     > For all Spectrum Scale clusters that are currently at 4.X.X, it is
>     > possible to migrate to 5.0.0 with no offline data migration and no
>     > need to move data. Once these clusters are at 5.0.0, they will
>     > benefit from the performance improvements, new features (such as
>     > file audit logging), and various enhancements that are included in
>     5.0.0.
>     >
>     > That being said, there is one enhancement that will not be applied
>     > to these clusters, and that is the increased number of sub-blocks
>     > per block for small file allocation. This means that for file
>     > systems with a large block size and a lot of small files, the
>     > overall space utilization will be the same it currently is in 4.X.X.
>     > Since file systems created at 4.X.X and earlier used a block size
>     > that kept this allocation in mind, there should be very little
>     > impact on existing file systems.
>     >
>     > Outside of that one particular function, the remainder of the
>     > performance improvements, metadata improvements, updated
>     > compatibility, new functionality, and all of the other enhancements
>     > will be immediately available to you once you complete the upgrade
>     > to 5.0.0 -- with no need to reformat, move data, or take your data
>     offline.
>     >
>     > I hope that clarifies things a little and makes the upgrade path
>     > more accessible.
>     >
>     > Please let me know if there are any other questions or concerns.
>     >
>     > Thank you,
>     > Nikhil Khandelwal
>     > Spectrum Scale Development
>     > Client Adoption
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at _spectrumscale.org_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
>     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
> 
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at _spectrumscale.org_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
> 
>     > _https://urldefense.proofpoint.com/v2/url?_
>     >
>     u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>     >
>     siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> 
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at _spectrumscale.org_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>_
>     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>_______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org
>     https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> 
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776


From alex at calicolabs.com  Fri Dec 15 20:48:16 2017
From: alex at calicolabs.com (Alex Chekholko)
Date: Fri, 15 Dec 2017 12:48:16 -0800
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
Message-ID: <CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>

Hey Aaron,

Can you define your sizes for "large blocks" and "small files"?  If you
dial one up and the other down, your performance will be worse.  And in any
case it's a pathological corner case so it shouldn't matter much for your
workflow, unless you've designed your system with the wrong values.

For example, for bioinformatics workloads, I prefer to use 256KB filesystem
block size, and I'd consider 4MB+ to be "large block size", which would
make the filesystem obviously unsuitable for processing millions of 8KB
files.

You can make a histogram of file sizes in your existing filesystems and
then make your subblock size (1/32 of block size) on the smaller end of
that.   Also definitely use the "small file in inode" feature and put your
metadata on SSD.

Regards,
Alex

On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister <aaron.s.knister at nasa.gov>
wrote:

> Thanks, Bill.
>
> I still don't feel like I've got an clear answer from IBM and frankly
> the core issue of a lack of migration tool was totally dodged.
>
> Again in Sven's presentation from SSUG @ SC17
> (http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf)
> he mentions "It has a significant performance penalty for small files in
> large block size filesystems" and the demonstrates that with several
> mdtest runs (which show the effect with and without the >32 subblocks
> code):
>
>
> 4.2.1 base code - SUMMARY: (of 3 iterations)
> File creation : Mean = 2237.644
>
> zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY: (of
> 3 iterations)
> File creation : Mean = 12866.842
>
> more sub blocks per block (4.2.2 + morethan32subblock code):
> File creation : Mean = 40316.721
>
> Can someone (ideally Sven) give me a straight answer as to whether or
> not the > 32 subblock code actually makes a performance difference for
> small files in large block filesystems? And if not, help me understand
> why his slides and provided benchmark data have consistently indicated
> it does?
>
> -Aaron
>
> On 12/1/17 11:44 AM, Bill Hartner wrote:
> > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> > results @ user group meeting.
> >
> > -Bill
> >
> > Bill Hartner
> > IBM Systems
> > Scalable I/O Development
> > Austin, Texas
> > bhartner at us.ibm.com
> > home office 512-784-0980
> >
> >
> > Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
> > AM---Bill, could you say something about what the metadataJan-Frode
> > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> > about what the metadata-storage here was? ESS/NL-SAS/3way replication?
> >
> > From: Jan-Frode Myklebust <janfrode at tanso.net>
> > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> > Date: 12/01/2017 06:53 AM
> > Subject: Re: [gpfsug-discuss] Online data migration tool
> > Sent by: gpfsug-discuss-bounces at spectrumscale.org
> >
> > ------------------------------------------------------------------------
> >
> >
> >
> > Bill, could you say something about what the metadata-storage here was?
> > ESS/NL-SAS/3way replication?
> >
> > I just asked about this in the internal slack channel #scale-help today..
> >
> >
> >
> > -jf
> >
> > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
> > <mailto:bhartner at us.ibm.com>>:
> >
> >     > "It has a significant performance penalty for small files in large
> >     > block size filesystems"
> >
> >     Aaron,
> >
> >     Below are mdtest results for a test we ran for CORAL - file size was
> >     32k.
> >
> >     We have not gone back and ran the test on a file system formatted
> >     without > 32 subblocks. We'll do that at some point...
> >
> >     -Bill
> >
> >     -- started at 10/28/2017 17:51:38 --
> >
> >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
> >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360
> >     -u -y
> >     Path: /ibm/fs2-16m-10
> >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> >
> >     228 tasks, 67239936 files
> >
> >     SUMMARY: (of 3 iterations)
> >     Operation Max Min Mean Std Dev
> >     --------- --- --- ---- -------
> >     File creation : 51953.498 50558.517 51423.221 616.643
> >     File stat : 0.000 0.000 0.000 0.000
> >     File read : 0.000 0.000 0.000 0.000
> >     File removal : 96746.376 92149.535 94658.774 1900.187
> >     Tree creation : 1.588 0.070 0.599 0.700
> >     Tree removal : 0.213 0.034 0.097 0.082
> >
> >     -- finished at 10/28/2017 19:51:54 --
> >
> >     Bill Hartner
> >     IBM Systems
> >     Scalable I/O Development
> >     Austin, Texas_
> >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com>
> >     home office 512-784-0980
> >
> >     _
> >     __gpfsug-discuss-bounces at spectrumscale.org_
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org> wrote on
> >     11/29/2017 04:41:48 PM:
> >
> >     > From: Aaron Knister <_aaron.knister at gmail.com_
> >     <mailto:aaron.knister at gmail.com>>
> >
> >
> >     > To: gpfsug main discussion list
> >     <_gpfsug-discuss at spectrumscale.org_
> >     <mailto:gpfsug-discuss at spectrumscale.org>>
> >
> >     > Date: 11/29/2017 04:42 PM
> >
> >
> >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> >
> >     >
> >
> >     > Thanks, Nikhil. Most of that was consistent with my understnading,
> >     > however I was under the impression that the >32 subblocks code is
> >     > required to achieve the touted 50k file creates/second that Sven
> has
> >     > talked about a bunch of times:
> >     >
> >     >
> >     _http://files.gpfsug.org/presentations/2017/Manchester/
> 08_Research_Topics.pdf_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__files.gpfsug.org_presentations_2017_Manchester_
> 08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>
> >     >
> >     _http://files.gpfsug.org/presentations/2017/Ehningen/
> 31_-_SSUG17DE_-_ <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__files.gpfsug.org_presentations_2017_Ehningen_
> 31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>
> >     > _Sven_Oehme_-_News_from_Research.pdf
> >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__files.gpfsug.org_presentations_2016_SC16_12-5F-
> 2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW
> UGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--
> uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>
> >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> >
> >
> >     > from those presentations regarding 32 subblocks:
> >     >
> >     > "It has a significant performance penalty for small files in large
> >     > block size filesystems"
> >
> >     > although I'm not clear on the specific definition of "large". Many
> >     > filesystems I encounter only have a 1M block size so it may not
> >     > matter there, although that same presentation clearly shows the
> >     > benefit of larger block sizes which is yet *another* thing for
> which
> >     > a migration tool would be helpful.
> >
> >     > -Aaron
> >     >
> >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
> >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com>> wrote:
> >
> >     > Hi,
> >     >
> >     > I would like to clarify migration path to 5.0.0 from 4.X.X
> clusters.
> >     > For all Spectrum Scale clusters that are currently at 4.X.X, it is
> >     > possible to migrate to 5.0.0 with no offline data migration and no
> >     > need to move data. Once these clusters are at 5.0.0, they will
> >     > benefit from the performance improvements, new features (such as
> >     > file audit logging), and various enhancements that are included in
> >     5.0.0.
> >     >
> >     > That being said, there is one enhancement that will not be applied
> >     > to these clusters, and that is the increased number of sub-blocks
> >     > per block for small file allocation. This means that for file
> >     > systems with a large block size and a lot of small files, the
> >     > overall space utilization will be the same it currently is in
> 4.X.X.
> >     > Since file systems created at 4.X.X and earlier used a block size
> >     > that kept this allocation in mind, there should be very little
> >     > impact on existing file systems.
> >     >
> >     > Outside of that one particular function, the remainder of the
> >     > performance improvements, metadata improvements, updated
> >     > compatibility, new functionality, and all of the other enhancements
> >     > will be immediately available to you once you complete the upgrade
> >     > to 5.0.0 -- with no need to reformat, move data, or take your data
> >     offline.
> >     >
> >     > I hope that clarifies things a little and makes the upgrade path
> >     > more accessible.
> >     >
> >     > Please let me know if there are any other questions or concerns.
> >     >
> >     > Thank you,
> >     > Nikhil Khandelwal
> >     > Spectrum Scale Development
> >     > Client Adoption
> >     >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at _spectrumscale.org_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
> >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=
> DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW
> UGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_
> TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
> >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at _spectrumscale.org_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
> >
> >     > _https://urldefense.proofpoint.com/v2/url?_
> >     >
> >     u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=
> DwICAg&c=jf_iaSHvJObTbx-
> >     >
> >     siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=
> DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_
> W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> >
> >     _______________________________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at _spectrumscale.org_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >_
> >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=
> DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW
> UGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_
> TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>___________________
> ____________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at spectrumscale.org
> >     https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/5f33cc4a/attachment-0001.htm>

From aaron.s.knister at nasa.gov  Fri Dec 15 22:47:43 2017
From: aaron.s.knister at nasa.gov (Aaron Knister)
Date: Fri, 15 Dec 2017 17:47:43 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
Message-ID: <03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>

Thanks, Alex. I'm all too familiar with the trade offs between large
blocks and small files and we do use pretty robust SSD storage for our
metadata. We support a wide range of workloads and we have some folks
with many small (<1M) files and other folks with many large (>256MB) files.

My point in this thread is that IBM has said over and over again in
presentations that there is a significant performance gain with the >32
subblocks code on filesystems with large block sizes (although to your
point I'm not clear on exactly what large means since I didn't define
large in this context). Therefore given that the >32 subblock code gives
a significant performance gain one could reasonably assume that having a
filesystem with >32 subblocks is required to see this gain (rather than
just running the >32 subblocks code on an fs w/o > 32 subblocks).

This lead me to ask about a migration tool because in my mind if there's
a performance gain from having >32 subblocks on the FS I'd like that
feature and having to manually copy 10's of PB to new hardware to get
this performance boost is unacceptable. However, IBM can't seem to make
up their mind about whether or not the >32 subblocks code *actually*
provides a performance increase. This seems like a pretty
straightforward question.

-Aaron

On 12/15/17 3:48 PM, Alex Chekholko wrote:
> Hey Aaron,
> 
> Can you define your sizes for "large blocks" and "small files"?? If you
> dial one up and the other down, your performance will be worse.? And in
> any case it's a pathological corner case so it shouldn't matter much for
> your workflow, unless you've designed your system with the wrong values.
> 
> For example, for bioinformatics workloads, I prefer to use 256KB
> filesystem block size, and I'd consider 4MB+ to be "large block size",
> which would make the filesystem obviously unsuitable for processing
> millions of 8KB files.
> 
> You can make a histogram of file sizes in your existing filesystems and
> then make your subblock size (1/32 of block size) on the smaller end of
> that.? ?Also definitely use the "small file in inode" feature and put
> your metadata on SSD.
> 
> Regards,
> Alex
> 
> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
> 
>     Thanks, Bill.
> 
>     I still don't feel like I've got an clear answer from IBM and frankly
>     the core issue of a lack of migration tool was totally dodged.
> 
>     Again in Sven's presentation from SSUG @ SC17
>     (http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
>     he mentions "It has a significant performance penalty for small files in
>     large block size filesystems" and the demonstrates that with several
>     mdtest runs (which show the effect with and without the >32
>     subblocks code):
> 
> 
>     4.2.1 base code - SUMMARY: (of 3 iterations)
>     File creation : Mean = 2237.644
> 
>     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):? SUMMARY: (of
>     3 iterations)
>     File creation : Mean = 12866.842
> 
>     more sub blocks per block (4.2.2 + morethan32subblock code):
>     File creation : Mean = 40316.721
> 
>     Can someone (ideally Sven) give me a straight answer as to whether or
>     not the > 32 subblock code actually makes a performance difference for
>     small files in large block filesystems? And if not, help me understand
>     why his slides and provided benchmark data have consistently indicated
>     it does?
> 
>     -Aaron
> 
>     On 12/1/17 11:44 AM, Bill Hartner wrote:
>     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
>     > results @ user group meeting.
>     >
>     > -Bill
>     >
>     > Bill Hartner
>     > IBM Systems
>     > Scalable I/O Development
>     > Austin, Texas
>     > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
>     > home office 512-784-0980 <tel:512-784-0980>
>     >
>     >
>     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
>     > AM---Bill, could you say something about what the metadataJan-Frode
>     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
>     > about what the metadata-storage here was? ESS/NL-SAS/3way replication?
>     >
>     > From: Jan-Frode Myklebust <janfrode at tanso.net <mailto:janfrode at tanso.net>>
>     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org
>     <mailto:gpfsug-discuss at spectrumscale.org>>
>     > Date: 12/01/2017 06:53 AM
>     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     >
>     >
>     ------------------------------------------------------------------------
>     >
>     >
>     >
>     > Bill, could you say something about what the metadata-storage here was?
>     > ESS/NL-SAS/3way replication?
>     >
>     > I just asked about this in the internal slack channel #scale-help today..
>     >
>     >
>     >
>     > -jf
>     >
>     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
>     > <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>:
>     >
>     >? ? ?> "It has a significant performance penalty for small files in
>     large
>     >? ? ?> block size filesystems"
>     >
>     >? ? ?Aaron,
>     >
>     >? ? ?Below are mdtest results for a test we ran for CORAL - file
>     size was
>     >? ? ?32k.
>     >
>     >? ? ?We have not gone back and ran the test on a file system formatted
>     >? ? ?without > 32 subblocks. We'll do that at some point...
>     >
>     >? ? ?-Bill
>     >
>     >? ? ?-- started at 10/28/2017 17:51:38 --
>     >
>     >? ? ?mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>     >? ? ?Command line used: /tmp/mdtest-binary-dir/mdtest -d
>     >? ? ?/ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
>     -p 360
>     >? ? ?-u -y
>     >? ? ?Path: /ibm/fs2-16m-10
>     >? ? ?FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>     >
>     >? ? ?228 tasks, 67239936 files
>     >
>     >? ? ?SUMMARY: (of 3 iterations)
>     >? ? ?Operation Max Min Mean Std Dev
>     >? ? ?--------- --- --- ---- -------
>     >? ? ?File creation : 51953.498 50558.517 51423.221 616.643
>     >? ? ?File stat : 0.000 0.000 0.000 0.000
>     >? ? ?File read : 0.000 0.000 0.000 0.000
>     >? ? ?File removal : 96746.376 92149.535 94658.774 1900.187
>     >? ? ?Tree creation : 1.588 0.070 0.599 0.700
>     >? ? ?Tree removal : 0.213 0.034 0.097 0.082
>     >
>     >? ? ?-- finished at 10/28/2017 19:51:54 --
>     >
>     >? ? ?Bill Hartner
>     >? ? ?IBM Systems
>     >? ? ?Scalable I/O Development
>     >? ? ?Austin, Texas_
>     >? ? ?__bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
>     <mailto:bhartner at us.ibm.com>>
>     >? ? ?home office 512-784-0980 <tel:512-784-0980>
>     >
>     >? ? ?_
>     >? ? ?__gpfsug-discuss-bounces at spectrumscale.org_
>     >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>?wrote on
>     >? ? ?11/29/2017 04:41:48 PM:
>     >
>     >? ? ?> From: Aaron Knister <_aaron.knister at gmail.com_
>     >? ? ?<mailto:aaron.knister at gmail.com <mailto:aaron.knister at gmail.com>>>
>     >
>     >
>     >? ? ?> To: gpfsug main discussion list
>     >? ? ?<_gpfsug-discuss at spectrumscale.org_
>     >? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>     <mailto:gpfsug-discuss at spectrumscale.org>>>
>     >
>     >? ? ?> Date: 11/29/2017 04:42 PM
>     >
>     >
>     >? ? ?> Subject: Re: [gpfsug-discuss] Online data migration tool
>     >? ? ?> Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>     >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>     >
>     >? ? ?>
>     >
>     >? ? ?> Thanks, Nikhil. Most of that was consistent with my understnading,
>     >? ? ?> however I was under the impression that the >32 subblocks code is
>     >? ? ?> required to achieve the touted 50k file creates/second that Sven has
>     >? ? ?> talked about a bunch of times:
>     >? ? ?>
>     >? ? ?>
>     >? ?
>     ?_http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>     <http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>     >? ? ?>
>     >? ?
>     ?_http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
>     <http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>     >? ? ?> _Sven_Oehme_-_News_from_Research.pdf
>     >? ? ?> _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>     >? ? ?> _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>     >
>     >
>     >? ? ?> from those presentations regarding 32 subblocks:
>     >? ? ?>
>     >? ? ?> "It has a significant performance penalty for small files in large
>     >? ? ?> block size filesystems"
>     >
>     >? ? ?> although I'm not clear on the specific definition of "large". Many
>     >? ? ?> filesystems I encounter only have a 1M block size so it may not
>     >? ? ?> matter there, although that same presentation clearly shows the
>     >? ? ?> benefit of larger block sizes which is yet *another* thing for which
>     >? ? ?> a migration tool would be helpful.
>     >
>     >? ? ?> -Aaron
>     >? ? ?>
>     >? ? ?> On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>     >? ? ?<_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
>     <mailto:nikhilk at us.ibm.com>>> wrote:
>     >
>     >? ? ?> Hi,
>     >? ? ?>
>     >? ? ?> I would like to clarify migration path to 5.0.0 from 4.X.X
>     clusters.
>     >? ? ?> For all Spectrum Scale clusters that are currently at 4.X.X,
>     it is
>     >? ? ?> possible to migrate to 5.0.0 with no offline data migration
>     and no
>     >? ? ?> need to move data. Once these clusters are at 5.0.0, they will
>     >? ? ?> benefit from the performance improvements, new features (such as
>     >? ? ?> file audit logging), and various enhancements that are
>     included in
>     >? ? ?5.0.0.
>     >? ? ?>
>     >? ? ?> That being said, there is one enhancement that will not be
>     applied
>     >? ? ?> to these clusters, and that is the increased number of
>     sub-blocks
>     >? ? ?> per block for small file allocation. This means that for file
>     >? ? ?> systems with a large block size and a lot of small files, the
>     >? ? ?> overall space utilization will be the same it currently is
>     in 4.X.X.
>     >? ? ?> Since file systems created at 4.X.X and earlier used a block
>     size
>     >? ? ?> that kept this allocation in mind, there should be very little
>     >? ? ?> impact on existing file systems.
>     >? ? ?>
>     >? ? ?> Outside of that one particular function, the remainder of the
>     >? ? ?> performance improvements, metadata improvements, updated
>     >? ? ?> compatibility, new functionality, and all of the other
>     enhancements
>     >? ? ?> will be immediately available to you once you complete the
>     upgrade
>     >? ? ?> to 5.0.0 -- with no need to reformat, move data, or take
>     your data
>     >? ? ?offline.
>     >? ? ?>
>     >? ? ?> I hope that clarifies things a little and makes the upgrade path
>     >? ? ?> more accessible.
>     >? ? ?>
>     >? ? ?> Please let me know if there are any other questions or concerns.
>     >? ? ?>
>     >? ? ?> Thank you,
>     >? ? ?> Nikhil Khandelwal
>     >? ? ?> Spectrum Scale Development
>     >? ? ?> Client Adoption
>     >? ? ?>
>     >? ? ?> _______________________________________________
>     >? ? ?> gpfsug-discuss mailing list
>     >? ? ?> gpfsug-discuss at _spectrumscale.org_
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >? ? ?> _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>     >
>     >? ? ?> _______________________________________________
>     >? ? ?> gpfsug-discuss mailing list
>     >? ? ?> gpfsug-discuss at _spectrumscale.org_
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >
>     >? ? ?> _https://urldefense.proofpoint.com/v2/url?_
>     <https://urldefense.proofpoint.com/v2/url?_>
>     >? ? ?>
>     >? ? ?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>     >? ? ?>
>     >? ? ?siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>     >
>     >? ? ?_______________________________________________
>     >? ? ?gpfsug-discuss mailing list
>     >? ? ?gpfsug-discuss at _spectrumscale.org_
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>     >? ? ?__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>     >? ? ?gpfsug-discuss mailing list
>     >? ? ?gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     >? ?
>     ?https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>     >
> 
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776 <tel:%28301%29%20286-2776>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> 
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776


From oehmes at gmail.com  Fri Dec 15 23:06:52 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Fri, 15 Dec 2017 23:06:52 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
Message-ID: <CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>

i thought i answered that already, but maybe i just thought about answering
it and then forgot about it :-D

so yes more than 32 subblocks per block significant increase the
performance of filesystems with small files, for the sake of the argument
let's say 32k in a large block filesystem again for sake of argument say
16MB.

you probably ask why ?

if you create a file and write 32k into it in a pre 5.0.0 Version 16 MB
filesystem your client actually doesn't write 32k to the NSD Server, it
writes 512k, because thats the subblock size and we need to write the full
subblock (for security reasons). so first you waste significant memory on
the client to cache that zero padding, you waste network bandwidth and you
waste NSD Server cache because you store it there too. this means you
overrun the cache more quickly, means you start doing read/modify writes
earlier on all your nice large raid tracks... i guess you get the story by
now.

in fact,  if you have a good raid code that can drive really a lot of
bandwidth out of individual drives like a GNR system you get more
performance for small file writes as larger your blocksize is, because we
can 'pack' more files into larger i/os and therefore turn a small file
create workload into a bandwidth workload, essentially exactly what we did
and i demonstrated in the CORAL presentation .

hope that makes this crystal clear now .

sven



On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister <aaron.s.knister at nasa.gov>
wrote:

> Thanks, Alex. I'm all too familiar with the trade offs between large
> blocks and small files and we do use pretty robust SSD storage for our
> metadata. We support a wide range of workloads and we have some folks
> with many small (<1M) files and other folks with many large (>256MB) files.
>
> My point in this thread is that IBM has said over and over again in
> presentations that there is a significant performance gain with the >32
> subblocks code on filesystems with large block sizes (although to your
> point I'm not clear on exactly what large means since I didn't define
> large in this context). Therefore given that the >32 subblock code gives
> a significant performance gain one could reasonably assume that having a
> filesystem with >32 subblocks is required to see this gain (rather than
> just running the >32 subblocks code on an fs w/o > 32 subblocks).
>
> This lead me to ask about a migration tool because in my mind if there's
> a performance gain from having >32 subblocks on the FS I'd like that
> feature and having to manually copy 10's of PB to new hardware to get
> this performance boost is unacceptable. However, IBM can't seem to make
> up their mind about whether or not the >32 subblocks code *actually*
> provides a performance increase. This seems like a pretty
> straightforward question.
>
> -Aaron
>
> On 12/15/17 3:48 PM, Alex Chekholko wrote:
> > Hey Aaron,
> >
> > Can you define your sizes for "large blocks" and "small files"?  If you
> > dial one up and the other down, your performance will be worse.  And in
> > any case it's a pathological corner case so it shouldn't matter much for
> > your workflow, unless you've designed your system with the wrong values.
> >
> > For example, for bioinformatics workloads, I prefer to use 256KB
> > filesystem block size, and I'd consider 4MB+ to be "large block size",
> > which would make the filesystem obviously unsuitable for processing
> > millions of 8KB files.
> >
> > You can make a histogram of file sizes in your existing filesystems and
> > then make your subblock size (1/32 of block size) on the smaller end of
> > that.   Also definitely use the "small file in inode" feature and put
> > your metadata on SSD.
> >
> > Regards,
> > Alex
> >
> > On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> > <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
> >
> >     Thanks, Bill.
> >
> >     I still don't feel like I've got an clear answer from IBM and frankly
> >     the core issue of a lack of migration tool was totally dodged.
> >
> >     Again in Sven's presentation from SSUG @ SC17
> >     (
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
> >     he mentions "It has a significant performance penalty for small
> files in
> >     large block size filesystems" and the demonstrates that with several
> >     mdtest runs (which show the effect with and without the >32
> >     subblocks code):
> >
> >
> >     4.2.1 base code - SUMMARY: (of 3 iterations)
> >     File creation : Mean = 2237.644
> >
> >     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY:
> (of
> >     3 iterations)
> >     File creation : Mean = 12866.842
> >
> >     more sub blocks per block (4.2.2 + morethan32subblock code):
> >     File creation : Mean = 40316.721
> >
> >     Can someone (ideally Sven) give me a straight answer as to whether or
> >     not the > 32 subblock code actually makes a performance difference
> for
> >     small files in large block filesystems? And if not, help me
> understand
> >     why his slides and provided benchmark data have consistently
> indicated
> >     it does?
> >
> >     -Aaron
> >
> >     On 12/1/17 11:44 AM, Bill Hartner wrote:
> >     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> >     > results @ user group meeting.
> >     >
> >     > -Bill
> >     >
> >     > Bill Hartner
> >     > IBM Systems
> >     > Scalable I/O Development
> >     > Austin, Texas
> >     > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
> >     > home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >
> >     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017
> 06:53:44
> >     > AM---Bill, could you say something about what the metadataJan-Frode
> >     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> >     > about what the metadata-storage here was? ESS/NL-SAS/3way
> replication?
> >     >
> >     > From: Jan-Frode Myklebust <janfrode at tanso.net <mailto:
> janfrode at tanso.net>>
> >     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>
> >     > Date: 12/01/2017 06:53 AM
> >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     > Sent by: gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> >     >
> >     >
> >
>  ------------------------------------------------------------------------
> >     >
> >     >
> >     >
> >     > Bill, could you say something about what the metadata-storage here
> was?
> >     > ESS/NL-SAS/3way replication?
> >     >
> >     > I just asked about this in the internal slack channel #scale-help
> today..
> >     >
> >     >
> >     >
> >     > -jf
> >     >
> >     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
> <_bhartner at us.ibm.com_
> >     > <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>:
> >     >
> >     >     > "It has a significant performance penalty for small files in
> >     large
> >     >     > block size filesystems"
> >     >
> >     >     Aaron,
> >     >
> >     >     Below are mdtest results for a test we ran for CORAL - file
> >     size was
> >     >     32k.
> >     >
> >     >     We have not gone back and ran the test on a file system
> formatted
> >     >     without > 32 subblocks. We'll do that at some point...
> >     >
> >     >     -Bill
> >     >
> >     >     -- started at 10/28/2017 17:51:38 --
> >     >
> >     >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> >     >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
> >     >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
> >     -p 360
> >     >     -u -y
> >     >     Path: /ibm/fs2-16m-10
> >     >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> >     >
> >     >     228 tasks, 67239936 files
> >     >
> >     >     SUMMARY: (of 3 iterations)
> >     >     Operation Max Min Mean Std Dev
> >     >     --------- --- --- ---- -------
> >     >     File creation : 51953.498 50558.517 51423.221 616.643
> >     >     File stat : 0.000 0.000 0.000 0.000
> >     >     File read : 0.000 0.000 0.000 0.000
> >     >     File removal : 96746.376 92149.535 94658.774 1900.187
> >     >     Tree creation : 1.588 0.070 0.599 0.700
> >     >     Tree removal : 0.213 0.034 0.097 0.082
> >     >
> >     >     -- finished at 10/28/2017 19:51:54 --
> >     >
> >     >     Bill Hartner
> >     >     IBM Systems
> >     >     Scalable I/O Development
> >     >     Austin, Texas_
> >     >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
> >     <mailto:bhartner at us.ibm.com>>
> >     >     home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >     _
> >     >     __gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>> wrote on
> >     >     11/29/2017 04:41:48 PM:
> >     >
> >     >     > From: Aaron Knister <_aaron.knister at gmail.com_
> >     >     <mailto:aaron.knister at gmail.com <mailto:
> aaron.knister at gmail.com>>>
> >     >
> >     >
> >     >     > To: gpfsug main discussion list
> >     >     <_gpfsug-discuss at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>>
> >     >
> >     >     > Date: 11/29/2017 04:42 PM
> >     >
> >     >
> >     >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
> >     >
> >     >     >
> >     >
> >     >     > Thanks, Nikhil. Most of that was consistent with my
> understnading,
> >     >     > however I was under the impression that the >32 subblocks
> code is
> >     >     > required to achieve the touted 50k file creates/second that
> Sven has
> >     >     > talked about a bunch of times:
> >     >     >
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >     <
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >>
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
> >     <
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >>
> >     >     > _Sven_Oehme_-_News_from_Research.pdf
> >     >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
> >     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >>
> >     >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> >     >
> >     >
> >     >     > from those presentations regarding 32 subblocks:
> >     >     >
> >     >     > "It has a significant performance penalty for small files in
> large
> >     >     > block size filesystems"
> >     >
> >     >     > although I'm not clear on the specific definition of
> "large". Many
> >     >     > filesystems I encounter only have a 1M block size so it may
> not
> >     >     > matter there, although that same presentation clearly shows
> the
> >     >     > benefit of larger block sizes which is yet *another* thing
> for which
> >     >     > a migration tool would be helpful.
> >     >
> >     >     > -Aaron
> >     >     >
> >     >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
> >     >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
> >     <mailto:nikhilk at us.ibm.com>>> wrote:
> >     >
> >     >     > Hi,
> >     >     >
> >     >     > I would like to clarify migration path to 5.0.0 from 4.X.X
> >     clusters.
> >     >     > For all Spectrum Scale clusters that are currently at 4.X.X,
> >     it is
> >     >     > possible to migrate to 5.0.0 with no offline data migration
> >     and no
> >     >     > need to move data. Once these clusters are at 5.0.0, they
> will
> >     >     > benefit from the performance improvements, new features
> (such as
> >     >     > file audit logging), and various enhancements that are
> >     included in
> >     >     5.0.0.
> >     >     >
> >     >     > That being said, there is one enhancement that will not be
> >     applied
> >     >     > to these clusters, and that is the increased number of
> >     sub-blocks
> >     >     > per block for small file allocation. This means that for file
> >     >     > systems with a large block size and a lot of small files, the
> >     >     > overall space utilization will be the same it currently is
> >     in 4.X.X.
> >     >     > Since file systems created at 4.X.X and earlier used a block
> >     size
> >     >     > that kept this allocation in mind, there should be very
> little
> >     >     > impact on existing file systems.
> >     >     >
> >     >     > Outside of that one particular function, the remainder of the
> >     >     > performance improvements, metadata improvements, updated
> >     >     > compatibility, new functionality, and all of the other
> >     enhancements
> >     >     > will be immediately available to you once you complete the
> >     upgrade
> >     >     > to 5.0.0 -- with no need to reformat, move data, or take
> >     your data
> >     >     offline.
> >     >     >
> >     >     > I hope that clarifies things a little and makes the upgrade
> path
> >     >     > more accessible.
> >     >     >
> >     >     > Please let me know if there are any other questions or
> concerns.
> >     >     >
> >     >     > Thank you,
> >     >     > Nikhil Khandelwal
> >     >     > Spectrum Scale Development
> >     >     > Client Adoption
> >     >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>
> >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >
> >     >     > _https://urldefense.proofpoint.com/v2/url?_
> >     <https://urldefense.proofpoint.com/v2/url?_>
> >     >     >
> >     >
>  u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
> >     >     >
> >     >
>  siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> >     >
> >     >     _______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>_
> >     >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>_______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >
> >     >
> >     >
> >     >
> >     >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >     >
> >
> >     --
> >     Aaron Knister
> >     NASA Center for Climate Simulation (Code 606.2)
> >     Goddard Space Flight Center
> >     (301) 286-2776 <tel:%28301%29%20286-2776>
> >     _______________________________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/1d465799/attachment-0001.htm>

From Robert.Oesterlin at nuance.com  Mon Dec 18 19:43:35 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Mon, 18 Dec 2017 19:43:35 +0000
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=> replied to the IBM Spectrum Scale V5.0 announcements<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=> topic thread in the General Parallel File System - Announce (GPFS - Announce)<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=> forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
as updated, April 2017

Changes to this release of the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library include the following:

Added DMPs for TIP events

       A topic is added listing the directed maintenance procedures for TIP events.
       The DMPs help users resolve issues caused due to TIP events.

AFM and AFM DR

       - Compression and snapshot ILM policy supported.
       - A general recommendation added for the Gateway node.
       - Configuration parameters added - afmMaxParallelRecoveries, afmAsyncOpWaitTimeout,
           afmSyncOpWaitTimeout, and afmRevalOpWaitTimeout.
       - Configuration parameters modified - afmRPO and afmHashVersion.

Authentication: Authentication packages

       Updated the authentication page to include packages specific to Ubuntu.

Authentication: AD-based authentication

       New information is added on NFS with server-side group
       lookup and Active Directory authentication.

Authentication: Primary group selection configurable for AD + RFC2307 based authentication

       Ability to choose primary group as set in "UNIX attributes" of a user on
       Active Directory introduced with AD + RFC2307 based authentication scheme.
       Earlier, Windows primary group was by default selected as the primary group.

Big data and analytics
       - The GPFS Ambari integration package is now called the IBM Spectrum
         Scale Ambari management pack (in short, management pack or MPack).
       - IBM Spectrum Scale Ambari management pack version 2.4.2.1 with
         HDFS Transparency version 2.7.3.1 supports BI 4.2/BI 4.2.5 IOP migration
         to HDP 2.6.2.
       - Supports the remote mount configuration in Ambari.
       - Supports the multiple file systems configuration. In management
         pack version 2.4.2.1, the current limit is two file systems.
       - The Short circuit write is supported for better performance.
       - In HDFS Transparency, the Ranger performance is enhanced.

Changes to IBM Spectrum Scale management API

   Added the following new commands:

       GET /perfmon/data
       GET /filesystems/{filesystemName}/afm/state
       DELETE /nodes/{name}
       POST /nodes
       GET /nodeclasses
       POST /nodeclasses
       DELETE /nodeclasses/{nodeclassName}
       GET /nodeclasses/{nodeclassName}
       PUT /nodeclasses/{nodeclassName}
       DELETE /jobs/jobId
       POST /filesystems/{filesystemName}/filesets/{filesetName}/psnaps
       DELETE /filesystems/{filesystemName}/filesets/{filesetName}/psnaps/{snapshotName}
       GET /thresholds
       GET /thresholds/{name}
       POST /thresholds
       DELETE /thresholds/{name}

IBM Spectrum Scale GUI changes

   - Added new Networks page to monitor the performance, configuration, and
      adapters of network configurations in the cluster. You can monitor the
      network performance with respect to the IP and RDMA interfaces used in the configuration.
    - Added new Monitoring > Thresholds page to create and monitor the threshold rules
      that are defined in the system.
    - Added Access > Remote Connections page to enable the GUI node of the
      local cluster to monitor the remote cluster by establishing a connection
      with the GUI node of the remote cluster.
    - Added Settings > Call Home page to configure call home. Configuring
      the call home feature helps the IBM? Support to monitor the system.
      Configuring call home also helps to reduce the response time of the
      IBM Support to resolve any issues.
      The diagnostic data that is downloaded through the Settings > Diagnostic Data
      can be uploaded to a problem management record (PMR) by using
      the call home feature in the backend. To upload the diagnostic data, right-click
      the relevant data set from the Previously Collected Diagnostic Data, and
      select Upload to PMR.
    - Added file system creation capabilities in GUI. Use the Files > File Systems > Create File System
      option to launch the Create File System wizard. In the Create File System wizard, you can specify
      the following details of the file system:
         - File system name
         - Storage pools
         - NSDs for the file systems
         - Failure groups
         - NSD order for data writes
         - Maximum number of Spectrum Scale clients
         - Maximum number of inodes of the root fileset
         - Whether to enable quota and scope for the quota definition
         - Whether to enable DMAPI
         - Mount point and automatic mount mode
    - Added the aggregation levels Access Point and Filesets and removed Account
      for the resource type Transparent Cloud Tiering in the Monitoring > Statisitics page.
    - The Files > Transparent Cloud Tiering page now displays the file systems
      and filesets that are mapped with the cloud service. It also shows the
      connection of such a container pair configuration to a cloud account and
      the corresponding CSAP that is configured for the cloud account.
    - Changes to capacity monitoring in the GUI
         - Moved the Capacity page from Monitoring to Files menu in the navigation
           and renamed the GUI page to User Capacity.
         - Only the file data user capacity can be viewed from the Files > User Capacity page.
           Removed the pools, filesets, file system capacity monitoring options from the Files > User Capacity page.
           You can monitor the capacity of these components from the respective GUI pages.
         - Replaced the GPFSPoolCap sensor with the GPFSPool sensor and separate data and metadata
           level capacity monitoring are introduced in the performance charts available in the

           Files > File Systems and Storage > Pools pages.
         - New GPFSPool-based data and metadata performance monitoring metrics are available
           for selection in the Files > Statistics > Edit > Capacity section.
           You need to select the aggregation level as Pool to view these metrics.
    - AFM monitoring changes in the GUI
         - Provides the number of AFM filesets and the corresponding export server maps.
           Each export map establishes a mapping between the gateway node and the NFS
           host name to allow parallel data transfers from cache to home.
         - By using the Request Access option available in the Files > Active File Management
           or Access > Remote Connection page in the GUI, you can now establish connection with remote clusters.
           After establishing the connection, you can monitor the following AFM and AFM DR
           configuration details across clusters:
              * On home and secondary, you can see the AFM relationships configuration,
                health status, and performance values of the Cache and Disaster Recovery grids.
              * On the Overview tab of the detailed view, the available home and secondary inodes are available.
              * On the Overview tab of the detailed view, the details such as NFS throughput,
                IOPs, and latency details are available, if the protocol is NFS.
    - New option to create AFM peer snapshots through GUI. Use the Create Peer Snapshot option
      in the Files > Snapshots page to create peer snapshots. You can view and delete these peer snapshots from
      the Snapshots page and also from the detailed view of the Files > Active File Management page.

Encryption: GSKit V8 improves cryptographic performance on IBM POWER8

   The IBM Global Security Kit (GSKit) Version 8 and later improves cryptographic performance
   on IBM POWER8 hardware. The version of GSKit that is shipped with IBM Spectrum Scale v5.0.0
   offers better performance on POWER8, compared with the versions shipped with earlier releases.

File compression: The lz4 library provides fast access to compressed data

   File compression supports the lz4 compression library. Lz4 is intended primarily for active data and
   favors read-access speed over maximized space saving.

File data: Block and subblock sizes improve I/O performance and reduce fragmentation

   The default block size is larger, 4 MiB instead of 256 KiB, and the sizes of subblocks relative to blocks
   s are smaller, for example, 8 KiB subblocks in a 4 MiB block. A larger block size improves the
   file system performance and a smaller subblock size reduces the amount of unused space. For many business
   applications, the default value of 4 MiB provides the best balance of improved performance
   and reduced fragmentation.

File encryption: AES-XTS encryption is faster on x86 in non-FIPS mode

    On x86 architecture in non-FIPS mode, file encryption with the AES algorithm in XTS mode
    is faster than it was.

File systems: File system rebalancing is faster

    Rebalancing is implemented by a lenient round-robin method that typically runs faster than the
    previously used method of strict round robin. The strict round robin method is available as an option.

installation toolkit changes

    - The installation toolkit has added support for the installation and the deployment
      of IBM Spectrum Scale in a cluster containing Elastic Storage Server (ESS).
    - The installation toolkit has added support for enabling and configuring call home.
    - The installation toolkit has added support for enabling and configuring file audit logging.
    - The installation toolkit has added support for the installation and the
      deployment of IBM Spectrum Scale on Ubuntu 16.04 LTS nodes.
    - The installation toolkit has added verification of passwordless SSH during
      prechecks before installation, deployment, or upgrade.
    - The installation toolkit has added support for cumulative object upgrade.

mmafmctl command

    The --outband parameter is deprecated.

mmcallhome command: Enhancements

    - Addition of -Y option
        * The -Y displays the command output in a parseable format with a colon (:) as a field delimiter.
    - Addition of --pmr option
        * The --pmr option allows you to upload data to existing PMRs using the
          mmcallhome run SendFile command.

mmchconfig command: Enhancements

    - Encrypted files can be copied into an LROC device
        * With the lrocEnableStoringClearText attribute, you can control whether file
          data from encrypted files, which is held in memory as cleartext, is
          copied into a local read-only cache (LROC) device.
    - InfiniBand addresses can be specified for RDMA transfers
        * In the verbsPorts attribute, you can specify InfiniBand addresses
          for RDMA transfers between an NSD client and server.

mmchnsd command: Change NSDs without unmounting the file system

    When you add or remove NSDs or do other operations with mmchnsd,
    you do not need to unmount the file system.

mmcrfs command: Enhancements

     - The default data block size is 4 MiB with an 8 KiB subblock size
         * If no block size is specified, a file system is created with a 4 MiB block size and an
           8 KiB subblock size. The minimum release level (minReleaseLevel) of the
           cluster must be 5.0.0 or greater when the file system is created.
     - The default log file size depends on block size and metadata size
         * If the block size is 512 KiB or larger and the metadata block size 256 KiB or larger,
           then the default log file size is 32 MiB. Otherwise, the default log file
           size is 4 MiB or the metadata block size, whichever is larger.
     - The default method for updating atime is relatime
         * If the minimum release level (minReleaseLevel) of the cluster is 5.0.0 or greater
           when the file system is created, the default method for updating atime is relatime.

mmdsh command: Several options are no longer supported

     The --ignoreSignal, -I, and -d options are no longer supported.
     Do not use these options unless instructed to by IBM support personnel.

mmfsck command: Display an interim status report at any time

     While a long-running instance of mmfsck is in progress, you can start another instance
     of mmfsck with the --status-report parameter to display current
     status information from all the nodes that are participating in the mmfsck run.

mmgetstate command: Display the unresponsive state

     The command returns the unresponsive state when the GPFS
     daemon is running but is not responding.

mmhealth command: Addition to measurement options

     Measurement options for filesystem, SMB node, and NFS node
     has been added to the mmhealth command.

mmkeyserv command: The simplified method supports certificate chains from a certificate authority.

     In the simplified method, with the --kmip-cert parameter, you can set up encryption with IBM?
     Security Key Lifecycle Manager (SKLM) as the key management server and with a certificate signed
     by a certificate authority (CA) on the KMIP port of the SKLM server.

mmnetverify command: Enhancements

     - Verify the network operation of nodes in a subnet
         * With the --subnets parameters, you can specify the subnet
           addresses of the nodes that you want to verify.
     - Verify that nodes can handle a new MTU size
         * With the ping-packet-size parameter, you can specify the size
           of the ICMP echo request packets that are sent between local node and the target
           node during the ping test.

mmtracectl command: Display the tracing status

     The --status parameter displays the tracing status of the specified nodes.

New feature for threshold monitoring

      Starting from IBM Spectrum Scale version 5.0.0, if multiple thresholds rules
      have overlapping entities for the same metrics, only one of the concurrent rules is
      made actively eligible.

NFS: Dynamic export changes

      You can dynamically change the export configuration without restarting the NFS service.

Object

      - Support for Ubuntu
      - Support for sudo wrapper for Object on Ubuntu
      - Support for cumulative upgrades from older versions
      - Object snap enhancement to contain keystore logs

Protocol support: Enhanced

      Protocol support is extended to Ubuntu 10.04

Setting up a system for storing crash files for Ubuntu

      A topic is added to describe how to set up a system for storing crash files for Ubuntu.

SMB: DFS redirects for SMB shares

      New option to configure DFS redirects for SMB shares.

SMB: SMB server upgrade changes

      Two events on CTDB version match/mismatch are added to the RAS events.

Sudo wrappers: Root-level processes can call administration commands directly

      Root-level background processes, such as cron and callback programs, can
      successfully call administration commands directly rather than through sudo
      when sudo wrappers are enabled.

Supported clients for NFS

      A topic is added listing the clients that are supported by NFS protocol.

Transparent cloud tiering

      - Support for multiple cloud storage accounts
      - Support for multiple file systems or filesets per node group
      - Enhanced support for large file systems provided by container spillover
      - Support for associating file sets with containers for enhanced granularity
      - Multiple URL and region support at the node level
      - Support for creating a cloud service separately for tiering and sharing operations.
      - Unique encryption key per cloud container
      - Support for remotely mounted clients.
      - Support for Amazon S3 regions requiring Sigv4 security support,
        including the US government cloud region.
      - Ability to enable or disable transparent recall for files for a given file
        system instantly, without having to rewrite a policy.
      - Support for backing up and restoring the Cloud services configuration in case of any disaster.
      - Support for backing up the Cloud services database to the cloud.
      - Support for restoring Transparent cloud tiering service on an identical backup cluster.
      - Support for checking the integrity of the Cloud services
        database after any system crash or outage.
      - Support for auditing events relating to each operation performed in Cloud services.

New commands

      mmaudit mmmsgqueue

Changed commands

      mmafmctl mmadddisk mmcallhome mmchattr mmchcluster mmchconfig mmchfs mmchnsd
      mmcloudgateway mmcrcluster mmcrfs mmdeldisk mmdsh mmfsck mmgetstate
      mmkeyserv mmnetverify mmnfs mmrestripefile mmrestripefs mmsmb mmtracectl
      mmuserauth

Deleted commands

      mmrest

New messages

      6027-1264, 6027-1757, 6027-2394, 6027-2395, 6027-2396, 6027-2397
      6027-2398, 6027-2399, 6027-2400, 6027-2401, 6027-3259, 6027-3408
      6027-3597, 6027-3598, 6027-3599, 6027-3600, 6027-3601, 6027-3602
      6027-3603, 6027-3604, 6027-3730, 6027-3921, 6027-3922, 6027-3923
      6027-3924, 6027-3925, 6027-3926, 6027-3927, 6027-3928
      6027-3929, 6027-3930, 6027-3931, 6027-4019

Changed messages

      6027-928


     Open this item<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab-234019da9b-2D248a-2D49ec-2D9503-2De612becd953e&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=Q6E0fg8sor2mORS3ugkurXkuV2ZUDqRd_C6_jpYXVVM&e=>


Posting Date:

Monday, December 18, 2017 at 2:26:38 PM EST



To unsubscribe or change settings, please go to your developerWorks community Settings.<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_news&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=5C4z0hhuaas4Ghs1g4YfFTCE0eyNYJVmLPYrGPCuE7Q&e=>
This is a notification sent from developerWorks community.

The information contained in this e-mail message is intended only for the use of the named recipients and is subject to the IBM Online Privacy Statement<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ibm.com_privacy_details_us_en_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=7gE9uoypP_yHb1jMMSjdCN_xyJ7UU2Ifrx1TchwMkQA&e=>. If you have received this e-mail in error, please contact the sender to report the error and then delete this message from your system.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/9c66dc47/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 289 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/9c66dc47/attachment-0001.png>

From Kevin.Buterbaugh at Vanderbilt.Edu  Mon Dec 18 19:51:56 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Mon, 18 Dec 2017 19:51:56 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
	<DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu>

Hi All,

Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up.

Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this.  Richards? suggestion below errors for me on the ?-g Administrator? part.  Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked.

No matter what I do, I cannot log in to the GUI.  Thanks for the suggestions, though?

Kevin

On Dec 7, 2017, at 6:10 AM, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:

Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error.

That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself.

[root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin
EFSSG0019I The user admin2 has been successfully created.
EFSSG1000I The command completed successfully.


Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 07 December 2017 11:57
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.
Get Outlook for Android<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fghei36&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=sPRr5u4lo%2BAwBPBQ2%2BdXw%2F2EWUAqy30Fk0UNssRFWHU%3D&reserved=0>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin


On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/8eab1462/attachment-0001.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Mon Dec 18 20:06:30 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Mon, 18 Dec 2017 20:06:30 +0000
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
 Central
In-Reply-To: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
References: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
Message-ID: <662BADD6-27D5-4DE6-9EC8-5BE1E73C5873@vanderbilt.edu>

Hi All,

GPFS 5.0 was announced on Friday ? and today:

IBM Spectrum Scale
: IBM Spectrum Scale: NFS operations may fail with IO-Error<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ibm.com%2Fsupport%2Fdocview.wss%3Fuid%3Dssg1S1011791%26myns%3Ds033%26mynp%3DOCSTXKQY%26mynp%3DOCSWJ00%26mync%3DE%26cm_sp%3Ds033-_-OCSTXKQY-OCSWJ00-_-E&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C07b900d4a1aa4e92852e08d54634aa16%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492114479674351&sdata=i%2B8WSjOObZoaSpNLnEZLHq%2FqXwj%2BetYSJzojl9Rh%2FgQ%3D&reserved=0>

IBM has identified an issue with IBM Spectrum Scale 5.0.0.0 Protocol support for NFSv3/v4 in which IO-errors may be returned to the NFS client if the NFS server accumulates file-descriptor resources beyond the defined limit. Accumulation of file descriptor resources will occur when NFSv3 file create operations are sent against files that are already in use.
Bob?s suggestion in a previous e-mail to the list about installing this on a test cluster is almost certainly very, VERY good advice.  That?s certainly what we will do after the holiday break...

Kevin

On Dec 18, 2017, at 1:43 PM, Oesterlin, Robert <Robert.Oesterlin at nuance.com<mailto:Robert.Oesterlin at nuance.com>> wrote:

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Reply-To: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com<mailto:gpfs at us.ibm.com>' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum.

<image001.png> gpfs at us.ibm.com<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DYIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=4BVRC4gq9NXkxODASI9hJ5BV5dQQ94rUjzVc1fDndgs%3D&reserved=0> replied to the IBM Spectrum Scale V5.0 announcements<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3D05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=eCCEi0WsUIaKWOw1BHWueMPQ5O1LF6rYFZnHouwvZ8Y%3D&reserved=0> topic thread in the General Parallel File System - Announce (GPFS - Announce)<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0000-2D0000-2D000000001606%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DzTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=uxfVFydD9TI9l6%2BF%2F3rP7y%2F9Uo0NnBNlmr2tiMrEzbM%3D&reserved=0> forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www-2D933.ibm.com_support_fixcentral%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DiHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=cAwptvUoVH%2B57%2Fat%2FZgt3ft8Gv%2FLsNRWXQzv%2BiuAFuQ%3D&reserved=0>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
as updated, April 2017

Changes to this release of the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library include the following:

Added DMPs for TIP events

       A topic is added listing the directed maintenance procedures for TIP events.
       The DMPs help users resolve issues caused due to TIP events.

AFM and AFM DR

       - Compression and snapshot ILM policy supported.
       - A general recommendation added for the Gateway node.
       - Configuration parameters added - afmMaxParallelRecoveries, afmAsyncOpWaitTimeout,
           afmSyncOpWaitTimeout, and afmRevalOpWaitTimeout.
       - Configuration parameters modified - afmRPO and afmHashVersion.

Authentication: Authentication packages

       Updated the authentication page to include packages specific to Ubuntu.

Authentication: AD-based authentication

       New information is added on NFS with server-side group
       lookup and Active Directory authentication.

Authentication: Primary group selection configurable for AD + RFC2307 based authentication

       Ability to choose primary group as set in "UNIX attributes" of a user on
       Active Directory introduced with AD + RFC2307 based authentication scheme.
       Earlier, Windows primary group was by default selected as the primary group.

Big data and analytics
       - The GPFS Ambari integration package is now called the IBM Spectrum
         Scale Ambari management pack (in short, management pack or MPack).
       - IBM Spectrum Scale Ambari management pack version 2.4.2.1 with
         HDFS Transparency version 2.7.3.1 supports BI 4.2/BI 4.2.5 IOP migration
         to HDP 2.6.2.
       - Supports the remote mount configuration in Ambari.
       - Supports the multiple file systems configuration. In management
         pack version 2.4.2.1, the current limit is two file systems.
       - The Short circuit write is supported for better performance.
       - In HDFS Transparency, the Ranger performance is enhanced.

Changes to IBM Spectrum Scale management API

   Added the following new commands:

       GET /perfmon/data
       GET /filesystems/{filesystemName}/afm/state
       DELETE /nodes/{name}
       POST /nodes
       GET /nodeclasses
       POST /nodeclasses
       DELETE /nodeclasses/{nodeclassName}
       GET /nodeclasses/{nodeclassName}
       PUT /nodeclasses/{nodeclassName}
       DELETE /jobs/jobId
       POST /filesystems/{filesystemName}/filesets/{filesetName}/psnaps
       DELETE /filesystems/{filesystemName}/filesets/{filesetName}/psnaps/{snapshotName}
       GET /thresholds
       GET /thresholds/{name}
       POST /thresholds
       DELETE /thresholds/{name}

IBM Spectrum Scale GUI changes

   - Added new Networks page to monitor the performance, configuration, and
      adapters of network configurations in the cluster. You can monitor the
      network performance with respect to the IP and RDMA interfaces used in the configuration.
    - Added new Monitoring > Thresholds page to create and monitor the threshold rules
      that are defined in the system.
    - Added Access > Remote Connections page to enable the GUI node of the
      local cluster to monitor the remote cluster by establishing a connection
      with the GUI node of the remote cluster.
    - Added Settings > Call Home page to configure call home. Configuring
      the call home feature helps the IBM? Support to monitor the system.
      Configuring call home also helps to reduce the response time of the
      IBM Support to resolve any issues.
      The diagnostic data that is downloaded through the Settings > Diagnostic Data
      can be uploaded to a problem management record (PMR) by using
      the call home feature in the backend. To upload the diagnostic data, right-click
      the relevant data set from the Previously Collected Diagnostic Data, and
      select Upload to PMR.
    - Added file system creation capabilities in GUI. Use the Files > File Systems > Create File System
      option to launch the Create File System wizard. In the Create File System wizard, you can specify
      the following details of the file system:
         - File system name
         - Storage pools
         - NSDs for the file systems
         - Failure groups
         - NSD order for data writes
         - Maximum number of Spectrum Scale clients
         - Maximum number of inodes of the root fileset
         - Whether to enable quota and scope for the quota definition
         - Whether to enable DMAPI
         - Mount point and automatic mount mode
    - Added the aggregation levels Access Point and Filesets and removed Account
      for the resource type Transparent Cloud Tiering in the Monitoring > Statisitics page.
    - The Files > Transparent Cloud Tiering page now displays the file systems
      and filesets that are mapped with the cloud service. It also shows the
      connection of such a container pair configuration to a cloud account and
      the corresponding CSAP that is configured for the cloud account.
    - Changes to capacity monitoring in the GUI
         - Moved the Capacity page from Monitoring to Files menu in the navigation
           and renamed the GUI page to User Capacity.
         - Only the file data user capacity can be viewed from the Files > User Capacity page.
           Removed the pools, filesets, file system capacity monitoring options from the Files > User Capacity page.
           You can monitor the capacity of these components from the respective GUI pages.
         - Replaced the GPFSPoolCap sensor with the GPFSPool sensor and separate data and metadata
           level capacity monitoring are introduced in the performance charts available in the

           Files > File Systems and Storage > Pools pages.
         - New GPFSPool-based data and metadata performance monitoring metrics are available
           for selection in the Files > Statistics > Edit > Capacity section.
           You need to select the aggregation level as Pool to view these metrics.
    - AFM monitoring changes in the GUI
         - Provides the number of AFM filesets and the corresponding export server maps.
           Each export map establishes a mapping between the gateway node and the NFS
           host name to allow parallel data transfers from cache to home.
         - By using the Request Access option available in the Files > Active File Management
           or Access > Remote Connection page in the GUI, you can now establish connection with remote clusters.
           After establishing the connection, you can monitor the following AFM and AFM DR
           configuration details across clusters:
              * On home and secondary, you can see the AFM relationships configuration,
                health status, and performance values of the Cache and Disaster Recovery grids.
              * On the Overview tab of the detailed view, the available home and secondary inodes are available.
              * On the Overview tab of the detailed view, the details such as NFS throughput,
                IOPs, and latency details are available, if the protocol is NFS.
    - New option to create AFM peer snapshots through GUI. Use the Create Peer Snapshot option
      in the Files > Snapshots page to create peer snapshots. You can view and delete these peer snapshots from
      the Snapshots page and also from the detailed view of the Files > Active File Management page.

Encryption: GSKit V8 improves cryptographic performance on IBM POWER8

   The IBM Global Security Kit (GSKit) Version 8 and later improves cryptographic performance
   on IBM POWER8 hardware. The version of GSKit that is shipped with IBM Spectrum Scale v5.0.0
   offers better performance on POWER8, compared with the versions shipped with earlier releases.

File compression: The lz4 library provides fast access to compressed data

   File compression supports the lz4 compression library. Lz4 is intended primarily for active data and
   favors read-access speed over maximized space saving.

File data: Block and subblock sizes improve I/O performance and reduce fragmentation

   The default block size is larger, 4 MiB instead of 256 KiB, and the sizes of subblocks relative to blocks
   s are smaller, for example, 8 KiB subblocks in a 4 MiB block. A larger block size improves the
   file system performance and a smaller subblock size reduces the amount of unused space. For many business
   applications, the default value of 4 MiB provides the best balance of improved performance
   and reduced fragmentation.

File encryption: AES-XTS encryption is faster on x86 in non-FIPS mode

    On x86 architecture in non-FIPS mode, file encryption with the AES algorithm in XTS mode
    is faster than it was.

File systems: File system rebalancing is faster

    Rebalancing is implemented by a lenient round-robin method that typically runs faster than the
    previously used method of strict round robin. The strict round robin method is available as an option.

installation toolkit changes

    - The installation toolkit has added support for the installation and the deployment
      of IBM Spectrum Scale in a cluster containing Elastic Storage Server (ESS).
    - The installation toolkit has added support for enabling and configuring call home.
    - The installation toolkit has added support for enabling and configuring file audit logging.
    - The installation toolkit has added support for the installation and the
      deployment of IBM Spectrum Scale on Ubuntu 16.04 LTS nodes.
    - The installation toolkit has added verification of passwordless SSH during
      prechecks before installation, deployment, or upgrade.
    - The installation toolkit has added support for cumulative object upgrade.

mmafmctl command

    The --outband parameter is deprecated.

mmcallhome command: Enhancements

    - Addition of -Y option
        * The -Y displays the command output in a parseable format with a colon (:) as a field delimiter.
    - Addition of --pmr option
        * The --pmr option allows you to upload data to existing PMRs using the
          mmcallhome run SendFile command.

mmchconfig command: Enhancements

    - Encrypted files can be copied into an LROC device
        * With the lrocEnableStoringClearText attribute, you can control whether file
          data from encrypted files, which is held in memory as cleartext, is
          copied into a local read-only cache (LROC) device.
    - InfiniBand addresses can be specified for RDMA transfers
        * In the verbsPorts attribute, you can specify InfiniBand addresses
          for RDMA transfers between an NSD client and server.

mmchnsd command: Change NSDs without unmounting the file system

    When you add or remove NSDs or do other operations with mmchnsd,
    you do not need to unmount the file system.

mmcrfs command: Enhancements

     - The default data block size is 4 MiB with an 8 KiB subblock size
         * If no block size is specified, a file system is created with a 4 MiB block size and an
           8 KiB subblock size. The minimum release level (minReleaseLevel) of the
           cluster must be 5.0.0 or greater when the file system is created.
     - The default log file size depends on block size and metadata size
         * If the block size is 512 KiB or larger and the metadata block size 256 KiB or larger,
           then the default log file size is 32 MiB. Otherwise, the default log file
           size is 4 MiB or the metadata block size, whichever is larger.
     - The default method for updating atime is relatime
         * If the minimum release level (minReleaseLevel) of the cluster is 5.0.0 or greater
           when the file system is created, the default method for updating atime is relatime.

mmdsh command: Several options are no longer supported

     The --ignoreSignal, -I, and -d options are no longer supported.
     Do not use these options unless instructed to by IBM support personnel.

mmfsck command: Display an interim status report at any time

     While a long-running instance of mmfsck is in progress, you can start another instance
     of mmfsck with the --status-report parameter to display current
     status information from all the nodes that are participating in the mmfsck run.

mmgetstate command: Display the unresponsive state

     The command returns the unresponsive state when the GPFS
     daemon is running but is not responding.

mmhealth command: Addition to measurement options

     Measurement options for filesystem, SMB node, and NFS node
     has been added to the mmhealth command.

mmkeyserv command: The simplified method supports certificate chains from a certificate authority.

     In the simplified method, with the --kmip-cert parameter, you can set up encryption with IBM?
     Security Key Lifecycle Manager (SKLM) as the key management server and with a certificate signed
     by a certificate authority (CA) on the KMIP port of the SKLM server.

mmnetverify command: Enhancements

     - Verify the network operation of nodes in a subnet
         * With the --subnets parameters, you can specify the subnet
           addresses of the nodes that you want to verify.
     - Verify that nodes can handle a new MTU size
         * With the ping-packet-size parameter, you can specify the size
           of the ICMP echo request packets that are sent between local node and the target
           node during the ping test.

mmtracectl command: Display the tracing status

     The --status parameter displays the tracing status of the specified nodes.

New feature for threshold monitoring

      Starting from IBM Spectrum Scale version 5.0.0, if multiple thresholds rules
      have overlapping entities for the same metrics, only one of the concurrent rules is
      made actively eligible.

NFS: Dynamic export changes

      You can dynamically change the export configuration without restarting the NFS service.

Object

      - Support for Ubuntu
      - Support for sudo wrapper for Object on Ubuntu
      - Support for cumulative upgrades from older versions
      - Object snap enhancement to contain keystore logs

Protocol support: Enhanced

      Protocol support is extended to Ubuntu 10.04

Setting up a system for storing crash files for Ubuntu

      A topic is added to describe how to set up a system for storing crash files for Ubuntu.

SMB: DFS redirects for SMB shares

      New option to configure DFS redirects for SMB shares.

SMB: SMB server upgrade changes

      Two events on CTDB version match/mismatch are added to the RAS events.

Sudo wrappers: Root-level processes can call administration commands directly

      Root-level background processes, such as cron and callback programs, can
      successfully call administration commands directly rather than through sudo
      when sudo wrappers are enabled.

Supported clients for NFS

      A topic is added listing the clients that are supported by NFS protocol.

Transparent cloud tiering

      - Support for multiple cloud storage accounts
      - Support for multiple file systems or filesets per node group
      - Enhanced support for large file systems provided by container spillover
      - Support for associating file sets with containers for enhanced granularity
      - Multiple URL and region support at the node level
      - Support for creating a cloud service separately for tiering and sharing operations.
      - Unique encryption key per cloud container
      - Support for remotely mounted clients.
      - Support for Amazon S3 regions requiring Sigv4 security support,
        including the US government cloud region.
      - Ability to enable or disable transparent recall for files for a given file
        system instantly, without having to rewrite a policy.
      - Support for backing up and restoring the Cloud services configuration in case of any disaster.
      - Support for backing up the Cloud services database to the cloud.
      - Support for restoring Transparent cloud tiering service on an identical backup cluster.
      - Support for checking the integrity of the Cloud services
        database after any system crash or outage.
      - Support for auditing events relating to each operation performed in Cloud services.

New commands

      mmaudit mmmsgqueue

Changed commands

      mmafmctl mmadddisk mmcallhome mmchattr mmchcluster mmchconfig mmchfs mmchnsd
      mmcloudgateway mmcrcluster mmcrfs mmdeldisk mmdsh mmfsck mmgetstate
      mmkeyserv mmnetverify mmnfs mmrestripefile mmrestripefs mmsmb mmtracectl
      mmuserauth

Deleted commands

      mmrest

New messages

      6027-1264, 6027-1757, 6027-2394, 6027-2395, 6027-2396, 6027-2397
      6027-2398, 6027-2399, 6027-2400, 6027-2401, 6027-3259, 6027-3408
      6027-3597, 6027-3598, 6027-3599, 6027-3600, 6027-3601, 6027-3602
      6027-3603, 6027-3604, 6027-3730, 6027-3921, 6027-3922, 6027-3923
      6027-3924, 6027-3925, 6027-3926, 6027-3927, 6027-3928
      6027-3929, 6027-3930, 6027-3931, 6027-4019

Changed messages

      6027-928



     Open this item<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab-234019da9b-2D248a-2D49ec-2D9503-2De612becd953e%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DQ6E0fg8sor2mORS3ugkurXkuV2ZUDqRd_C6_jpYXVVM%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=wQawdxgurt9CU5mLHBaUeqAshZxOd7tn7HrOeP4CZR8%3D&reserved=0>


Posting Date:

Monday, December 18, 2017 at 2:26:38 PM EST



To unsubscribe or change settings, please go to your developerWorks community Settings.<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_news%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3D5C4z0hhuaas4Ghs1g4YfFTCE0eyNYJVmLPYrGPCuE7Q%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=KMU2UypqoZXXJlTopsMbsV3x0RUuNMJXEv1UyZEQoRY%3D&reserved=0>

This is a notification sent from developerWorks community.

The information contained in this e-mail message is intended only for the use of the named recipients and is subject to the IBM Online Privacy Statement<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__www.ibm.com_privacy_details_us_en_%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3D7gE9uoypP_yHb1jMMSjdCN_xyJ7UU2Ifrx1TchwMkQA%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=NzaHT12VmX5MY%2FRIm3m4ovKIYNz%2FGMTnJj1%2FM3cOM7I%3D&reserved=0>. If you have received this e-mail in error, please contact the sender to report the error and then delete this message from your system.


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=pHmKmOoSbQ5W4e2DiOfphjWgRdqT8D9NMBjtYZcH%2Bog%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/fd0d1cca/attachment-0001.htm>

From valdis.kletnieks at vt.edu  Mon Dec 18 20:08:48 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Mon, 18 Dec 2017 15:08:48 -0500
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID: <32207.1513627728@turing-police.cc.vt.edu>

Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux

that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4."

We don't use AFM or the install toolkit.  But we *do* make fairly heavy use
of mmces and nfs-ganesha - is that what they mean by "Integrated Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are doing
their upgrades - and going to 7.4 would be nice.  If there's a mine field there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)


From Kevin.Buterbaugh at Vanderbilt.Edu  Mon Dec 18 20:11:28 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Mon, 18 Dec 2017 20:11:28 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
	<DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu>
	<1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov>
Message-ID: <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu>

Hi Jesse,

Thanks for the suggestion ? I find the following error very interesting:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/rmuser admin
EFSSP0010C CLI parser: The object "admin" specified for "userID" does not exist.
/root
root at testnsd1#

That says to me that I don?t have an admin user, which - if true - would explain why not a single password I can think of works.  ;-)

But as I mentioned in my original post I had this up and working earlier this fall.  While I can?t prove anything, I can?t imagine a scenario where I would deliberately choose a non-default username.  So if ?admin? has been the default login for the GPFS GUI all along then I am really mystified.

Thanks!

Kevin

On Dec 18, 2017, at 1:58 PM, Hanley, Jesse A. <hanleyja at ornl.gov<mailto:hanleyja at ornl.gov>> wrote:

Kevin,

I ran into this a couple times using 4.2.3.  This is what we used to get around it:

/usr/lpp/mmfs/gui/cli/rmuser admin
/usr/lpp/mmfs/gui/cli/mkuser admin -p <password> -g Administrator,SecurityAdmin

You may need to run the initgui command if those objects are present.  That typically gets run on first login to the GUI.

Thanks,
--
Jesse


From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Monday, December 18, 2017 at 2:52 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

Hi All,

Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up.

Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this.  Richards? suggestion below errors for me on the ?-g Administrator? part.  Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked.

No matter what I do, I cannot log in to the GUI.  Thanks for the suggestions, though?

Kevin


On Dec 7, 2017, at 6:10 AM, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:

Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error.

That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself.

[root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin
EFSSG0019I The user admin2 has been successfully created.
EFSSG1000I The command completed successfully.


Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 07 December 2017 11:57
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.
Get Outlook for Android<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fghei36&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=sPRr5u4lo%2BAwBPBQ2%2BdXw%2F2EWUAqy30Fk0UNssRFWHU%3D&reserved=0>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin



On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C3f1a258c20a240166fbe08d54651c592%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492239506213881&sdata=qGxMvy3f6cfKQEmwH%2Bcc0SKVDAcaJ4Z66p%2BE9mWDwHg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/656e148e/attachment-0001.htm>

From aaron.s.knister at nasa.gov  Mon Dec 18 20:13:01 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Mon, 18 Dec 2017 20:13:01 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>,
	<CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
Message-ID: <6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>

Thanks Sven! That makes sense to me and is what I thought was the case which is why I was confused when I saw the reply to the thread that said the >32 subblocks code had no performance impact.

A couple more question for you? in your presentation there?s a benchmark that shows the file create performance without the zero padding. Since you mention this is done for security reasons was that feature ever promoted to a GA Scale release? I?m also wondering if you could explain the performance difference between the no zero padding code and the > 32 subblock code since given your the example of 32K files and 16MB block size I figure both cases ought to write the same amount to disk.

Thanks!

-Aaron





On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com> wrote:
i thought i answered that already, but maybe i just thought about answering it and then forgot about it :-D

so yes more than 32 subblocks per block significant increase the performance of filesystems with small files, for the sake of the argument let's say 32k in a large block filesystem again for sake of argument say 16MB.

you probably ask why ?

if you create a file and write 32k into it in a pre 5.0.0 Version 16 MB filesystem your client actually doesn't write 32k to the NSD Server, it writes 512k, because thats the subblock size and we need to write the full subblock (for security reasons). so first you waste significant memory on the client to cache that zero padding, you waste network bandwidth and you waste NSD Server cache because you store it there too. this means you overrun the cache more quickly, means you start doing read/modify writes earlier on all your nice large raid tracks... i guess you get the story by now.

in fact,  if you have a good raid code that can drive really a lot of bandwidth out of individual drives like a GNR system you get more performance for small file writes as larger your blocksize is, because we can 'pack' more files into larger i/os and therefore turn a small file create workload into a bandwidth workload, essentially exactly what we did and i demonstrated in the CORAL presentation .

hope that makes this crystal clear now .

sven



On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>> wrote:
Thanks, Alex. I'm all too familiar with the trade offs between large
blocks and small files and we do use pretty robust SSD storage for our
metadata. We support a wide range of workloads and we have some folks
with many small (<1M) files and other folks with many large (>256MB) files.

My point in this thread is that IBM has said over and over again in
presentations that there is a significant performance gain with the >32
subblocks code on filesystems with large block sizes (although to your
point I'm not clear on exactly what large means since I didn't define
large in this context). Therefore given that the >32 subblock code gives
a significant performance gain one could reasonably assume that having a
filesystem with >32 subblocks is required to see this gain (rather than
just running the >32 subblocks code on an fs w/o > 32 subblocks).

This lead me to ask about a migration tool because in my mind if there's
a performance gain from having >32 subblocks on the FS I'd like that
feature and having to manually copy 10's of PB to new hardware to get
this performance boost is unacceptable. However, IBM can't seem to make
up their mind about whether or not the >32 subblocks code *actually*
provides a performance increase. This seems like a pretty
straightforward question.

-Aaron

On 12/15/17 3:48 PM, Alex Chekholko wrote:
> Hey Aaron,
>
> Can you define your sizes for "large blocks" and "small files"?  If you
> dial one up and the other down, your performance will be worse.  And in
> any case it's a pathological corner case so it shouldn't matter much for
> your workflow, unless you've designed your system with the wrong values.
>
> For example, for bioinformatics workloads, I prefer to use 256KB
> filesystem block size, and I'd consider 4MB+ to be "large block size",
> which would make the filesystem obviously unsuitable for processing
> millions of 8KB files.
>
> You can make a histogram of file sizes in your existing filesystems and
> then make your subblock size (1/32 of block size) on the smaller end of
> that.   Also definitely use the "small file in inode" feature and put
> your metadata on SSD.
>
> Regards,
> Alex
>
> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov> <mailto:aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>>> wrote:
>
>     Thanks, Bill.
>
>     I still don't feel like I've got an clear answer from IBM and frankly
>     the core issue of a lack of migration tool was totally dodged.
>
>     Again in Sven's presentation from SSUG @ SC17
>     (http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
>     he mentions "It has a significant performance penalty for small files in
>     large block size filesystems" and the demonstrates that with several
>     mdtest runs (which show the effect with and without the >32
>     subblocks code):
>
>
>     4.2.1 base code - SUMMARY: (of 3 iterations)
>     File creation : Mean = 2237.644
>
>     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY: (of
>     3 iterations)
>     File creation : Mean = 12866.842
>
>     more sub blocks per block (4.2.2 + morethan32subblock code):
>     File creation : Mean = 40316.721
>
>     Can someone (ideally Sven) give me a straight answer as to whether or
>     not the > 32 subblock code actually makes a performance difference for
>     small files in large block filesystems? And if not, help me understand
>     why his slides and provided benchmark data have consistently indicated
>     it does?
>
>     -Aaron
>
>     On 12/1/17 11:44 AM, Bill Hartner wrote:
>     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
>     > results @ user group meeting.
>     >
>     > -Bill
>     >
>     > Bill Hartner
>     > IBM Systems
>     > Scalable I/O Development
>     > Austin, Texas
>     > bhartner at us.ibm.com<mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>>
>     > home office 512-784-0980<tel:(512)%20784-0980> <tel:512-784-0980<tel:(512)%20784-0980>>
>     >
>     >
>     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
>     > AM---Bill, could you say something about what the metadataJan-Frode
>     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
>     > about what the metadata-storage here was? ESS/NL-SAS/3way replication?
>     >
>     > From: Jan-Frode Myklebust <janfrode at tanso.net<mailto:janfrode at tanso.net> <mailto:janfrode at tanso.net<mailto:janfrode at tanso.net>>>
>     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
>     <mailto:gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>>
>     > Date: 12/01/2017 06:53 AM
>     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     > Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>     >
>     >
>     ------------------------------------------------------------------------
>     >
>     >
>     >
>     > Bill, could you say something about what the metadata-storage here was?
>     > ESS/NL-SAS/3way replication?
>     >
>     > I just asked about this in the internal slack channel #scale-help today..
>     >
>     >
>     >
>     > -jf
>     >
>     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
>     > <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>>>>:
>     >
>     >     > "It has a significant performance penalty for small files in
>     large
>     >     > block size filesystems"
>     >
>     >     Aaron,
>     >
>     >     Below are mdtest results for a test we ran for CORAL - file
>     size was
>     >     32k.
>     >
>     >     We have not gone back and ran the test on a file system formatted
>     >     without > 32 subblocks. We'll do that at some point...
>     >
>     >     -Bill
>     >
>     >     -- started at 10/28/2017 17:51:38 --
>     >
>     >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>     >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>     >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
>     -p 360
>     >     -u -y
>     >     Path: /ibm/fs2-16m-10
>     >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>     >
>     >     228 tasks, 67239936 files
>     >
>     >     SUMMARY: (of 3 iterations)
>     >     Operation Max Min Mean Std Dev
>     >     --------- --- --- ---- -------
>     >     File creation : 51953.498 50558.517 51423.221 616.643
>     >     File stat : 0.000 0.000 0.000 0.000
>     >     File read : 0.000 0.000 0.000 0.000
>     >     File removal : 96746.376 92149.535 94658.774 1900.187
>     >     Tree creation : 1.588 0.070 0.599 0.700
>     >     Tree removal : 0.213 0.034 0.097 0.082
>     >
>     >     -- finished at 10/28/2017 19:51:54 --
>     >
>     >     Bill Hartner
>     >     IBM Systems
>     >     Scalable I/O Development
>     >     Austin, Texas_
>     >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>
>     <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>>>
>     >     home office 512-784-0980<tel:(512)%20784-0980> <tel:512-784-0980<tel:(512)%20784-0980>>
>     >
>     >     _
>     >     __gpfsug-discuss-bounces at spectrumscale.org_
>     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>> wrote on
>     >     11/29/2017 04:41:48 PM:
>     >
>     >     > From: Aaron Knister <_aaron.knister at gmail.com_
>     >     <mailto:aaron.knister at gmail.com<mailto:aaron.knister at gmail.com> <mailto:aaron.knister at gmail.com<mailto:aaron.knister at gmail.com>>>>
>     >
>     >
>     >     > To: gpfsug main discussion list
>     >     <_gpfsug-discuss at spectrumscale.org_
>     >     <mailto:gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
>     <mailto:gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>>>
>     >
>     >     > Date: 11/29/2017 04:42 PM
>     >
>     >
>     >     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>>
>     >
>     >     >
>     >
>     >     > Thanks, Nikhil. Most of that was consistent with my understnading,
>     >     > however I was under the impression that the >32 subblocks code is
>     >     > required to achieve the touted 50k file creates/second that Sven has
>     >     > talked about a bunch of times:
>     >     >
>     >     >
>     >
>      _http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>     <http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>     >     >
>     >
>      _http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
>     <http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>     >     > _Sven_Oehme_-_News_from_Research.pdf
>     >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>     >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>     >
>     >
>     >     > from those presentations regarding 32 subblocks:
>     >     >
>     >     > "It has a significant performance penalty for small files in large
>     >     > block size filesystems"
>     >
>     >     > although I'm not clear on the specific definition of "large". Many
>     >     > filesystems I encounter only have a 1M block size so it may not
>     >     > matter there, although that same presentation clearly shows the
>     >     > benefit of larger block sizes which is yet *another* thing for which
>     >     > a migration tool would be helpful.
>     >
>     >     > -Aaron
>     >     >
>     >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>     >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com<mailto:nikhilk at us.ibm.com>
>     <mailto:nikhilk at us.ibm.com<mailto:nikhilk at us.ibm.com>>>> wrote:
>     >
>     >     > Hi,
>     >     >
>     >     > I would like to clarify migration path to 5.0.0 from 4.X.X
>     clusters.
>     >     > For all Spectrum Scale clusters that are currently at 4.X.X,
>     it is
>     >     > possible to migrate to 5.0.0 with no offline data migration
>     and no
>     >     > need to move data. Once these clusters are at 5.0.0, they will
>     >     > benefit from the performance improvements, new features (such as
>     >     > file audit logging), and various enhancements that are
>     included in
>     >     5.0.0.
>     >     >
>     >     > That being said, there is one enhancement that will not be
>     applied
>     >     > to these clusters, and that is the increased number of
>     sub-blocks
>     >     > per block for small file allocation. This means that for file
>     >     > systems with a large block size and a lot of small files, the
>     >     > overall space utilization will be the same it currently is
>     in 4.X.X.
>     >     > Since file systems created at 4.X.X and earlier used a block
>     size
>     >     > that kept this allocation in mind, there should be very little
>     >     > impact on existing file systems.
>     >     >
>     >     > Outside of that one particular function, the remainder of the
>     >     > performance improvements, metadata improvements, updated
>     >     > compatibility, new functionality, and all of the other
>     enhancements
>     >     > will be immediately available to you once you complete the
>     upgrade
>     >     > to 5.0.0 -- with no need to reformat, move data, or take
>     your data
>     >     offline.
>     >     >
>     >     > I hope that clarifies things a little and makes the upgrade path
>     >     > more accessible.
>     >     >
>     >     > Please let me know if there are any other questions or concerns.
>     >     >
>     >     > Thank you,
>     >     > Nikhil Khandelwal
>     >     > Spectrum Scale Development
>     >     > Client Adoption
>     >     >
>     >     > _______________________________________________
>     >     > gpfsug-discuss mailing list
>     >     > gpfsug-discuss at _spectrumscale.org_
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>     >
>     >     > _______________________________________________
>     >     > gpfsug-discuss mailing list
>     >     > gpfsug-discuss at _spectrumscale.org_
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >
>     >     > _https://urldefense.proofpoint.com/v2/url?_
>     <https://urldefense.proofpoint.com/v2/url?_>
>     >     >
>     >     u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>     >     >
>     >     siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>     >
>     >     _______________________________________________
>     >     gpfsug-discuss mailing list
>     >     gpfsug-discuss at _spectrumscale.org_
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>     >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>     >     gpfsug-discuss mailing list
>     >     gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> <http://spectrumscale.org>
>     >
>      https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> <http://spectrumscale.org>
>     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>     >
>
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776<tel:(301)%20286-2776> <tel:%28301%29%20286-2776>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776<tel:(301)%20286-2776>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/62da1e91/attachment-0001.htm>

From taylorm at us.ibm.com  Mon Dec 18 20:27:42 2017
From: taylorm at us.ibm.com (Michael L Taylor)
Date: Mon, 18 Dec 2017 13:27:42 -0700
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
	Central
In-Reply-To: <mailman.3372.1513626221.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3372.1513626221.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <OFD183EA98.E6691217-ON002581FA.006FA7A4-072581FA.00706687@notes.na.collabserv.com>



Hi Bob,
Thanks for the note on 5.0.0
One correction however.... clusters can do a rolling upgrade to 5.0.0 from
any 4.2.x level (not just 4.2.3).




Today's Topics:

   1. FW: Spectrum Scale 5.0 now available on Fix Central
      (Oesterlin, Robert)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 19:43:35 +0000
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
		 Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53 at nuance.com>
Content-Type: text/plain; charset="utf-8"

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to
this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum
Scale V5.0 announcements' topic thread in the 'General Parallel File System
- Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=
> replied to the IBM Spectrum Scale V5.0 announcements<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=
> topic thread in the General Parallel File System - Announce (GPFS -
Announce)<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0


000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=>
 forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=
>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/157250d2/attachment-0001.htm>

From stockf at us.ibm.com  Mon Dec 18 21:10:55 2017
From: stockf at us.ibm.com (Frederick Stock)
Date: Mon, 18 Dec 2017 16:10:55 -0500
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
In-Reply-To: <32207.1513627728@turing-police.cc.vt.edu>
References: <32207.1513627728@turing-police.cc.vt.edu>
Message-ID: <OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF@notes.na.collabserv.com>

Yes the integrated protocols are the Samba and Ganesha that are bundled 
with Spectrum Scale.  These require the use of the CES component for 
monitoring the protocols.  If you do use them then you need to wait for a 
release of Spectrum Scale in which the integrated protocols are also 
supported on RHEL 7.4.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com



From:   valdis.kletnieks at vt.edu
To:     gpfsug-discuss at spectrumscale.org
Date:   12/18/2017 03:09 PM
Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux


that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on 
RHEL 7.4."

We don't use AFM or the install toolkit.  But we *do* make fairly heavy 
use
of mmces and nfs-ganesha - is that what they mean by "Integrated 
Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are 
doing
their upgrades - and going to 7.4 would be nice.  If there's a mine field 
there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e=






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/7c272339/attachment-0001.htm>

From erich at uw.edu  Mon Dec 18 21:46:02 2017
From: erich at uw.edu (Eric Horst)
Date: Mon, 18 Dec 2017 21:46:02 +0000
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
In-Reply-To: <OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF@notes.na.collabserv.com>
References: <32207.1513627728@turing-police.cc.vt.edu>
	<OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF@notes.na.collabserv.com>
Message-ID: <CAF-To-3w7FGHf_ks8cv4WZ=ZF42ZHVOZo4rbu2K8VTQtgAQUWg@mail.gmail.com>

Grr, this might explain why I experienced unhappiness when I tried to start
my long-delayed AFM based migration over the weekend. I had previously
tested AFM and found everything working, but 7.4 may have slipped in last
month. The AFM relationship seems to work but `mmafmctl premigrate`
commands fail. I would revert packages if I could figure out where the
issue lies.

-Eric

On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock <stockf at us.ibm.com> wrote:

> Yes the integrated protocols are the Samba and Ganesha that are bundled
> with Spectrum Scale.  These require the use of the CES component for
> monitoring the protocols.  If you do use them then you need to wait for a
> release of Spectrum Scale in which the integrated protocols are also
> supported on RHEL 7.4.
>
> Fred
> __________________________________________________
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821>
> stockf at us.ibm.com
>
>
>
> From:        valdis.kletnieks at vt.edu
> To:        gpfsug-discuss at spectrumscale.org
> Date:        12/18/2017 03:09 PM
> Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Currently, the IBM support matrix says:
>
> https://www.ibm.com/support/knowledgecenter/STXKQY/
> gpfsclustersfaq.html#linux
>
> that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:
>
> "AFM, Integrated Protocols, and Installation Toolkit are not supported on
> RHEL 7.4."
>
> We don't use AFM or the install toolkit.  But we *do* make fairly heavy use
> of mmces and nfs-ganesha - is that what they mean by "Integrated
> Protocols"?
>
> (We're looking at doing upgrades next month while our HPC clusters are
> doing
> their upgrades - and going to 7.4 would be nice.  If there's a mine field
> there, I need to
> make sure we stay at 7.3 - plus applicable non-7.4 updates)
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_
> iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=
> 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_
> I4NKvn8aFGXCqcJARCbu4AOE&e=
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/c01cc906/attachment-0001.htm>

From a.khiredine at meteo.dz  Tue Dec 19 08:08:03 2017
From: a.khiredine at meteo.dz (atmane khiredine)
Date: Tue, 19 Dec 2017 08:08:03 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35
In-Reply-To: <mailman.3382.1513633600.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3382.1513633600.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz>

IBM Spectrum Scale V5.0 not support RHEL 6.x

Only RHEL 7.1 or later

https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux


Atmane Khiredine
HPC System Administrator | Office National de la M?t?orologie
T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz

________________________________________
De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org]
Envoy? : lundi 18 d?cembre 2017 22:46
? : gpfsug-discuss at spectrumscale.org
Objet : gpfsug-discuss Digest, Vol 71, Issue 35

Send gpfsug-discuss mailing list submissions to
        gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
        gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
        gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Re: FW: Spectrum Scale 5.0 now available on Fix   Central
      (Michael L Taylor)
   2. Re: gpfs 4.2.3.5 and RHEL 7.4... (Frederick Stock)
   3. Re: gpfs 4.2.3.5 and RHEL 7.4... (Eric Horst)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 13:27:42 -0700
From: "Michael L Taylor" <taylorm at us.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on
        Fix     Central
Message-ID:
        <OFD183EA98.E6691217-ON002581FA.006FA7A4-072581FA.00706687 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"



Hi Bob,
Thanks for the note on 5.0.0
One correction however.... clusters can do a rolling upgrade to 5.0.0 from
any 4.2.x level (not just 4.2.3).




Today's Topics:

   1. FW: Spectrum Scale 5.0 now available on Fix Central
      (Oesterlin, Robert)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 19:43:35 +0000
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
                 Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53 at nuance.com>
Content-Type: text/plain; charset="utf-8"

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to
this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum
Scale V5.0 announcements' topic thread in the 'General Parallel File System
- Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=
> replied to the IBM Spectrum Scale V5.0 announcements<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=
> topic thread in the General Parallel File System - Announce (GPFS -
Announce)<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0


000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=>
 forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=
>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/157250d2/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 18 Dec 2017 16:10:55 -0500
From: "Frederick Stock" <stockf at us.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
        <OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

Yes the integrated protocols are the Samba and Ganesha that are bundled
with Spectrum Scale.  These require the use of the CES component for
monitoring the protocols.  If you do use them then you need to wait for a
release of Spectrum Scale in which the integrated protocols are also
supported on RHEL 7.4.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com



From:   valdis.kletnieks at vt.edu
To:     gpfsug-discuss at spectrumscale.org
Date:   12/18/2017 03:09 PM
Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux


that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on
RHEL 7.4."

We don't use AFM or the install toolkit.  But we *do* make fairly heavy
use
of mmces and nfs-ganesha - is that what they mean by "Integrated
Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are
doing
their upgrades - and going to 7.4 would be nice.  If there's a mine field
there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e=






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/7c272339/attachment-0001.html>

------------------------------

Message: 3
Date: Mon, 18 Dec 2017 21:46:02 +0000
From: Eric Horst <erich at uw.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
        <CAF-To-3w7FGHf_ks8cv4WZ=ZF42ZHVOZo4rbu2K8VTQtgAQUWg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Grr, this might explain why I experienced unhappiness when I tried to start
my long-delayed AFM based migration over the weekend. I had previously
tested AFM and found everything working, but 7.4 may have slipped in last
month. The AFM relationship seems to work but `mmafmctl premigrate`
commands fail. I would revert packages if I could figure out where the
issue lies.

-Eric

On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock <stockf at us.ibm.com> wrote:

> Yes the integrated protocols are the Samba and Ganesha that are bundled
> with Spectrum Scale.  These require the use of the CES component for
> monitoring the protocols.  If you do use them then you need to wait for a
> release of Spectrum Scale in which the integrated protocols are also
> supported on RHEL 7.4.
>
> Fred
> __________________________________________________
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821>
> stockf at us.ibm.com
>
>
>
> From:        valdis.kletnieks at vt.edu
> To:        gpfsug-discuss at spectrumscale.org
> Date:        12/18/2017 03:09 PM
> Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Currently, the IBM support matrix says:
>
> https://www.ibm.com/support/knowledgecenter/STXKQY/
> gpfsclustersfaq.html#linux
>
> that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:
>
> "AFM, Integrated Protocols, and Installation Toolkit are not supported on
> RHEL 7.4."
>
> We don't use AFM or the install toolkit.  But we *do* make fairly heavy use
> of mmces and nfs-ganesha - is that what they mean by "Integrated
> Protocols"?
>
> (We're looking at doing upgrades next month while our HPC clusters are
> doing
> their upgrades - and going to 7.4 would be nice.  If there's a mine field
> there, I need to
> make sure we stay at 7.3 - plus applicable non-7.4 updates)
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_
> iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=
> 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_
> I4NKvn8aFGXCqcJARCbu4AOE&e=
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/c01cc906/attachment.html>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 71, Issue 35
**********************************************


From aaron.s.knister at nasa.gov  Tue Dec 19 08:31:37 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Tue, 19 Dec 2017 08:31:37 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35
In-Reply-To: <4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz>
References: <mailman.3382.1513633600.19082.gpfsug-discuss@spectrumscale.org>,
	<4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz>
Message-ID: <44E99F55-25CC-48DB-9AD6-E7D6794694DC@nasa.gov>

It?s not supported on SLES11 either.


IBM didn?t (that I saw) talk much about this publicly or give customers a chance to provide feedback about the decision. I know it was raised at the UG in NY and I recall a number of people saying it would be a significant issue for them (myself included) as is the fact they no longer support Debian with scale 5.0.

I?d raised the issue on the mailing list after the UG trying to start the discussion but IBM said they weren?t ready to talk about it publicly and I can only guess they had already set their sights and didn?t actually want feedback. This is actually pretty frustrating. I?m tempted to open an RFE but most of my RFEs either have been rejected or just sit idle so I?m not clear there?s a benefit.



On December 19, 2017 at 03:08:27 EST, atmane khiredine <a.khiredine at meteo.dz> wrote:
IBM Spectrum Scale V5.0 not support RHEL 6.x

Only RHEL 7.1 or later

https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux


Atmane Khiredine
HPC System Administrator | Office National de la M?t?orologie
T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz

________________________________________
De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org]
Envoy? : lundi 18 d?cembre 2017 22:46
? : gpfsug-discuss at spectrumscale.org
Objet : gpfsug-discuss Digest, Vol 71, Issue 35

Send gpfsug-discuss mailing list submissions to
gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

1. Re: FW: Spectrum Scale 5.0 now available on Fix Central
(Michael L Taylor)
2. Re: gpfs 4.2.3.5 and RHEL 7.4... (Frederick Stock)
3. Re: gpfs 4.2.3.5 and RHEL 7.4... (Eric Horst)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 13:27:42 -0700
From: "Michael L Taylor" <taylorm at us.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on
Fix Central
Message-ID:
<OFD183EA98.E6691217-ON002581FA.006FA7A4-072581FA.00706687 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"



Hi Bob,
Thanks for the note on 5.0.0
One correction however.... clusters can do a rolling upgrade to 5.0.0 from
any 4.2.x level (not just 4.2.3).




Today's Topics:

1. FW: Spectrum Scale 5.0 now available on Fix Central
(Oesterlin, Robert)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 19:43:35 +0000
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53 at nuance.com>
Content-Type: text/plain; charset="utf-8"

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to
this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum
Scale V5.0 announcements' topic thread in the 'General Parallel File System
- Announce (GPFS - Announce)' forum.

[mid:4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75 at SDEB-EXC02.meteo.dz/forums.png] gpfs at us.ibm.com<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=
> replied to the IBM Spectrum Scale V5.0 announcements<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=
> topic thread in the General Parallel File System - Announce (GPFS -
Announce)<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0


000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=>
forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=
>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/157250d2/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 18 Dec 2017 16:10:55 -0500
From: "Frederick Stock" <stockf at us.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
<OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

Yes the integrated protocols are the Samba and Ganesha that are bundled
with Spectrum Scale. These require the use of the CES component for
monitoring the protocols. If you do use them then you need to wait for a
release of Spectrum Scale in which the integrated protocols are also
supported on RHEL 7.4.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com



From: valdis.kletnieks at vt.edu
To: gpfsug-discuss at spectrumscale.org
Date: 12/18/2017 03:09 PM
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Sent by: gpfsug-discuss-bounces at spectrumscale.org



Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux


that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on
RHEL 7.4."

We don't use AFM or the install toolkit. But we *do* make fairly heavy
use
of mmces and nfs-ganesha - is that what they mean by "Integrated
Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are
doing
their upgrades - and going to 7.4 would be nice. If there's a mine field
there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e=






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/7c272339/attachment-0001.html>

------------------------------

Message: 3
Date: Mon, 18 Dec 2017 21:46:02 +0000
From: Eric Horst <erich at uw.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
<CAF-To-3w7FGHf_ks8cv4WZ=ZF42ZHVOZo4rbu2K8VTQtgAQUWg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Grr, this might explain why I experienced unhappiness when I tried to start
my long-delayed AFM based migration over the weekend. I had previously
tested AFM and found everything working, but 7.4 may have slipped in last
month. The AFM relationship seems to work but `mmafmctl premigrate`
commands fail. I would revert packages if I could figure out where the
issue lies.

-Eric

On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock <stockf at us.ibm.com> wrote:

> Yes the integrated protocols are the Samba and Ganesha that are bundled
> with Spectrum Scale. These require the use of the CES component for
> monitoring the protocols. If you do use them then you need to wait for a
> release of Spectrum Scale in which the integrated protocols are also
> supported on RHEL 7.4.
>
> Fred
> __________________________________________________
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821>
> stockf at us.ibm.com
>
>
>
> From: valdis.kletnieks at vt.edu
> To: gpfsug-discuss at spectrumscale.org
> Date: 12/18/2017 03:09 PM
> Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Currently, the IBM support matrix says:
>
> https://www.ibm.com/support/knowledgecenter/STXKQY/
> gpfsclustersfaq.html#linux
>
> that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:
>
> "AFM, Integrated Protocols, and Installation Toolkit are not supported on
> RHEL 7.4."
>
> We don't use AFM or the install toolkit. But we *do* make fairly heavy use
> of mmces and nfs-ganesha - is that what they mean by "Integrated
> Protocols"?
>
> (We're looking at doing upgrades next month while our HPC clusters are
> doing
> their upgrades - and going to 7.4 would be nice. If there's a mine field
> there, I need to
> make sure we stay at 7.3 - plus applicable non-7.4 updates)
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_
> iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=
> 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_
> I4NKvn8aFGXCqcJARCbu4AOE&e=
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/c01cc906/attachment.html>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 71, Issue 35
**********************************************
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/78969e45/attachment-0001.htm>

From r.sobey at imperial.ac.uk  Tue Dec 19 09:06:08 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 19 Dec 2017 09:06:08 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
In-Reply-To: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
References: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
Message-ID: <DB6PR0601MB2101BA78601B08F6F06469F8DF0F0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Hi Robert

Do you mean the minReleaseLevel from mmlsconfig or just making sure all the nodes are running 4.2.3?

Cheers!
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert
Sent: 18 December 2017 19:44
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Reply-To: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=> replied to the IBM Spectrum Scale V5.0 announcements<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=> topic thread in the General Parallel File System - Announce (GPFS - Announce)<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=> forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
as updated, April 2017

Changes to this release of the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library include the following:

Added DMPs for TIP events

       A topic is added listing the directed maintenance procedures for TIP events.
       The DMPs help users resolve issues caused due to TIP events.

AFM and AFM DR

       - Compression and snapshot ILM policy supported.
       - A general recommendation added for the Gateway node.
       - Configuration parameters added - afmMaxParallelRecoveries, afmAsyncOpWaitTimeout,
           afmSyncOpWaitTimeout, and afmRevalOpWaitTimeout.
       - Configuration parameters modified - afmRPO and afmHashVersion.

Authentication: Authentication packages

       Updated the authentication page to include packages specific to Ubuntu.

Authentication: AD-based authentication

       New information is added on NFS with server-side group
       lookup and Active Directory authentication.

Authentication: Primary group selection configurable for AD + RFC2307 based authentication

       Ability to choose primary group as set in "UNIX attributes" of a user on
       Active Directory introduced with AD + RFC2307 based authentication scheme.
       Earlier, Windows primary group was by default selected as the primary group.

Big data and analytics
       - The GPFS Ambari integration package is now called the IBM Spectrum
         Scale Ambari management pack (in short, management pack or MPack).
       - IBM Spectrum Scale Ambari management pack version 2.4.2.1 with
         HDFS Transparency version 2.7.3.1 supports BI 4.2/BI 4.2.5 IOP migration
         to HDP 2.6.2.
       - Supports the remote mount configuration in Ambari.
       - Supports the multiple file systems configuration. In management
         pack version 2.4.2.1, the current limit is two file systems.
       - The Short circuit write is supported for better performance.
       - In HDFS Transparency, the Ranger performance is enhanced.

Changes to IBM Spectrum Scale management API

   Added the following new commands:

       GET /perfmon/data
       GET /filesystems/{filesystemName}/afm/state
       DELETE /nodes/{name}
       POST /nodes
       GET /nodeclasses
       POST /nodeclasses
       DELETE /nodeclasses/{nodeclassName}
       GET /nodeclasses/{nodeclassName}
       PUT /nodeclasses/{nodeclassName}
       DELETE /jobs/jobId
       POST /filesystems/{filesystemName}/filesets/{filesetName}/psnaps
       DELETE /filesystems/{filesystemName}/filesets/{filesetName}/psnaps/{snapshotName}
       GET /thresholds
       GET /thresholds/{name}
       POST /thresholds
       DELETE /thresholds/{name}

IBM Spectrum Scale GUI changes

   - Added new Networks page to monitor the performance, configuration, and
      adapters of network configurations in the cluster. You can monitor the
      network performance with respect to the IP and RDMA interfaces used in the configuration.
    - Added new Monitoring > Thresholds page to create and monitor the threshold rules
      that are defined in the system.
    - Added Access > Remote Connections page to enable the GUI node of the
      local cluster to monitor the remote cluster by establishing a connection
      with the GUI node of the remote cluster.
    - Added Settings > Call Home page to configure call home. Configuring
      the call home feature helps the IBM? Support to monitor the system.
      Configuring call home also helps to reduce the response time of the
      IBM Support to resolve any issues.
      The diagnostic data that is downloaded through the Settings > Diagnostic Data
      can be uploaded to a problem management record (PMR) by using
      the call home feature in the backend. To upload the diagnostic data, right-click
      the relevant data set from the Previously Collected Diagnostic Data, and
      select Upload to PMR.
    - Added file system creation capabilities in GUI. Use the Files > File Systems > Create File System
      option to launch the Create File System wizard. In the Create File System wizard, you can specify
      the following details of the file system:
         - File system name
         - Storage pools
         - NSDs for the file systems
         - Failure groups
         - NSD order for data writes
         - Maximum number of Spectrum Scale clients
         - Maximum number of inodes of the root fileset
         - Whether to enable quota and scope for the quota definition
         - Whether to enable DMAPI
         - Mount point and automatic mount mode
    - Added the aggregation levels Access Point and Filesets and removed Account
      for the resource type Transparent Cloud Tiering in the Monitoring > Statisitics page.
    - The Files > Transparent Cloud Tiering page now displays the file systems
      and filesets that are mapped with the cloud service. It also shows the
      connection of such a container pair configuration to a cloud account and
      the corresponding CSAP that is configured for the cloud account.
    - Changes to capacity monitoring in the GUI
         - Moved the Capacity page from Monitoring to Files menu in the navigation
           and renamed the GUI page to User Capacity.
         - Only the file data user capacity can be viewed from the Files > User Capacity page.
           Removed the pools, filesets, file system capacity monitoring options from the Files > User Capacity page.
           You can monitor the capacity of these components from the respective GUI pages.
         - Replaced the GPFSPoolCap sensor with the GPFSPool sensor and separate data and metadata
           level capacity monitoring are introduced in the performance charts available in the

           Files > File Systems and Storage > Pools pages.
         - New GPFSPool-based data and metadata performance monitoring metrics are available
           for selection in the Files > Statistics > Edit > Capacity section.
           You need to select the aggregation level as Pool to view these metrics.
    - AFM monitoring changes in the GUI
         - Provides the number of AFM filesets and the corresponding export server maps.
           Each export map establishes a mapping between the gateway node and the NFS
           host name to allow parallel data transfers from cache to home.
         - By using the Request Access option available in the Files > Active File Management
           or Access > Remote Connection page in the GUI, you can now establish connection with remote clusters.
           After establishing the connection, you can monitor the following AFM and AFM DR
           configuration details across clusters:
              * On home and secondary, you can see the AFM relationships configuration,
                health status, and performance values of the Cache and Disaster Recovery grids.
              * On the Overview tab of the detailed view, the available home and secondary inodes are available.
              * On the Overview tab of the detailed view, the details such as NFS throughput,
                IOPs, and latency details are available, if the protocol is NFS.
    - New option to create AFM peer snapshots through GUI. Use the Create Peer Snapshot option
      in the Files > Snapshots page to create peer snapshots. You can view and delete these peer snapshots from
      the Snapshots page and also from the detailed view of the Files > Active File Management page.

Encryption: GSKit V8 improves cryptographic performance on IBM POWER8

   The IBM Global Security Kit (GSKit) Version 8 and later improves cryptographic performance
   on IBM POWER8 hardware. The version of GSKit that is shipped with IBM Spectrum Scale v5.0.0
   offers better performance on POWER8, compared with the versions shipped with earlier releases.

File compression: The lz4 library provides fast access to compressed data

   File compression supports the lz4 compression library. Lz4 is intended primarily for active data and
   favors read-access speed over maximized space saving.

File data: Block and subblock sizes improve I/O performance and reduce fragmentation

   The default block size is larger, 4 MiB instead of 256 KiB, and the sizes of subblocks relative to blocks
   s are smaller, for example, 8 KiB subblocks in a 4 MiB block. A larger block size improves the
   file system performance and a smaller subblock size reduces the amount of unused space. For many business
   applications, the default value of 4 MiB provides the best balance of improved performance
   and reduced fragmentation.

File encryption: AES-XTS encryption is faster on x86 in non-FIPS mode

    On x86 architecture in non-FIPS mode, file encryption with the AES algorithm in XTS mode
    is faster than it was.

File systems: File system rebalancing is faster

    Rebalancing is implemented by a lenient round-robin method that typically runs faster than the
    previously used method of strict round robin. The strict round robin method is available as an option.

installation toolkit changes

    - The installation toolkit has added support for the installation and the deployment
      of IBM Spectrum Scale in a cluster containing Elastic Storage Server (ESS).
    - The installation toolkit has added support for enabling and configuring call home.
    - The installation toolkit has added support for enabling and configuring file audit logging.
    - The installation toolkit has added support for the installation and the
      deployment of IBM Spectrum Scale on Ubuntu 16.04 LTS nodes.
    - The installation toolkit has added verification of passwordless SSH during
      prechecks before installation, deployment, or upgrade.
    - The installation toolkit has added support for cumulative object upgrade.

mmafmctl command

    The --outband parameter is deprecated.

mmcallhome command: Enhancements

    - Addition of -Y option
        * The -Y displays the command output in a parseable format with a colon (:) as a field delimiter.
    - Addition of --pmr option
        * The --pmr option allows you to upload data to existing PMRs using the
          mmcallhome run SendFile command.

mmchconfig command: Enhancements

    - Encrypted files can be copied into an LROC device
        * With the lrocEnableStoringClearText attribute, you can control whether file
          data from encrypted files, which is held in memory as cleartext, is
          copied into a local read-only cache (LROC) device.
    - InfiniBand addresses can be specified for RDMA transfers
        * In the verbsPorts attribute, you can specify InfiniBand addresses
          for RDMA transfers between an NSD client and server.

mmchnsd command: Change NSDs without unmounting the file system

    When you add or remove NSDs or do other operations with mmchnsd,
    you do not need to unmount the file system.

mmcrfs command: Enhancements

     - The default data block size is 4 MiB with an 8 KiB subblock size
         * If no block size is specified, a file system is created with a 4 MiB block size and an
           8 KiB subblock size. The minimum release level (minReleaseLevel) of the
           cluster must be 5.0.0 or greater when the file system is created.
     - The default log file size depends on block size and metadata size
         * If the block size is 512 KiB or larger and the metadata block size 256 KiB or larger,
           then the default log file size is 32 MiB. Otherwise, the default log file
           size is 4 MiB or the metadata block size, whichever is larger.
     - The default method for updating atime is relatime
         * If the minimum release level (minReleaseLevel) of the cluster is 5.0.0 or greater
           when the file system is created, the default method for updating atime is relatime.

mmdsh command: Several options are no longer supported

     The --ignoreSignal, -I, and -d options are no longer supported.
     Do not use these options unless instructed to by IBM support personnel.

mmfsck command: Display an interim status report at any time

     While a long-running instance of mmfsck is in progress, you can start another instance
     of mmfsck with the --status-report parameter to display current
     status information from all the nodes that are participating in the mmfsck run.

mmgetstate command: Display the unresponsive state

     The command returns the unresponsive state when the GPFS
     daemon is running but is not responding.

mmhealth command: Addition to measurement options

     Measurement options for filesystem, SMB node, and NFS node
     has been added to the mmhealth command.

mmkeyserv command: The simplified method supports certificate chains from a certificate authority.

     In the simplified method, with the --kmip-cert parameter, you can set up encryption with IBM?
     Security Key Lifecycle Manager (SKLM) as the key management server and with a certificate signed
     by a certificate authority (CA) on the KMIP port of the SKLM server.

mmnetverify command: Enhancements

     - Verify the network operation of nodes in a subnet
         * With the --subnets parameters, you can specify the subnet
           addresses of the nodes that you want to verify.
     - Verify that nodes can handle a new MTU size
         * With the ping-packet-size parameter, you can specify the size
           of the ICMP echo request packets that are sent between local node and the target
           node during the ping test.

mmtracectl command: Display the tracing status

     The --status parameter displays the tracing status of the specified nodes.

New feature for threshold monitoring

      Starting from IBM Spectrum Scale version 5.0.0, if multiple thresholds rules
      have overlapping entities for the same metrics, only one of the concurrent rules is
      made actively eligible.

NFS: Dynamic export changes

      You can dynamically change the export configuration without restarting the NFS service.

Object

      - Support for Ubuntu
      - Support for sudo wrapper for Object on Ubuntu
      - Support for cumulative upgrades from older versions
      - Object snap enhancement to contain keystore logs

Protocol support: Enhanced

      Protocol support is extended to Ubuntu 10.04

Setting up a system for storing crash files for Ubuntu

      A topic is added to describe how to set up a system for storing crash files for Ubuntu.

SMB: DFS redirects for SMB shares

      New option to configure DFS redirects for SMB shares.

SMB: SMB server upgrade changes

      Two events on CTDB version match/mismatch are added to the RAS events.

Sudo wrappers: Root-level processes can call administration commands directly

      Root-level background processes, such as cron and callback programs, can
      successfully call administration commands directly rather than through sudo
      when sudo wrappers are enabled.

Supported clients for NFS

      A topic is added listing the clients that are supported by NFS protocol.

Transparent cloud tiering

      - Support for multiple cloud storage accounts
      - Support for multiple file systems or filesets per node group
      - Enhanced support for large file systems provided by container spillover
      - Support for associating file sets with containers for enhanced granularity
      - Multiple URL and region support at the node level
      - Support for creating a cloud service separately for tiering and sharing operations.
      - Unique encryption key per cloud container
      - Support for remotely mounted clients.
      - Support for Amazon S3 regions requiring Sigv4 security support,
        including the US government cloud region.
      - Ability to enable or disable transparent recall for files for a given file
        system instantly, without having to rewrite a policy.
      - Support for backing up and restoring the Cloud services configuration in case of any disaster.
      - Support for backing up the Cloud services database to the cloud.
      - Support for restoring Transparent cloud tiering service on an identical backup cluster.
      - Support for checking the integrity of the Cloud services
        database after any system crash or outage.
      - Support for auditing events relating to each operation performed in Cloud services.

New commands

      mmaudit mmmsgqueue

Changed commands

      mmafmctl mmadddisk mmcallhome mmchattr mmchcluster mmchconfig mmchfs mmchnsd
      mmcloudgateway mmcrcluster mmcrfs mmdeldisk mmdsh mmfsck mmgetstate
      mmkeyserv mmnetverify mmnfs mmrestripefile mmrestripefs mmsmb mmtracectl
      mmuserauth

Deleted commands

      mmrest

New messages

      6027-1264, 6027-1757, 6027-2394, 6027-2395, 6027-2396, 6027-2397
      6027-2398, 6027-2399, 6027-2400, 6027-2401, 6027-3259, 6027-3408
      6027-3597, 6027-3598, 6027-3599, 6027-3600, 6027-3601, 6027-3602
      6027-3603, 6027-3604, 6027-3730, 6027-3921, 6027-3922, 6027-3923
      6027-3924, 6027-3925, 6027-3926, 6027-3927, 6027-3928
      6027-3929, 6027-3930, 6027-3931, 6027-4019

Changed messages

      6027-928


     Open this item<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab-234019da9b-2D248a-2D49ec-2D9503-2De612becd953e&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=Q6E0fg8sor2mORS3ugkurXkuV2ZUDqRd_C6_jpYXVVM&e=>


Posting Date:

Monday, December 18, 2017 at 2:26:38 PM EST



To unsubscribe or change settings, please go to your developerWorks community Settings.<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_news&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=5C4z0hhuaas4Ghs1g4YfFTCE0eyNYJVmLPYrGPCuE7Q&e=>
This is a notification sent from developerWorks community.

The information contained in this e-mail message is intended only for the use of the named recipients and is subject to the IBM Online Privacy Statement<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ibm.com_privacy_details_us_en_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=7gE9uoypP_yHb1jMMSjdCN_xyJ7UU2Ifrx1TchwMkQA&e=>. If you have received this e-mail in error, please contact the sender to report the error and then delete this message from your system.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/ced8c773/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 289 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/ced8c773/attachment-0001.png>

From Robert.Oesterlin at nuance.com  Tue Dec 19 12:38:12 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 19 Dec 2017 12:38:12 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
Message-ID: <73C34DA5-36DF-48C4-9E60-4DE75AA92405@nuance.com>

IBM corrected me a bit after I posted ? the cluster can be at any 4.2.X level (minreleaselevel)


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Sobey, Richard A" <r.sobey at imperial.ac.uk>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, December 19, 2017 at 3:07 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central

Hi Robert

Do you mean the minReleaseLevel from mmlsconfig or just making sure all the nodes are running 4.2.3?

Cheers!
Richard


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/c8a838b8/attachment-0001.htm>

From Robert.Oesterlin at nuance.com  Tue Dec 19 12:49:40 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 19 Dec 2017 12:49:40 +0000
Subject: [gpfsug-discuss] Scale 5.0 - Linux supported levels
Message-ID: <6844F00F-F07D-4FEF-B42D-CD32179A057E@nuance.com>

It?s a major problem for me that the dropped RH 6.X support with Scale 5.0. Now, I need to get my clients all to RH7 (which is problematic due to some older code needing the 6.X level) before I can take full advantage of Scale 5.0.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]" <aaron.s.knister at nasa.gov>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, December 19, 2017 at 2:32 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35

It?s not supported on SLES11 either.


IBM didn?t (that I saw) talk much about this publicly or give customers a chance to provide feedback about the decision. I know it was raised at the UG in NY and I recall a number of people saying it would be a significant issue for them (myself included) as is the fact they no longer support Debian with scale 5.0.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/66dbccf7/attachment-0001.htm>

From duersch at us.ibm.com  Tue Dec 19 13:18:08 2017
From: duersch at us.ibm.com (Steve Duersch)
Date: Tue, 19 Dec 2017 08:18:08 -0500
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
In-Reply-To: <mailman.3390.1513674371.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3390.1513674371.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <OF252F90AC.16B92D2A-ON852581FB.00486631-852581FB.00491295@notes.na.collabserv.com>


As Mike Taylor pointed out in a previous post this was an incorrect
statement.
You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a
rolling upgrade.
The minReleaseLevel is not pertinent to a rolling upgrade.  The running
daemon is the important part.  So you can't have any 4.1.x nodes in your
cluster and do a rolling upgrade to 5.0.

Also, Aaron, as to the OS support.  This decision was not made without some
angst.  As I mentioned at the user group meeting in NYC...the key point is
that we would like to get to a more current compiler.  This will allow us
to take advantage of newer features and functions and hopefully make the
code better for customers.  SLES 12 has been around for over 2 years.

I hope this helps give some thinking behind the decision.


Steve Duersch
Spectrum Scale
845-433-7902
IBM Poughkeepsie, New York


> Today's Topics:
>
>    1. Re: Spectrum Scale 5.0 now available on Fix Central
>       (Sobey, Richard A)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 19 Dec 2017 09:06:08 +0000
> From: "Sobey, Richard A" <r.sobey at imperial.ac.uk>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix
>    Central
> Message-ID:
>
>
<DB6PR0601MB2101BA78601B08F6F06469F8DF0F0 at DB6PR0601MB2101.eurprd06.prod.outlook.com>

>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Robert
>
> Do you mean the minReleaseLevel from mmlsconfig or just making sure
> all the nodes are running 4.2.3?
>
> Cheers!
> Richard
>
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-
> discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert
> Sent: 18 December 2017 19:44
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
Central
>
> The Scale 5.0 fix level is now up on Fix Central.
>
> You need to be at Scale 4.2.3 (cluster level) to do a rolling
> upgrade to this level.
>
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/07b56fa1/attachment-0001.htm>

From Robert.Oesterlin at nuance.com  Tue Dec 19 13:41:50 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 19 Dec 2017 13:41:50 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
Message-ID: <1A2E660D-A3F6-410E-9504-4B3BB5A34B09@nuance.com>

Thanks for clearing that up!


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Steve Duersch <duersch at us.ibm.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, December 19, 2017 at 7:20 AM
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central

As Mike Taylor pointed out in a previous post this was an incorrect statement.
You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a rolling upgrade.
The minReleaseLevel is not pertinent to a rolling upgrade. The running daemon is the important part. So you can't have any 4.1.x nodes in your cluster and do a rolling upgrade to 5.0.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/ea1c3e28/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Tue Dec 19 13:55:34 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Tue, 19 Dec 2017 13:55:34 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
In-Reply-To: <OF252F90AC.16B92D2A-ON852581FB.00486631-852581FB.00491295@notes.na.collabserv.com>
References: <mailman.3390.1513674371.19082.gpfsug-discuss@spectrumscale.org>
	<OF252F90AC.16B92D2A-ON852581FB.00486631-852581FB.00491295@notes.na.collabserv.com>
Message-ID: <591BE98E-9C39-41D4-BAEA-DCFF53623E01@bham.ac.uk>

Maybe it would have been a good idea to make this clear in the ?What?s new in 5.0? slide decks used at SC. I don?t recall it being there. And the lack of forward public notification on this is not great, particularly for those not in NYC. Sure most of my clusters are on EL7 now, but I still have some nodes still running 6.x (notably some of our Spectrum Protect nodes which are not just systems we can reinstall).

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "duersch at us.ibm.com" <duersch at us.ibm.com>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, 19 December 2017 at 13:19
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central


As Mike Taylor pointed out in a previous post this was an incorrect statement.
You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a rolling upgrade.
The minReleaseLevel is not pertinent to a rolling upgrade. The running daemon is the important part. So you can't have any 4.1.x nodes in your cluster and do a rolling upgrade to 5.0.

Also, Aaron, as to the OS support. This decision was not made without some angst. As I mentioned at the user group meeting in NYC...the key point is that we would like to get to a more current compiler. This will allow us to take advantage of newer features and functions and hopefully make the code better for customers. SLES 12 has been around for over 2 years.

I hope this helps give some thinking behind the decision.


Steve Duersch
Spectrum Scale
845-433-7902
IBM Poughkeepsie, New York


> Today's Topics:
>
>    1. Re: Spectrum Scale 5.0 now available on Fix Central
>       (Sobey, Richard A)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 19 Dec 2017 09:06:08 +0000
> From: "Sobey, Richard A" <r.sobey at imperial.ac.uk>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix
>    Central
> Message-ID:
>
> <DB6PR0601MB2101BA78601B08F6F06469F8DF0F0 at DB6PR0601MB2101.eurprd06.prod.outlook.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Robert
>
> Do you mean the minReleaseLevel from mmlsconfig or just making sure
> all the nodes are running 4.2.3?
>
> Cheers!
> Richard
>
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-
> discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert
> Sent: 18 December 2017 19:44
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central
>
> The Scale 5.0 fix level is now up on Fix Central.
>
> You need to be at Scale 4.2.3 (cluster level) to do a rolling
> upgrade to this level.
>
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/3894c70b/attachment-0001.htm>

From Mark.Bush at siriuscom.com  Tue Dec 19 14:29:21 2017
From: Mark.Bush at siriuscom.com (Mark Bush)
Date: Tue, 19 Dec 2017 14:29:21 +0000
Subject: [gpfsug-discuss] pmcollector and NSD perf
Message-ID: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>

I've noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in the GUI on the monitoring screen with the default view the NSD Server Throughput graph shows "Performance Collector did not return any data".  I've seen that in other items (SMB before for example) but never for NSD.  Is there something that must be enabled in the zimon sensor or collector config file to grab this or is this a bug?


Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/d814d122/attachment-0001.htm>

From rohwedder at de.ibm.com  Tue Dec 19 15:23:55 2017
From: rohwedder at de.ibm.com (Markus Rohwedder)
Date: Tue, 19 Dec 2017 16:23:55 +0100
Subject: [gpfsug-discuss] pmcollector and NSD perf
In-Reply-To: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
References: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
Message-ID: <OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>

Hello Mark,

the NSD sensor is GPFSNSDDisk.

Some things to check:

1. Is the sensor activated?
In a GPFS managed sensor config you should be able to see something like
this when you call mmperfmon config show:

{
        name = "GPFSNSDDisk"
        period = 10
        restrict = "nsdNodes"
},

2. Perfmon designation
The NSD server nodes should have the perfmon designation.


[root at cache-41 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfsgui-cluster-4.localnet.com
  GPFS cluster id:           10583479681538672379
  GPFS UID domain:           localnet.com
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

 Node  Daemon node name       IP address   Admin node name
Designation
------------------------------------------------------------------------------
   1   cache-41.localnet.com  10.0.100.41  cache-41.localnet.com
quorum-perfmon
   2   cache-42.localnet.com  10.0.100.42  cache-42.localnet.com
quorum-gateway-perfmon
   3   cache-43.localnet.com  10.0.100.43  cache-43.localnet.com
gateway-perfmon

3. Direct Disk writes?
One reason why there may be no data on your system is if you are not using
the NSD protocol,
meaning the clients can directly write to disk as in a SAN environment.
In this case the sensor does not catch the transactions.

4. Cross cluster mount
Or maybe you are using a cross cluster mount.



Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

Spectrum Scale GUI Development
                                                                              
                                                                              
                                                                              
                                                                              
                                                                              
 Phone:            +49 7034 6430190      IBM Deutschland                      
                                                                              
 E-Mail:           rohwedder at de.ibm.com  Am Weiher 24                         
                                                                              
                                         65451 Kelsterbach                    
                                                                              
                                         Germany                              
                                                                              
                                                                              
                                                                              
                                                                              
                                                                              
 IBM Deutschland                                                              
 Research &                                                                   
 Development                                                                  
 GmbH /                                                                       
 Vorsitzender des                                                             
 Aufsichtsrats:                                                               
 Martina K?deritz                                                             
 Gesch?ftsf?hrung:                                                            
 Dirk Wittkopp                                                                
 Sitz der                                                                     
 Gesellschaft:                                                                
 B?blingen /                                                                  
 Registergericht:                                                             
 Amtsgericht                                                                  
 Stuttgart, HRB                                                               
 243294                                                                       
                                                                              





From:	Mark Bush <Mark.Bush at siriuscom.com>
To:	"gpfsug-discuss at spectrumscale.org"
            <gpfsug-discuss at spectrumscale.org>
Date:	12/19/2017 03:30 PM
Subject:	[gpfsug-discuss] pmcollector and NSD perf
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



I?ve noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in
the GUI on the monitoring screen with the default view the NSD Server
Throughput graph shows ?Performance Collector did not return any data?.
I?ve seen that in other items (SMB before for example) but never for NSD.
Is there something that must be enabled in the zimon sensor or collector
config file to grab this or is this a bug?


Mark_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=a6GCq72qeADy6hsfA-24PmWHU06W5z2xqx9tKIJ8qJ4&s=OQccy8ikWB-ByYgLsJFgI8szDs1ZrwnsaFrLCwTfTwI&e=



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment-0003.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1A013420.gif
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment-0004.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment-0005.gif>

From Mark.Bush at siriuscom.com  Tue Dec 19 15:49:59 2017
From: Mark.Bush at siriuscom.com (Mark Bush)
Date: Tue, 19 Dec 2017 15:49:59 +0000
Subject: [gpfsug-discuss] pmcollector and NSD perf
In-Reply-To: <OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>
References: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
	<OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>
Message-ID: <DM5PR05MB3577119DC6FE73462E83BC72E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>

It appears number 3 on your list is the case.  My nodes are all SAN connected and until I get separate CES nodes no NSD is necessary (currently run CES on the NSD servers ? just for a test cluster).

Mark

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Markus Rohwedder
Sent: Tuesday, December 19, 2017 9:24 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] pmcollector and NSD perf


Hello Mark,

the NSD sensor is GPFSNSDDisk.

Some things to check:

1. Is the sensor activated?
In a GPFS managed sensor config you should be able to see something like this when you call mmperfmon config show:

{
name = "GPFSNSDDisk"
period = 10
restrict = "nsdNodes"
},

2. Perfmon designation
The NSD server nodes should have the perfmon designation.


[root at cache-41 ~]# mmlscluster

GPFS cluster information
========================
GPFS cluster name: gpfsgui-cluster-4.localnet.com
GPFS cluster id: 10583479681538672379
GPFS UID domain: localnet.com
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCR

Node Daemon node name IP address Admin node name Designation
------------------------------------------------------------------------------
1 cache-41.localnet.com 10.0.100.41 cache-41.localnet.com quorum-perfmon
2 cache-42.localnet.com 10.0.100.42 cache-42.localnet.com quorum-gateway-perfmon
3 cache-43.localnet.com 10.0.100.43 cache-43.localnet.com gateway-perfmon

3. Direct Disk writes?
One reason why there may be no data on your system is if you are not using the NSD protocol,
meaning the clients can directly write to disk as in a SAN environment.
In this case the sensor does not catch the transactions.

4. Cross cluster mount
Or maybe you are using a cross cluster mount.



Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

Spectrum Scale GUI Development

________________________________



Phone:

+49 7034 6430190

IBM Deutschland

[cid:image002.gif at 01D378AE.BAB5B6F0]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________



IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina K?deritz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294



[Inactive hide details for Mark Bush ---12/19/2017 03:30:14 PM---I've noticed this in my test cluster both in 4.2.3.4 and 5.0.0.]Mark Bush ---12/19/2017 03:30:14 PM---I've noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in the GUI on the monitoring s

From: Mark Bush <Mark.Bush at siriuscom.com<mailto:Mark.Bush at siriuscom.com>>
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: 12/19/2017 03:30 PM
Subject: [gpfsug-discuss] pmcollector and NSD perf
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



I?ve noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in the GUI on the monitoring screen with the default view the NSD Server Throughput graph shows ?Performance Collector did not return any data?. I?ve seen that in other items (SMB before for example) but never for NSD. Is there something that must be enabled in the zimon sensor or collector config file to grab this or is this a bug?


Mark_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=a6GCq72qeADy6hsfA-24PmWHU06W5z2xqx9tKIJ8qJ4&s=OQccy8ikWB-ByYgLsJFgI8szDs1ZrwnsaFrLCwTfTwI&e=


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 166 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.gif
Type: image/gif
Size: 1851 bytes
Desc: image002.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.gif
Type: image/gif
Size: 105 bytes
Desc: image003.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment-0003.gif>

From hoov at us.ibm.com  Tue Dec 19 18:40:56 2017
From: hoov at us.ibm.com (Theodore Hoover Jr)
Date: Tue, 19 Dec 2017 18:40:56 +0000
Subject: [gpfsug-discuss] Spectrum Scale on AWS - Join Sponsor User Program
Message-ID: <OFA4E895DE.FCD6420A-ON002581FB.00665ACE-002581FB.00669FEE@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/98449b9b/attachment-0001.htm>

From oehmes at gmail.com  Tue Dec 19 20:20:29 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Tue, 19 Dec 2017 20:20:29 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
	<CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
	<6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>
Message-ID: <CALssuR3wX4_6T668BEupSvdt75pxgZuBSj5rzbVVy=tWWoFhxQ@mail.gmail.com>

Hi,

the zero padding was never promoted into a GA stream, it was an experiment
to proof we are on the right track when we eliminate the overhead from
client to NSD Server, but also showed that alone is not good enough. the
work for the client is the same compared to the >32 subblocks, but the NSD
Server has more work as it can't pack as many subblocks and therefore files
into larger blocks, so you need to do more writes to store the same number
of files.
thats why there is the additional substantial improvement  when we then
went to >32 subblocks.

sven

On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:

> Thanks Sven! That makes sense to me and is what I thought was the case
> which is why I was confused when I saw the reply to the thread that said
> the >32 subblocks code had no performance impact.
>
> A couple more question for you? in your presentation there?s a benchmark
> that shows the file create performance without the zero padding. Since you
> mention this is done for security reasons was that feature ever promoted to
> a GA Scale release? I?m also wondering if you could explain the performance
> difference between the no zero padding code and the > 32 subblock code
> since given your the example of 32K files and 16MB block size I figure both
> cases ought to write the same amount to disk.
>
> Thanks!
>
> -Aaron
>
>
>
>
>
> On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com> wrote:
>
> i thought i answered that already, but maybe i just thought about
> answering it and then forgot about it :-D
>
> so yes more than 32 subblocks per block significant increase the
> performance of filesystems with small files, for the sake of the argument
> let's say 32k in a large block filesystem again for sake of argument say
> 16MB.
>
> you probably ask why ?
>
> if you create a file and write 32k into it in a pre 5.0.0 Version 16 MB
> filesystem your client actually doesn't write 32k to the NSD Server, it
> writes 512k, because thats the subblock size and we need to write the full
> subblock (for security reasons). so first you waste significant memory on
> the client to cache that zero padding, you waste network bandwidth and you
> waste NSD Server cache because you store it there too. this means you
> overrun the cache more quickly, means you start doing read/modify writes
> earlier on all your nice large raid tracks... i guess you get the story by
> now.
>
> in fact,  if you have a good raid code that can drive really a lot of
> bandwidth out of individual drives like a GNR system you get more
> performance for small file writes as larger your blocksize is, because we
> can 'pack' more files into larger i/os and therefore turn a small file
> create workload into a bandwidth workload, essentially exactly what we did
> and i demonstrated in the CORAL presentation .
>
> hope that makes this crystal clear now .
>
> sven
>
>
>
> On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister <aaron.s.knister at nasa.gov>
> wrote:
>
> Thanks, Alex. I'm all too familiar with the trade offs between large
> blocks and small files and we do use pretty robust SSD storage for our
> metadata. We support a wide range of workloads and we have some folks
> with many small (<1M) files and other folks with many large (>256MB) files.
>
> My point in this thread is that IBM has said over and over again in
> presentations that there is a significant performance gain with the >32
> subblocks code on filesystems with large block sizes (although to your
> point I'm not clear on exactly what large means since I didn't define
> large in this context). Therefore given that the >32 subblock code gives
> a significant performance gain one could reasonably assume that having a
> filesystem with >32 subblocks is required to see this gain (rather than
> just running the >32 subblocks code on an fs w/o > 32 subblocks).
>
> This lead me to ask about a migration tool because in my mind if there's
> a performance gain from having >32 subblocks on the FS I'd like that
> feature and having to manually copy 10's of PB to new hardware to get
> this performance boost is unacceptable. However, IBM can't seem to make
> up their mind about whether or not the >32 subblocks code *actually*
> provides a performance increase. This seems like a pretty
> straightforward question.
>
> -Aaron
>
> On 12/15/17 3:48 PM, Alex Chekholko wrote:
> > Hey Aaron,
> >
> > Can you define your sizes for "large blocks" and "small files"?  If you
> > dial one up and the other down, your performance will be worse.  And in
> > any case it's a pathological corner case so it shouldn't matter much for
> > your workflow, unless you've designed your system with the wrong values.
> >
> > For example, for bioinformatics workloads, I prefer to use 256KB
> > filesystem block size, and I'd consider 4MB+ to be "large block size",
> > which would make the filesystem obviously unsuitable for processing
> > millions of 8KB files.
> >
> > You can make a histogram of file sizes in your existing filesystems and
> > then make your subblock size (1/32 of block size) on the smaller end of
> > that.   Also definitely use the "small file in inode" feature and put
> > your metadata on SSD.
> >
> > Regards,
> > Alex
> >
> > On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> > <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
> >
> >     Thanks, Bill.
> >
> >     I still don't feel like I've got an clear answer from IBM and frankly
> >     the core issue of a lack of migration tool was totally dodged.
> >
> >     Again in Sven's presentation from SSUG @ SC17
> >     (
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
> >     he mentions "It has a significant performance penalty for small
> files in
> >     large block size filesystems" and the demonstrates that with several
> >     mdtest runs (which show the effect with and without the >32
> >     subblocks code):
> >
> >
> >     4.2.1 base code - SUMMARY: (of 3 iterations)
> >     File creation : Mean = 2237.644
> >
> >     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY:
> (of
> >     3 iterations)
> >     File creation : Mean = 12866.842
> >
> >     more sub blocks per block (4.2.2 + morethan32subblock code):
> >     File creation : Mean = 40316.721
> >
> >     Can someone (ideally Sven) give me a straight answer as to whether or
> >     not the > 32 subblock code actually makes a performance difference
> for
> >     small files in large block filesystems? And if not, help me
> understand
> >     why his slides and provided benchmark data have consistently
> indicated
> >     it does?
> >
> >     -Aaron
> >
> >     On 12/1/17 11:44 AM, Bill Hartner wrote:
> >     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> >     > results @ user group meeting.
> >     >
> >     > -Bill
> >     >
> >     > Bill Hartner
> >     > IBM Systems
> >     > Scalable I/O Development
> >     > Austin, Texas
> >     > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
> >     > home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >
> >     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017
> 06:53:44
> >     > AM---Bill, could you say something about what the metadataJan-Frode
> >     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> >     > about what the metadata-storage here was? ESS/NL-SAS/3way
> replication?
> >     >
> >     > From: Jan-Frode Myklebust <janfrode at tanso.net <mailto:
> janfrode at tanso.net>>
> >     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>
> >     > Date: 12/01/2017 06:53 AM
> >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     > Sent by: gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> >     >
> >     >
> >
>  ------------------------------------------------------------------------
> >     >
> >     >
> >     >
> >     > Bill, could you say something about what the metadata-storage here
> was?
> >     > ESS/NL-SAS/3way replication?
> >     >
> >     > I just asked about this in the internal slack channel #scale-help
> today..
> >     >
> >     >
> >     >
> >     > -jf
> >     >
> >     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
> <_bhartner at us.ibm.com_
> >     > <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>:
> >     >
> >     >     > "It has a significant performance penalty for small files in
> >     large
> >     >     > block size filesystems"
> >     >
> >     >     Aaron,
> >     >
> >     >     Below are mdtest results for a test we ran for CORAL - file
> >     size was
> >     >     32k.
> >     >
> >     >     We have not gone back and ran the test on a file system
> formatted
> >     >     without > 32 subblocks. We'll do that at some point...
> >     >
> >     >     -Bill
> >     >
> >     >     -- started at 10/28/2017 17:51:38 --
> >     >
> >     >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> >     >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
> >     >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
> >     -p 360
> >     >     -u -y
> >     >     Path: /ibm/fs2-16m-10
> >     >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> >     >
> >     >     228 tasks, 67239936 files
> >     >
> >     >     SUMMARY: (of 3 iterations)
> >     >     Operation Max Min Mean Std Dev
> >     >     --------- --- --- ---- -------
> >     >     File creation : 51953.498 50558.517 51423.221 616.643
> >     >     File stat : 0.000 0.000 0.000 0.000
> >     >     File read : 0.000 0.000 0.000 0.000
> >     >     File removal : 96746.376 92149.535 94658.774 1900.187
> >     >     Tree creation : 1.588 0.070 0.599 0.700
> >     >     Tree removal : 0.213 0.034 0.097 0.082
> >     >
> >     >     -- finished at 10/28/2017 19:51:54 --
> >     >
> >     >     Bill Hartner
> >     >     IBM Systems
> >     >     Scalable I/O Development
> >     >     Austin, Texas_
> >     >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
> >     <mailto:bhartner at us.ibm.com>>
> >     >     home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >     _
> >     >     __gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>> wrote on
> >     >     11/29/2017 04:41:48 PM:
> >     >
> >     >     > From: Aaron Knister <_aaron.knister at gmail.com_
> >     >     <mailto:aaron.knister at gmail.com <mailto:
> aaron.knister at gmail.com>>>
> >     >
> >     >
> >     >     > To: gpfsug main discussion list
> >     >     <_gpfsug-discuss at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>>
> >     >
> >     >     > Date: 11/29/2017 04:42 PM
> >     >
> >     >
> >     >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
> >     >
> >     >     >
> >     >
> >     >     > Thanks, Nikhil. Most of that was consistent with my
> understnading,
> >     >     > however I was under the impression that the >32 subblocks
> code is
> >     >     > required to achieve the touted 50k file creates/second that
> Sven has
> >     >     > talked about a bunch of times:
> >     >     >
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >     <
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >>
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
> >     <
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >>
> >     >     > _Sven_Oehme_-_News_from_Research.pdf
> >     >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
> >     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >>
> >     >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> >     >
> >     >
> >     >     > from those presentations regarding 32 subblocks:
> >     >     >
> >     >     > "It has a significant performance penalty for small files in
> large
> >     >     > block size filesystems"
> >     >
> >     >     > although I'm not clear on the specific definition of
> "large". Many
> >     >     > filesystems I encounter only have a 1M block size so it may
> not
> >     >     > matter there, although that same presentation clearly shows
> the
> >     >     > benefit of larger block sizes which is yet *another* thing
> for which
> >     >     > a migration tool would be helpful.
> >     >
> >     >     > -Aaron
> >     >     >
> >     >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
> >     >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
> >     <mailto:nikhilk at us.ibm.com>>> wrote:
> >     >
> >     >     > Hi,
> >     >     >
> >     >     > I would like to clarify migration path to 5.0.0 from 4.X.X
> >     clusters.
> >     >     > For all Spectrum Scale clusters that are currently at 4.X.X,
> >     it is
> >     >     > possible to migrate to 5.0.0 with no offline data migration
> >     and no
> >     >     > need to move data. Once these clusters are at 5.0.0, they
> will
> >     >     > benefit from the performance improvements, new features
> (such as
> >     >     > file audit logging), and various enhancements that are
> >     included in
> >     >     5.0.0.
> >     >     >
> >     >     > That being said, there is one enhancement that will not be
> >     applied
> >     >     > to these clusters, and that is the increased number of
> >     sub-blocks
> >     >     > per block for small file allocation. This means that for file
> >     >     > systems with a large block size and a lot of small files, the
> >     >     > overall space utilization will be the same it currently is
> >     in 4.X.X.
> >     >     > Since file systems created at 4.X.X and earlier used a block
> >     size
> >     >     > that kept this allocation in mind, there should be very
> little
> >     >     > impact on existing file systems.
> >     >     >
> >     >     > Outside of that one particular function, the remainder of the
> >     >     > performance improvements, metadata improvements, updated
> >     >     > compatibility, new functionality, and all of the other
> >     enhancements
> >     >     > will be immediately available to you once you complete the
> >     upgrade
> >     >     > to 5.0.0 -- with no need to reformat, move data, or take
> >     your data
> >     >     offline.
> >     >     >
> >     >     > I hope that clarifies things a little and makes the upgrade
> path
> >     >     > more accessible.
> >     >     >
> >     >     > Please let me know if there are any other questions or
> concerns.
> >     >     >
> >     >     > Thank you,
> >     >     > Nikhil Khandelwal
> >     >     > Spectrum Scale Development
> >     >     > Client Adoption
> >     >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>
> >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >
> >     >     > _https://urldefense.proofpoint.com/v2/url?_
> >     <https://urldefense.proofpoint.com/v2/url?_>
> >     >     >
> >     >
>  u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
> >     >     >
> >     >
>  siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> >     >
> >     >     _______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>_
> >     >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>_______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >
> >     >
> >     >
> >     >
> >     >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >     >
> >
> >     --
> >     Aaron Knister
> >     NASA Center for Climate Simulation (Code 606.2)
> >     Goddard Space Flight Center
> >     (301) 286-2776 <tel:%28301%29%20286-2776>
> >     _______________________________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/92903846/attachment-0001.htm>

From aaron.s.knister at nasa.gov  Tue Dec 19 21:32:00 2017
From: aaron.s.knister at nasa.gov (Aaron Knister)
Date: Tue, 19 Dec 2017 16:32:00 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CALssuR3wX4_6T668BEupSvdt75pxgZuBSj5rzbVVy=tWWoFhxQ@mail.gmail.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
	<CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
	<6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>
	<CALssuR3wX4_6T668BEupSvdt75pxgZuBSj5rzbVVy=tWWoFhxQ@mail.gmail.com>
Message-ID: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>

Thanks, Sven. Understood!

On 12/19/17 3:20 PM, Sven Oehme wrote:
> Hi,
> 
> the zero padding was never promoted into a GA stream, it was an
> experiment to proof we are on the right track when we eliminate the
> overhead from client to NSD Server, but also showed that alone is not
> good enough. the work for the client is the same compared to the >32
> subblocks, but the NSD Server has more work as it can't pack as many
> subblocks and therefore files into larger blocks, so you need to do more
> writes to store the same number of files.?
> thats why there is the additional substantial improvement? when we then
> went to >32 subblocks.?
> 
> sven
> 
> On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
> SCIENCE CORP] <aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>> wrote:
> 
>     Thanks Sven! That makes sense to me and is what I thought was the
>     case which is why I was confused when I saw the reply to the thread
>     that said the >32 subblocks code had no performance impact.?
> 
>     A couple more question for you? in your presentation there?s a
>     benchmark that shows the file create performance without the zero
>     padding. Since you mention this is done for security reasons was
>     that feature ever promoted to a GA Scale release? I?m also wondering
>     if you could explain the performance difference between the no zero
>     padding code and the > 32 subblock code since given your the example
>     of 32K files and 16MB block size I figure both cases ought to write
>     the same amount to disk.?
> 
>     Thanks!
> 
>     -Aaron
> 
> 
> 
> 
> 
>     On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com
>     <mailto:oehmes at gmail.com>> wrote:
>>     i thought i answered that already, but maybe i just thought about
>>     answering it and then forgot about it :-D
>>
>>     so yes more than 32 subblocks per block significant increase the
>>     performance of filesystems with small files, for the sake of the
>>     argument let's say 32k in a large block filesystem again for sake
>>     of argument say 16MB.?
>>
>>     you probably ask why ??
>>
>>     if you create a file and write 32k into it in a pre 5.0.0 Version
>>     16 MB filesystem your client actually doesn't write 32k to the NSD
>>     Server, it writes 512k, because thats the subblock size and we
>>     need to write the full subblock (for security reasons). so first
>>     you waste significant memory on the client to cache that zero
>>     padding, you waste network bandwidth and you waste NSD Server
>>     cache because you store it there too. this means you overrun the
>>     cache more quickly, means you start doing read/modify writes
>>     earlier on all your nice large raid tracks... i guess you get the
>>     story by now.?
>>
>>     in fact,? if you have a good raid code that can drive really a lot
>>     of bandwidth out of individual drives like a GNR system you get
>>     more performance for small file writes as larger your blocksize
>>     is, because we can 'pack' more files into larger i/os and
>>     therefore turn a small file create workload into a bandwidth
>>     workload, essentially exactly what we did and i demonstrated in
>>     the CORAL presentation .?
>>
>>     hope that makes this crystal clear now .?
>>
>>     sven
>>
>>
>>
>>     On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister
>>     <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
>>
>>         Thanks, Alex. I'm all too familiar with the trade offs between
>>         large
>>         blocks and small files and we do use pretty robust SSD storage
>>         for our
>>         metadata. We support a wide range of workloads and we have
>>         some folks
>>         with many small (<1M) files and other folks with many large
>>         (>256MB) files.
>>
>>         My point in this thread is that IBM has said over and over
>>         again in
>>         presentations that there is a significant performance gain
>>         with the >32
>>         subblocks code on filesystems with large block sizes (although
>>         to your
>>         point I'm not clear on exactly what large means since I didn't
>>         define
>>         large in this context). Therefore given that the >32 subblock
>>         code gives
>>         a significant performance gain one could reasonably assume
>>         that having a
>>         filesystem with >32 subblocks is required to see this gain
>>         (rather than
>>         just running the >32 subblocks code on an fs w/o > 32 subblocks).
>>
>>         This lead me to ask about a migration tool because in my mind
>>         if there's
>>         a performance gain from having >32 subblocks on the FS I'd
>>         like that
>>         feature and having to manually copy 10's of PB to new hardware
>>         to get
>>         this performance boost is unacceptable. However, IBM can't
>>         seem to make
>>         up their mind about whether or not the >32 subblocks code
>>         *actually*
>>         provides a performance increase. This seems like a pretty
>>         straightforward question.
>>
>>         -Aaron
>>
>>         On 12/15/17 3:48 PM, Alex Chekholko wrote:
>>         > Hey Aaron,
>>         >
>>         > Can you define your sizes for "large blocks" and "small
>>         files"?? If you
>>         > dial one up and the other down, your performance will be
>>         worse.? And in
>>         > any case it's a pathological corner case so it shouldn't
>>         matter much for
>>         > your workflow, unless you've designed your system with the
>>         wrong values.
>>         >
>>         > For example, for bioinformatics workloads, I prefer to use 256KB
>>         > filesystem block size, and I'd consider 4MB+ to be "large
>>         block size",
>>         > which would make the filesystem obviously unsuitable for
>>         processing
>>         > millions of 8KB files.
>>         >
>>         > You can make a histogram of file sizes in your existing
>>         filesystems and
>>         > then make your subblock size (1/32 of block size) on the
>>         smaller end of
>>         > that.? ?Also definitely use the "small file in inode"
>>         feature and put
>>         > your metadata on SSD.
>>         >
>>         > Regards,
>>         > Alex
>>         >
>>         > On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
>>         > <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>
>>         <mailto:aaron.s.knister at nasa.gov
>>         <mailto:aaron.s.knister at nasa.gov>>> wrote:
>>         >
>>         >? ? ?Thanks, Bill.
>>         >
>>         >? ? ?I still don't feel like I've got an clear answer from
>>         IBM and frankly
>>         >? ? ?the core issue of a lack of migration tool was totally
>>         dodged.
>>         >
>>         >? ? ?Again in Sven's presentation from SSUG @ SC17
>>         >? ?
>>         ?(http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf
>>         <http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
>>         >? ? ?he mentions "It has a significant performance penalty
>>         for small files in
>>         >? ? ?large block size filesystems" and the demonstrates that
>>         with several
>>         >? ? ?mdtest runs (which show the effect with and without the >32
>>         >? ? ?subblocks code):
>>         >
>>         >
>>         >? ? ?4.2.1 base code - SUMMARY: (of 3 iterations)
>>         >? ? ?File creation : Mean = 2237.644
>>         >
>>         >? ? ?zero-end-of-file-padding (4.2.2 + ifdef for zero
>>         padding):? SUMMARY: (of
>>         >? ? ?3 iterations)
>>         >? ? ?File creation : Mean = 12866.842
>>         >
>>         >? ? ?more sub blocks per block (4.2.2 + morethan32subblock code):
>>         >? ? ?File creation : Mean = 40316.721
>>         >
>>         >? ? ?Can someone (ideally Sven) give me a straight answer as
>>         to whether or
>>         >? ? ?not the > 32 subblock code actually makes a performance
>>         difference for
>>         >? ? ?small files in large block filesystems? And if not, help
>>         me understand
>>         >? ? ?why his slides and provided benchmark data have
>>         consistently indicated
>>         >? ? ?it does?
>>         >
>>         >? ? ?-Aaron
>>         >
>>         >? ? ?On 12/1/17 11:44 AM, Bill Hartner wrote:
>>         >? ? ?> ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported
>>         some of the
>>         >? ? ?> results @ user group meeting.
>>         >? ? ?>
>>         >? ? ?> -Bill
>>         >? ? ?>
>>         >? ? ?> Bill Hartner
>>         >? ? ?> IBM Systems
>>         >? ? ?> Scalable I/O Development
>>         >? ? ?> Austin, Texas
>>         >? ? ?> bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
>>         <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>
>>         >? ? ?> home office 512-784-0980 <tel:(512)%20784-0980>
>>         <tel:512-784-0980 <tel:(512)%20784-0980>>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> Inactive hide details for Jan-Frode Myklebust
>>         ---12/01/2017 06:53:44
>>         >? ? ?> AM---Bill, could you say something about what the
>>         metadataJan-Frode
>>         >? ? ?> Myklebust ---12/01/2017 06:53:44 AM---Bill, could you
>>         say something
>>         >? ? ?> about what the metadata-storage here was?
>>         ESS/NL-SAS/3way replication?
>>         >? ? ?>
>>         >? ? ?> From: Jan-Frode Myklebust <janfrode at tanso.net
>>         <mailto:janfrode at tanso.net> <mailto:janfrode at tanso.net
>>         <mailto:janfrode at tanso.net>>>
>>         >? ? ?> To: gpfsug main discussion list
>>         <gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>>>
>>         >? ? ?> Date: 12/01/2017 06:53 AM
>>         >? ? ?> Subject: Re: [gpfsug-discuss] Online data migration tool
>>         >? ? ?> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ?
>>         ?------------------------------------------------------------------------
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> Bill, could you say something about what the
>>         metadata-storage here was?
>>         >? ? ?> ESS/NL-SAS/3way replication?
>>         >? ? ?>
>>         >? ? ?> I just asked about this in the internal slack channel
>>         #scale-help today..
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> -jf
>>         >? ? ?>
>>         >? ? ?> fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
>>         <_bhartner at us.ibm.com_
>>         >? ? ?> <mailto:bhartner at us.ibm.com
>>         <mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com
>>         <mailto:bhartner at us.ibm.com>>>>:
>>         >? ? ?>
>>         >? ? ?>? ? ?> "It has a significant performance penalty for
>>         small files in
>>         >? ? ?large
>>         >? ? ?>? ? ?> block size filesystems"
>>         >? ? ?>
>>         >? ? ?>? ? ?Aaron,
>>         >? ? ?>
>>         >? ? ?>? ? ?Below are mdtest results for a test we ran for
>>         CORAL - file
>>         >? ? ?size was
>>         >? ? ?>? ? ?32k.
>>         >? ? ?>
>>         >? ? ?>? ? ?We have not gone back and ran the test on a file
>>         system formatted
>>         >? ? ?>? ? ?without > 32 subblocks. We'll do that at some point...
>>         >? ? ?>
>>         >? ? ?>? ? ?-Bill
>>         >? ? ?>
>>         >? ? ?>? ? ?-- started at 10/28/2017 17:51:38 --
>>         >? ? ?>
>>         >? ? ?>? ? ?mdtest-1.9.3 was launched with 228 total task(s)
>>         on 12 node(s)
>>         >? ? ?>? ? ?Command line used: /tmp/mdtest-binary-dir/mdtest -d
>>         >? ? ?>? ? ?/ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w
>>         32768 -C -F -r
>>         >? ? ?-p 360
>>         >? ? ?>? ? ?-u -y
>>         >? ? ?>? ? ?Path: /ibm/fs2-16m-10
>>         >? ? ?>? ? ?FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used
>>         Inodes: 0.0%
>>         >? ? ?>
>>         >? ? ?>? ? ?228 tasks, 67239936 files
>>         >? ? ?>
>>         >? ? ?>? ? ?SUMMARY: (of 3 iterations)
>>         >? ? ?>? ? ?Operation Max Min Mean Std Dev
>>         >? ? ?>? ? ?--------- --- --- ---- -------
>>         >? ? ?>? ? ?File creation : 51953.498 50558.517 51423.221 616.643
>>         >? ? ?>? ? ?File stat : 0.000 0.000 0.000 0.000
>>         >? ? ?>? ? ?File read : 0.000 0.000 0.000 0.000
>>         >? ? ?>? ? ?File removal : 96746.376 92149.535 94658.774 1900.187
>>         >? ? ?>? ? ?Tree creation : 1.588 0.070 0.599 0.700
>>         >? ? ?>? ? ?Tree removal : 0.213 0.034 0.097 0.082
>>         >? ? ?>
>>         >? ? ?>? ? ?-- finished at 10/28/2017 19:51:54 --
>>         >? ? ?>
>>         >? ? ?>? ? ?Bill Hartner
>>         >? ? ?>? ? ?IBM Systems
>>         >? ? ?>? ? ?Scalable I/O Development
>>         >? ? ?>? ? ?Austin, Texas_
>>         >? ? ?>? ? ?__bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
>>         <mailto:bhartner at us.ibm.com>
>>         >? ? ?<mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>
>>         >? ? ?>? ? ?home office 512-784-0980 <tel:(512)%20784-0980>
>>         <tel:512-784-0980 <tel:(512)%20784-0980>>
>>         >? ? ?>
>>         >? ? ?>? ? ?_
>>         >? ? ?>? ? ?__gpfsug-discuss-bounces at spectrumscale.org_
>>         >? ? ?>? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>>>?wrote on
>>         >? ? ?>? ? ?11/29/2017 04:41:48 PM:
>>         >? ? ?>
>>         >? ? ?>? ? ?> From: Aaron Knister <_aaron.knister at gmail.com_
>>         >? ? ?>? ? ?<mailto:aaron.knister at gmail.com
>>         <mailto:aaron.knister at gmail.com>
>>         <mailto:aaron.knister at gmail.com
>>         <mailto:aaron.knister at gmail.com>>>>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> To: gpfsug main discussion list
>>         >? ? ?>? ? ?<_gpfsug-discuss at spectrumscale.org_
>>         >? ? ?>? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>>>>
>>         >? ? ?>
>>         >? ? ?>? ? ?> Date: 11/29/2017 04:42 PM
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> Subject: Re: [gpfsug-discuss] Online data
>>         migration tool
>>         >? ? ?>? ? ?> Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>>         >? ? ?>? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>>>
>>         >? ? ?>
>>         >? ? ?>? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> Thanks, Nikhil. Most of that was consistent with
>>         my understnading,
>>         >? ? ?>? ? ?> however I was under the impression that the >32
>>         subblocks code is
>>         >? ? ?>? ? ?> required to achieve the touted 50k file
>>         creates/second that Sven has
>>         >? ? ?>? ? ?> talked about a bunch of times:
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         >? ?
>>         ??_http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>>         >? ?
>>         ?<http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         >? ?
>>         ??_http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
>>         >? ?
>>         ?<http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>>         >? ? ?>? ? ?> _Sven_Oehme_-_News_from_Research.pdf
>>         >? ? ?>? ? ?>
>>         _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>>         >? ? ?<http://files.gpfsug.org/presentations/2016/SC16/12_-_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>>         >? ? ?>? ? ?>
>>         _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> from those presentations regarding 32 subblocks:
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> "It has a significant performance penalty for
>>         small files in large
>>         >? ? ?>? ? ?> block size filesystems"
>>         >? ? ?>
>>         >? ? ?>? ? ?> although I'm not clear on the specific
>>         definition of "large". Many
>>         >? ? ?>? ? ?> filesystems I encounter only have a 1M block
>>         size so it may not
>>         >? ? ?>? ? ?> matter there, although that same presentation
>>         clearly shows the
>>         >? ? ?>? ? ?> benefit of larger block sizes which is yet
>>         *another* thing for which
>>         >? ? ?>? ? ?> a migration tool would be helpful.
>>         >? ? ?>
>>         >? ? ?>? ? ?> -Aaron
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>>         >? ? ?>? ? ?<_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
>>         <mailto:nikhilk at us.ibm.com>
>>         >? ? ?<mailto:nikhilk at us.ibm.com
>>         <mailto:nikhilk at us.ibm.com>>>> wrote:
>>         >? ? ?>
>>         >? ? ?>? ? ?> Hi,
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> I would like to clarify migration path to 5.0.0
>>         from 4.X.X
>>         >? ? ?clusters.
>>         >? ? ?>? ? ?> For all Spectrum Scale clusters that are
>>         currently at 4.X.X,
>>         >? ? ?it is
>>         >? ? ?>? ? ?> possible to migrate to 5.0.0 with no offline
>>         data migration
>>         >? ? ?and no
>>         >? ? ?>? ? ?> need to move data. Once these clusters are at
>>         5.0.0, they will
>>         >? ? ?>? ? ?> benefit from the performance improvements, new
>>         features (such as
>>         >? ? ?>? ? ?> file audit logging), and various enhancements
>>         that are
>>         >? ? ?included in
>>         >? ? ?>? ? ?5.0.0.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> That being said, there is one enhancement that
>>         will not be
>>         >? ? ?applied
>>         >? ? ?>? ? ?> to these clusters, and that is the increased
>>         number of
>>         >? ? ?sub-blocks
>>         >? ? ?>? ? ?> per block for small file allocation. This means
>>         that for file
>>         >? ? ?>? ? ?> systems with a large block size and a lot of
>>         small files, the
>>         >? ? ?>? ? ?> overall space utilization will be the same it
>>         currently is
>>         >? ? ?in 4.X.X.
>>         >? ? ?>? ? ?> Since file systems created at 4.X.X and earlier
>>         used a block
>>         >? ? ?size
>>         >? ? ?>? ? ?> that kept this allocation in mind, there should
>>         be very little
>>         >? ? ?>? ? ?> impact on existing file systems.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> Outside of that one particular function, the
>>         remainder of the
>>         >? ? ?>? ? ?> performance improvements, metadata improvements,
>>         updated
>>         >? ? ?>? ? ?> compatibility, new functionality, and all of the
>>         other
>>         >? ? ?enhancements
>>         >? ? ?>? ? ?> will be immediately available to you once you
>>         complete the
>>         >? ? ?upgrade
>>         >? ? ?>? ? ?> to 5.0.0 -- with no need to reformat, move data,
>>         or take
>>         >? ? ?your data
>>         >? ? ?>? ? ?offline.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> I hope that clarifies things a little and makes
>>         the upgrade path
>>         >? ? ?>? ? ?> more accessible.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> Please let me know if there are any other
>>         questions or concerns.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> Thank you,
>>         >? ? ?>? ? ?> Nikhil Khandelwal
>>         >? ? ?>? ? ?> Spectrum Scale Development
>>         >? ? ?>? ? ?> Client Adoption
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> _______________________________________________
>>         >? ? ?>? ? ?> gpfsug-discuss mailing list
>>         >? ? ?>? ? ?> gpfsug-discuss at _spectrumscale.org_
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>         >? ? ?>? ? ?> _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>>         >? ? ?>
>>         >? ? ?>? ? ?> _______________________________________________
>>         >? ? ?>? ? ?> gpfsug-discuss mailing list
>>         >? ? ?>? ? ?> gpfsug-discuss at _spectrumscale.org_
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>         >? ? ?>
>>         >? ? ?>? ? ?> _https://urldefense.proofpoint.com/v2/url?_
>>         >? ? ?<https://urldefense.proofpoint.com/v2/url?_>
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         ?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         ?siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>>         >? ? ?>
>>         >? ? ?>? ? ?_______________________________________________
>>         >? ? ?>? ? ?gpfsug-discuss mailing list
>>         >? ? ?>? ? ?gpfsug-discuss at _spectrumscale.org_
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>>         >? ? ?>? ? ?__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>>         >? ? ?>? ? ?gpfsug-discuss mailing list
>>         >? ? ?>? ? ?gpfsug-discuss at spectrumscale.org
>>         <http://spectrumscale.org> <http://spectrumscale.org>
>>         >? ? ?>? ?
>>         >? ?
>>         ??https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> _______________________________________________
>>         >? ? ?> gpfsug-discuss mailing list
>>         >? ? ?> gpfsug-discuss at spectrumscale.org
>>         <http://spectrumscale.org> <http://spectrumscale.org>
>>         >? ? ?> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>>         >? ? ?>
>>         >
>>         >? ? ?--
>>         >? ? ?Aaron Knister
>>         >? ? ?NASA Center for Climate Simulation (Code 606.2)
>>         >? ? ?Goddard Space Flight Center
>>         >? ? ?(301) 286-2776 <tel:(301)%20286-2776>
>>         <tel:%28301%29%20286-2776>
>>         >? ? ?_______________________________________________
>>         >? ? ?gpfsug-discuss mailing list
>>         >? ? ?gpfsug-discuss at spectrumscale.org
>>         <http://spectrumscale.org> <http://spectrumscale.org>
>>         >? ? ?http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>>         >
>>         >
>>         >
>>         >
>>         > _______________________________________________
>>         > gpfsug-discuss mailing list
>>         > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>>         > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>         >
>>
>>         --
>>         Aaron Knister
>>         NASA Center for Climate Simulation (Code 606.2)
>>         Goddard Space Flight Center
>>         (301) 286-2776 <tel:(301)%20286-2776>
>>         _______________________________________________
>>         gpfsug-discuss mailing list
>>         gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776


From bipcuds at gmail.com  Tue Dec 19 23:08:57 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Tue, 19 Dec 2017 18:08:57 -0500
Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the
	Spectrum Scale FAQ
Message-ID: <CAAxuGpF6nw6R_Zmowx3Xv_pUbxikgvTXeR7g-p08TpMvh_ZgzQ@mail.gmail.com>

Hi All,

I was recently trying to determine the latest RHEL release that will work
with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code
uses Scale 4.1.0.8.

A specific X.Y GSS code build, from my experience, is intended to use a
specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken),
which no longer has EUS support from RedHat (only 6.7 still does).

GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be
used, which is a surprising statement given GSS/ESS code's sensitivity to
OS levels (any ESS I have ever seen has never been supported on more than
one version of RHEL).

According to the Scale FAQ (
https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux),
A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying
RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported
by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From
V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27!

Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install
GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is
supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option
here).

Many Thanks,
  Keith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/fa1226b4/attachment-0001.htm>

From olaf.weiser at de.ibm.com  Wed Dec 20 07:35:26 2017
From: olaf.weiser at de.ibm.com (Olaf Weiser)
Date: Wed, 20 Dec 2017 08:35:26 +0100
Subject: [gpfsug-discuss] pmcollector and NSD perf
In-Reply-To: <DM5PR05MB3577119DC6FE73462E83BC72E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
References: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com><OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>
	<DM5PR05MB3577119DC6FE73462E83BC72E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
Message-ID: <OFF3295D55.B6D7C020-ONC12581FC.0028AADD-C12581FC.0029B20E@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0003.gif>

From john.hearns at asml.com  Wed Dec 20 08:41:07 2017
From: john.hearns at asml.com (John Hearns)
Date: Wed, 20 Dec 2017 08:41:07 +0000
Subject: [gpfsug-discuss] Expeliarmus
Message-ID: <HE1PR02MB1450E823234415275F6EADA3880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/a9561284/attachment-0001.htm>

From r.sobey at imperial.ac.uk  Wed Dec 20 08:58:20 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Wed, 20 Dec 2017 08:58:20 +0000
Subject: [gpfsug-discuss] Expeliarmus
In-Reply-To: <HE1PR02MB1450E823234415275F6EADA3880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
References: <HE1PR02MB1450E823234415275F6EADA3880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
Message-ID: <DB6PR0601MB21012910B19A735F4F5835DCDF0C0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Don't take it personally, I'm not entitled to even download 4.X. I can't be bothered to jump through the hoops necessary to get it going.

Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of John Hearns
Sent: 20 December 2017 08:41
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Expeliarmus

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/b4f62495/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Wed Dec 20 09:31:25 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Wed, 20 Dec 2017 09:31:25 +0000
Subject: [gpfsug-discuss] Expeliarmus
Message-ID: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>

This is assuming you directly have ?IBM? licenses (as opposed to OEM licenses where the route is different ? or where the licenses are held by your VAR rather than you ?)

You need to have an IBM account which is attached to a (Passport Advantage) PA site that has current support for the product?

If you go to fix central, its also listed there now as well so again assuming your account is appropriately attached to a support contract, you can download it.

If you are struggling with these, then go speak to your business partner or IBM account manager.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "john.hearns at asml.com" <john.hearns at asml.com>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, 20 December 2017 at 08:41
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Expeliarmus

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/376c5117/attachment-0001.htm>

From john.hearns at asml.com  Wed Dec 20 13:02:54 2017
From: john.hearns at asml.com (John Hearns)
Date: Wed, 20 Dec 2017 13:02:54 +0000
Subject: [gpfsug-discuss] Expeliarmus
In-Reply-To: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>
References: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>
Message-ID: <HE1PR02MB1450FFCA4DEC69B3E2F70E4A880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>

Thnkyou both. I tapped on the door of Fix Central with my wand, and now have the eye of newt, wing of bat and a Spectrum Scale 5 installer.


From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support)
Sent: Wednesday, December 20, 2017 10:31 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Expeliarmus

This is assuming you directly have ?IBM? licenses (as opposed to OEM licenses where the route is different ? or where the licenses are held by your VAR rather than you ?)

You need to have an IBM account which is attached to a (Passport Advantage) PA site that has current support for the product?

If you go to fix central, its also listed there now as well so again assuming your account is appropriately attached to a support contract, you can download it.

If you are struggling with these, then go speak to your business partner or IBM account manager.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "john.hearns at asml.com<mailto:john.hearns at asml.com>" <john.hearns at asml.com<mailto:john.hearns at asml.com>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, 20 December 2017 at 08:41
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] Expeliarmus

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/55a82b6a/attachment-0001.htm>

From ewahl at osc.edu  Wed Dec 20 14:53:20 2017
From: ewahl at osc.edu (Edward Wahl)
Date: Wed, 20 Dec 2017 09:53:20 -0500
Subject: [gpfsug-discuss] Expeliarmus
In-Reply-To: <HE1PR02MB1450FFCA4DEC69B3E2F70E4A880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
References: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>
	<HE1PR02MB1450FFCA4DEC69B3E2F70E4A880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
Message-ID: <20171220095320.1093db0d@osc.edu>


And it's not JUST a PA account. 

  You also have to have "someone" convert your old
GPFS part numbers into Spectrum Scale part numbers so that Passport Advantage
will show you the entitlements and downloads properly.  I went through this,
almost not painful at all, but rather slow.

Ed


On Wed, 20 Dec 2017 13:02:54 +0000
John Hearns <john.hearns at asml.com> wrote:

> Thnkyou both. I tapped on the door of Fix Central with my wand, and now have
> the eye of newt, wing of bat and a Spectrum Scale 5 installer.
> 
> 
> From: gpfsug-discuss-bounces at spectrumscale.org
> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson
> (IT Research Support) Sent: Wednesday, December 20, 2017 10:31 AM To: gpfsug
> main discussion list <gpfsug-discuss at spectrumscale.org> Subject: Re:
> [gpfsug-discuss] Expeliarmus
> 
> This is assuming you directly have ?IBM? licenses (as opposed to OEM licenses
> where the route is different ? or where the licenses are held by your VAR
> rather than you ?)
> 
> You need to have an IBM account which is attached to a (Passport Advantage)
> PA site that has current support for the product?
> 
> If you go to fix central, its also listed there now as well so again assuming
> your account is appropriately attached to a support contract, you can
> download it.
> 
> If you are struggling with these, then go speak to your business partner or
> IBM account manager.
> 
> Simon
> 
> From:
> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>
> on behalf of "john.hearns at asml.com<mailto:john.hearns at asml.com>"
> <john.hearns at asml.com<mailto:john.hearns at asml.com>> Reply-To:
> "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>"
> <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
> Date: Wednesday, 20 December 2017 at 08:41 To:
> "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>"
> <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
> Subject: [gpfsug-discuss] Expeliarmus
> 
> I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
> For the life of me I cannot summon the spell needed to put Spectrum Scale 5
> on my orders list.
> 
> Can some kindly witch give me the incantation please?
> I would like to install on a test cluster, as the wisdom of the mages and
> mavens here has it.
> 
> -- The information contained in this communication and any attachments is
> confidential and may be privileged, and is for the sole use of the intended
> recipient(s). Any unauthorized review, use, disclosure or distribution is
> prohibited. Unless explicitly stated otherwise in the body of this
> communication or the attachment thereto (if any), the information is provided
> on an AS-IS basis without any express or implied warranties or liabilities.
> To the extent you are relying on this information, you are doing so at your
> own risk. If you are not the intended recipient, please notify the sender
> immediately by replying to this message and destroy all copies of this
> message and any attachments. Neither the sender nor the company/group of
> companies he or she represents shall be liable for the proper and complete
> transmission of the information contained in this communication, or for any
> delay in its receipt. -- The information contained in this communication and
> any attachments is confidential and may be privileged, and is for the sole
> use of the intended recipient(s). Any unauthorized review, use, disclosure or
> distribution is prohibited. Unless explicitly stated otherwise in the body of
> this communication or the attachment thereto (if any), the information is
> provided on an AS-IS basis without any express or implied warranties or
> liabilities. To the extent you are relying on this information, you are doing
> so at your own risk. If you are not the intended recipient, please notify the
> sender immediately by replying to this message and destroy all copies of this
> message and any attachments. Neither the sender nor the company/group of
> companies he or she represents shall be liable for the proper and complete
> transmission of the information contained in this communication, or for any
> delay in its receipt.



-- 

Ed Wahl
Ohio Supercomputer Center
614-292-9302


From eric.wonderley at vt.edu  Wed Dec 20 19:40:27 2017
From: eric.wonderley at vt.edu (J. Eric Wonderley)
Date: Wed, 20 Dec 2017 14:40:27 -0500
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same host
Message-ID: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>

Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all
mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core
switch.  I'm somewhat fearful as to how this is implemented in the juniper
switch we are about to test.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/3cf2f760/attachment-0001.htm>

From aaron.s.knister at nasa.gov  Wed Dec 20 20:25:04 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Wed, 20 Dec 2017 20:25:04 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
 host
In-Reply-To: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
References: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
Message-ID: <A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>



We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper.

Are you attempting to do RoCE or just plain TCP/IP?


On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu> wrote:
Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/a0eb8e44/attachment-0001.htm>

From eric.wonderley at vt.edu  Wed Dec 20 20:37:09 2017
From: eric.wonderley at vt.edu (J. Eric Wonderley)
Date: Wed, 20 Dec 2017 15:37:09 -0500
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
	host
In-Reply-To: <A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
References: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
Message-ID: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>

Just plain tcpip.

We have dual port connectx4s in our nsd servers.  Upon adding a second
connectx4 hba...no links go up or show "up".  I have one port on each hba
configured for eth and ibv_devinfo looks sane.

I cannot find anything indicating that this should not work.  I have a
ticket opened with mellanox.

On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER
SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:

>
>
> We?ve done a fair amount of VPI work but admittedly not with connectx4. Is
> it possible the cards are trying to talk IB rather than Eth? I figured
> you?re Ethernet based because of the mention of Juniper.
>
> Are you attempting to do RoCE or just plain TCP/IP?
>
>
> On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <
> eric.wonderley at vt.edu> wrote:
>
> Hello:
>
> Does anyone have this type of config?
>
> The host configuration looks sane but we seem to observe link-down on all
> mlx adapters no matter what we do.
>
> Big picture is that we are attempting to do mc(multichassis)-lags to a
> core switch.  I'm somewhat fearful as to how this is implemented in the
> juniper switch we are about to test.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/f26c0b63/attachment-0001.htm>

From S.J.Thompson at bham.ac.uk  Wed Dec 20 20:45:37 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Wed, 20 Dec 2017 20:45:37 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in
	same	host
In-Reply-To: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
References: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>,
	<CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A42875@EX13.adf.bham.ac.uk>

I can't remember if this was on mlx4 or mlx5 driver cards, but we found we had to use LINKDELAY=20 when using bonding for Ethernet.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of J. Eric Wonderley [eric.wonderley at vt.edu]
Sent: 20 December 2017 20:37
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same      host

Just plain tcpip.

We have dual port connectx4s in our nsd servers.  Upon adding a second connectx4 hba...no links go up or show "up".  I have one port on each hba configured for eth and ibv_devinfo looks sane.

I cannot find anything indicating that this should not work.  I have a ticket opened with mellanox.

On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>> wrote:


We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper.

Are you attempting to do RoCE or just plain TCP/IP?


On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu<mailto:eric.wonderley at vt.edu>> wrote:
Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test.

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




From abeattie at au1.ibm.com  Wed Dec 20 21:46:53 2017
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Wed, 20 Dec 2017 21:46:53 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in
	samehost
In-Reply-To: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>,
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com><A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
Message-ID: <OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/4a535991/attachment-0001.htm>

From Paul.Sanchez at deshaw.com  Wed Dec 20 21:53:10 2017
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 20 Dec 2017 21:53:10 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter
	in	samehost
In-Reply-To: <OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>,
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com><A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
	<OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
Message-ID: <55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>

We have run multiple ConnectX-4 NICs in bonded MLAG (Arista) and VPC (Cisco) switch configurations on our NSD servers.  We used to see issues with firmware versions that didn?t support the optics we wanted to use (e.g. early CX3/CX4 and Cisco 40G-BiDi).  You may also want check mstflint to see whether the firmware levels match on the MLX cards, and if you upgrade firmware in some cases a power-cycle (not reboot) can be required to finish the process.

-Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Andrew Beattie
Sent: Wednesday, December 20, 2017 4:47 PM
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in samehost

IBM ESS building blocks can have up to 3 dual port 10GBEth, 40GB Eth, 56GB IB, 100GB IB Mlx adapater cards, because we have 2 IO nodes this is up to a total of 12 ports per building block
so there should not be any reason for this to fail.

I regularly see a Mix of 10GB / 40GB or 10GB / IB configurations



Regards
Andrew Beattie
Software Defined Storage  - IT Specialist
Phone: 614-2133-7927
E-mail: abeattie at au1.ibm.com<mailto:abeattie at au1.ibm.com>


----- Original message -----
From: "J. Eric Wonderley" <eric.wonderley at vt.edu<mailto:eric.wonderley at vt.edu>>
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc:
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same host
Date: Thu, Dec 21, 2017 6:37 AM

Just plain tcpip.

We have dual port connectx4s in our nsd servers.  Upon adding a second connectx4 hba...no links go up or show "up".  I have one port on each hba configured for eth and ibv_devinfo looks sane.

I cannot find anything indicating that this should not work.  I have a ticket opened with mellanox.

On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>> wrote:



We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper.

Are you attempting to do RoCE or just plain TCP/IP?

On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu<mailto:eric.wonderley at vt.edu>> wrote:
Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test.

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=I5Dq2T7aYvC87Wp12fsz6CRLw4uo2-RVnrnpxRYfYuA&e=>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/0b3daff0/attachment-0001.htm>

From david_johnson at brown.edu  Wed Dec 20 22:00:49 2017
From: david_johnson at brown.edu (David D Johnson)
Date: Wed, 20 Dec 2017 17:00:49 -0500
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter
	in	samehost
In-Reply-To: <55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
	<OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
	<55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>
Message-ID: <D14EB35E-865A-400D-B81A-60FAC4002D22@brown.edu>

We're trying to get 40 gbe connection between Mellanox switches and Cisco switches down at the other end of the machine room  
The BiDi part seems to be the best given about 30m  run on multimode.  However Mellanox support says it's not supported.
Want to use this to get close to IB speeds for GPFS on nodes that aren't on the IB fabric.  
Does anyone have any luck getting 40 or 100 gig at 20-30m when the switches are different brands?

Thanks,
 -- ddj

> On Dec 20, 2017, at 4:53 PM, Sanchez, Paul <Paul.Sanchez at deshaw.com> wrote:
> 
> We have run multiple ConnectX-4 NICs in bonded MLAG (Arista) and VPC (Cisco) switch configurations on our NSD servers.  We used to see issues with firmware versions that didn?t support the optics we wanted to use (e.g. early CX3/CX4 and Cisco 40G-BiDi).  You may also want check mstflint to see whether the firmware levels match on the MLX cards, and if you upgrade firmware in some cases a power-cycle (not reboot) can be required to finish the process.
>  
> -Paul
> ? <>
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org]On Behalf Of Andrew Beattie
> Sent: Wednesday, December 20, 2017 4:47 PM
> To: gpfsug-discuss at spectrumscale.org
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in samehost
>  
> IBM ESS building blocks can have up to 3 dual port 10GBEth, 40GB Eth, 56GB IB, 100GB IB Mlx adapater cards, because we have 2 IO nodes this is up to a total of 12 ports per building block
> so there should not be any reason for this to fail.
>  
> I regularly see a Mix of 10GB / 40GB or 10GB / IB configurations
>  
>  
>  
> Regards
> Andrew Beattie
> Software Defined Storage  - IT Specialist
> Phone: 614-2133-7927
> E-mail: abeattie at au1.ibm.com <mailto:abeattie at au1.ibm.com>
>  
>  
> ----- Original message -----
> From: "J. Eric Wonderley" <eric.wonderley at vt.edu <mailto:eric.wonderley at vt.edu>>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org <mailto:gpfsug-discuss at spectrumscale.org>>
> Cc:
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same host
> Date: Thu, Dec 21, 2017 6:37 AM
>   
> Just plain tcpip.
>  
> We have dual port connectx4s in our nsd servers.  Upon adding a second connectx4 hba...no links go up or show "up".  I have one port on each hba configured for eth and ibv_devinfo looks sane.
>  
> I cannot find anything indicating that this should not work.  I have a ticket opened with mellanox.
>  
> On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote: 
>  
> 
> 
> We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper. 
>  
> Are you attempting to do RoCE or just plain TCP/IP?
>  
> On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu <mailto:eric.wonderley at vt.edu>> wrote:
> Hello:
>  
> Does anyone have this type of config?
>  
> The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.
>  
> Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test. 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=I5Dq2T7aYvC87Wp12fsz6CRLw4uo2-RVnrnpxRYfYuA&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>
>  
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/2b7ae14b/attachment-0001.htm>

From oehmes at gmail.com  Thu Dec 21 01:09:19 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Thu, 21 Dec 2017 01:09:19 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in
	samehost
In-Reply-To: <D14EB35E-865A-400D-B81A-60FAC4002D22@brown.edu>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
	<OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
	<55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>
	<D14EB35E-865A-400D-B81A-60FAC4002D22@brown.edu>
Message-ID: <CALssuR3vrn_DO0o+mnquTNObOm=pALfxic1z4miMYnEwbC2vDw@mail.gmail.com>

i don't know if that works with cisco, but i use 50 an 100m cables for 40
as well as 100Gbit in my lab between 2 Mellanox switches :
http://www.mellanox.com/products/interconnect/ethernet-active-optical-cables.php
as paul pointed out one of the very first things one needs to do after
adding an adapter is to flash the firmware to a recent level. especially of
you have 2 adapters with different FW i have seen even once with higher
level not work properly, so before you do anything else get them to a
recent level and especially the same if its the same adapter types.

sven

On Wed, Dec 20, 2017 at 10:01 PM David D Johnson <david_johnson at brown.edu>
wrote:

> We're trying to get 40 gbe connection between Mellanox switches and Cisco
> switches down at the other end of the machine room
> The BiDi part seems to be the best given about 30m  run on multimode.
> However Mellanox support says it's not supported.
> Want to use this to get close to IB speeds for GPFS on nodes that aren't
> on the IB fabric.
> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
> switches are different brands?
>
> Thanks,
>  -- ddj
>
> On Dec 20, 2017, at 4:53 PM, Sanchez, Paul <Paul.Sanchez at deshaw.com>
> wrote:
>
> We have run multiple ConnectX-4 NICs in bonded MLAG (Arista) and VPC
> (Cisco) switch configurations on our NSD servers.  We used to see issues
> with firmware versions that didn?t support the optics we wanted to use
> (e.g. early CX3/CX4 and Cisco 40G-BiDi).  You may also want check mstflint
> to see whether the firmware levels match on the MLX cards, and if you
> upgrade firmware in some cases a power-cycle (not reboot) can be required
> to finish the process.
>
> -Paul
>
> *From:* gpfsug-discuss-bounces at spectrumscale.org [
> mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>]*On Behalf Of *Andrew Beattie
> *Sent:* Wednesday, December 20, 2017 4:47 PM
> *To:* gpfsug-discuss at spectrumscale.org
> *Subject:* Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in
> samehost
>
> IBM ESS building blocks can have up to 3 dual port 10GBEth, 40GB Eth, 56GB
> IB, 100GB IB Mlx adapater cards, because we have 2 IO nodes this is up to a
> total of 12 ports per building block
> so there should not be any reason for this to fail.
>
> I regularly see a Mix of 10GB / 40GB or 10GB / IB configurations
>
>
>
> Regards
> *Andrew Beattie*
> *Software Defined Storage  - IT Specialist*
> *Phone: *614-2133-7927
> *E-mail: *abeattie at au1.ibm.com
>
>
>
> ----- Original message -----
> From: "J. Eric Wonderley" <eric.wonderley at vt.edu>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
> host
> Date: Thu, Dec 21, 2017 6:37 AM
>
> Just plain tcpip.
>
> We have dual port connectx4s in our nsd servers.  Upon adding a second
> connectx4 hba...no links go up or show "up".  I have one port on each hba
> configured for eth and ibv_devinfo looks sane.
>
> I cannot find anything indicating that this should not work.  I have a
> ticket opened with mellanox.
>
> On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER
> SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:
>
>
>
>
> We?ve done a fair amount of VPI work but admittedly not with connectx4. Is
> it possible the cards are trying to talk IB rather than Eth? I figured
> you?re Ethernet based because of the mention of Juniper.
>
> Are you attempting to do RoCE or just plain TCP/IP?
>
> On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <
> eric.wonderley at vt.edu> wrote:
>
> Hello:
>
> Does anyone have this type of config?
>
> The host configuration looks sane but we seem to observe link-down on all
> mlx adapters no matter what we do.
>
> Big picture is that we are attempting to do mc(multichassis)-lags to a
> core switch.  I'm somewhat fearful as to how this is implemented in the
> juniper switch we are about to test.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=I5Dq2T7aYvC87Wp12fsz6CRLw4uo2-RVnrnpxRYfYuA&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/ef7b4e3c/attachment-0001.htm>

From kraemerf at de.ibm.com  Thu Dec 21 07:07:24 2017
From: kraemerf at de.ibm.com (Frank Kraemer)
Date: Thu, 21 Dec 2017 08:07:24 +0100
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
	host
In-Reply-To: <mailman.3419.1513818573.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3419.1513818573.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <OFE99AF1B2.2CCA57A0-ONC12581FD.00272158-C12581FD.00272158@notes.na.collabserv.com>

David,

> We're trying to get 40 gbe connection between Mellanox switches and Cisco
> switches down at the other end of the machine room
> The BiDi part seems to be the best given about 30m  run on multimode.
> However Mellanox support says it's not supported.
> Want to use this to get close to IB speeds for GPFS on nodes that aren't
> on the IB fabric.
> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
> switches are different brands?

maybe that's a good reason to get in contact with the team from Interoptic.
They claim a good expertise for these kind of problems - feedback is good.
http://packetpushers.net/podcast/podcasts/show-360-all-about-optics-interoptic-sponsored/
https://interoptic.com/

Frank Kraemer
IBM Consulting IT Specialist  / Client Technical Architect
Am Weiher 24, 65451 Kelsterbach
mailto:kraemerf at de.ibm.com
voice: +49-(0)171-3043699 / +4970342741078
IBM Germany
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/31aeae52/attachment-0001.htm>

From daniel.kidger at uk.ibm.com  Thu Dec 21 12:21:59 2017
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Thu, 21 Dec 2017 12:21:59 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
Message-ID: <OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>

My suggestion is that it is better to not think of the performance coming from having more than 32 sub-blocks but instead that the performance comes from smaller sub-blocks. The fact that there are now more of them in say a 4MB blocksize filesytem is just a side effect.

Daniel


 

 
 	
Dr Daniel Kidger 
IBM Technical Sales Specialist
Software Defined Solution Sales

+ 44-(0)7818 522 266 
daniel.kidger at uk.ibm.com

> On 19 Dec 2017, at 21:32, Aaron Knister <aaron.s.knister at nasa.gov> wrote:
> 
> Thanks, Sven. Understood!
> 
>> On 12/19/17 3:20 PM, Sven Oehme wrote:
>> Hi,
>> 
>> the zero padding was never promoted into a GA stream, it was an
>> experiment to proof we are on the right track when we eliminate the
>> overhead from client to NSD Server, but also showed that alone is not
>> good enough. the work for the client is the same compared to the >32
>> subblocks, but the NSD Server has more work as it can't pack as many
>> subblocks and therefore files into larger blocks, so you need to do more
>> writes to store the same number of files. 
>> thats why there is the additional substantial improvement  when we then
>> went to >32 subblocks. 
>> 
>> sven
>> 
>> On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
>> SCIENCE CORP] <aaron.s.knister at nasa.gov
>> <mailto:aaron.s.knister at nasa.gov>> wrote:
>> 
>>    Thanks Sven! That makes sense to me and is what I thought was the
>>    case which is why I was confused when I saw the reply to the thread
>>    that said the >32 subblocks code had no performance impact. 
>> 
>>    A couple more question for you? in your presentation there?s a
>>    benchmark that shows the file create performance without the zero
>>    padding. Since you mention this is done for security reasons was
>>    that feature ever promoted to a GA Scale release? I?m also wondering
>>    if you could explain the performance difference between the no zero
>>    padding code and the > 32 subblock code since given your the example
>>    of 32K files and 16MB block size I figure both cases ought to write
>>    the same amount to disk. 
>> 
>>    Thanks!
>> 
>>    -Aaron
>> 
>> 
>> 
>> 
>> 
>>    On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com
>>    <mailto:oehmes at gmail.com>> wrote:
>>>    i thought i answered that already, but maybe i just thought about
>>>    answering it and then forgot about it :-D
>>> 
>>>    so yes more than 32 subblocks per block significant increase the
>>>    performance of filesystems with small files, for the sake of the
>>>    argument let's say 32k in a large block filesystem again for sake
>>>    of argument say 16MB. 
>>> 
>>>    you probably ask why ? 
>>> 
>>>    if you create a file and write 32k into it in a pre 5.0.0 Version
>>>    16 MB filesystem your client actually doesn't write 32k to the NSD
>>>    Server, it writes 512k, because thats the subblock size and we
>>>    need to write the full subblock (for security reasons). so first
>>>    you waste significant memory on the client to cache that zero
>>>    padding, you waste network bandwidth and you waste NSD Server
>>>    cache because you store it there too. this means you overrun the
>>>    cache more quickly, means you start doing read/modify writes
>>>    earlier on all your nice large raid tracks... i guess you get the
>>>    story by now. 
>>> 
>>>    in fact,  if you have a good raid code that can drive really a lot
>>>    of bandwidth out of individual drives like a GNR system you get
>>>    more performance for small file writes as larger your blocksize
>>>    is, because we can 'pack' more files into larger i/os and
>>>    therefore turn a small file create workload into a bandwidth
>>>    workload, essentially exactly what we did and i demonstrated in
>>>    the CORAL presentation . 
>>> 
>>>    hope that makes this crystal clear now . 
>>> 
>>>    sven
>>> 
>>> 
>>> 
>>>    On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister
>>>    <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
>>> 
>>>        Thanks, Alex. I'm all too familiar with the trade offs between
>>>        large
>>>        blocks and small files and we do use pretty robust SSD storage
>>>        for our
>>>        metadata. We support a wide range of workloads and we have
>>>        some folks
>>>        with many small (<1M) files and other folks with many large
>>>        (>256MB) files.
>>> 
>>>        My point in this thread is that IBM has said over and over
>>>        again in
>>>        presentations that there is a significant performance gain
>>>        with the >32
>>>        subblocks code on filesystems with large block sizes (although
>>>        to your
>>>        point I'm not clear on exactly what large means since I didn't
>>>        define
>>>        large in this context). Therefore given that the >32 subblock
>>>        code gives
>>>        a significant performance gain one could reasonably assume
>>>        that having a
>>>        filesystem with >32 subblocks is required to see this gain
>>>        (rather than
>>>        just running the >32 subblocks code on an fs w/o > 32 subblocks).
>>> 
>>>        This lead me to ask about a migration tool because in my mind
>>>        if there's
>>>        a performance gain from having >32 subblocks on the FS I'd
>>>        like that
>>>        feature and having to manually copy 10's of PB to new hardware
>>>        to get
>>>        this performance boost is unacceptable. However, IBM can't
>>>        seem to make
>>>        up their mind about whether or not the >32 subblocks code
>>>        *actually*
>>>        provides a performance increase. This seems like a pretty
>>>        straightforward question.
>>> 
>>>        -Aaron
>>> 
>>>>        On 12/15/17 3:48 PM, Alex Chekholko wrote:
>>>> Hey Aaron,
>>>> 
>>>> Can you define your sizes for "large blocks" and "small
>>>        files"?  If you
>>>> dial one up and the other down, your performance will be
>>>        worse.  And in
>>>> any case it's a pathological corner case so it shouldn't
>>>        matter much for
>>>> your workflow, unless you've designed your system with the
>>>        wrong values.
>>>> 
>>>> For example, for bioinformatics workloads, I prefer to use 256KB
>>>> filesystem block size, and I'd consider 4MB+ to be "large
>>>        block size",
>>>> which would make the filesystem obviously unsuitable for
>>>        processing
>>>> millions of 8KB files.
>>>> 
>>>> You can make a histogram of file sizes in your existing
>>>        filesystems and
>>>> then make your subblock size (1/32 of block size) on the
>>>        smaller end of
>>>> that.   Also definitely use the "small file in inode"
>>>        feature and put
>>>> your metadata on SSD.
>>>> 
>>>> Regards,
>>>> Alex
>>>> 
>>>> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
>>>> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>
>>>        <mailto:aaron.s.knister at nasa.gov
>>>        <mailto:aaron.s.knister at nasa.gov>>> wrote:
>>>> 
>>>>      Thanks, Bill.
>>>> 
>>>>      I still don't feel like I've got an clear answer from
>>>        IBM and frankly
>>>>      the core issue of a lack of migration tool was totally
>>>        dodged.
>>>> 
>>>>      Again in Sven's presentation from SSUG @ SC17
>>>>    
>>>         (https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=>)
>>>>      he mentions "It has a significant performance penalty
>>>        for small files in
>>>>      large block size filesystems" and the demonstrates that
>>>        with several
>>>>      mdtest runs (which show the effect with and without the >32
>>>>      subblocks code):
>>>> 
>>>> 
>>>>      4.2.1 base code - SUMMARY: (of 3 iterations)
>>>>      File creation : Mean = 2237.644
>>>> 
>>>>      zero-end-of-file-padding (4.2.2 + ifdef for zero
>>>        padding):  SUMMARY: (of
>>>>      3 iterations)
>>>>      File creation : Mean = 12866.842
>>>> 
>>>>      more sub blocks per block (4.2.2 + morethan32subblock code):
>>>>      File creation : Mean = 40316.721
>>>> 
>>>>      Can someone (ideally Sven) give me a straight answer as
>>>        to whether or
>>>>      not the > 32 subblock code actually makes a performance
>>>        difference for
>>>>      small files in large block filesystems? And if not, help
>>>        me understand
>>>>      why his slides and provided benchmark data have
>>>        consistently indicated
>>>>      it does?
>>>> 
>>>>      -Aaron
>>>> 
>>>>      On 12/1/17 11:44 AM, Bill Hartner wrote:
>>>>      > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported
>>>        some of the
>>>>      > results @ user group meeting.
>>>>      >
>>>>      > -Bill
>>>>      >
>>>>      > Bill Hartner
>>>>      > IBM Systems
>>>>      > Scalable I/O Development
>>>>      > Austin, Texas
>>>>      > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
>>>        <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>
>>>>      > home office 512-784-0980 <tel:(512)%20784-0980>
>>>        <tel:512-784-0980 <tel:(512)%20784-0980>>
>>>>      >
>>>>      >
>>>>      > Inactive hide details for Jan-Frode Myklebust
>>>        ---12/01/2017 06:53:44
>>>>      > AM---Bill, could you say something about what the
>>>        metadataJan-Frode
>>>>      > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you
>>>        say something
>>>>      > about what the metadata-storage here was?
>>>        ESS/NL-SAS/3way replication?
>>>>      >
>>>>      > From: Jan-Frode Myklebust <janfrode at tanso.net
>>>        <mailto:janfrode at tanso.net> <mailto:janfrode at tanso.net
>>>        <mailto:janfrode at tanso.net>>>
>>>>      > To: gpfsug main discussion list
>>>        <gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>>>
>>>>      > Date: 12/01/2017 06:53 AM
>>>>      > Subject: Re: [gpfsug-discuss] Online data migration tool
>>>>      > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>>>>      >
>>>>      >
>>>>    
>>>         ------------------------------------------------------------------------
>>>>      >
>>>>      >
>>>>      >
>>>>      > Bill, could you say something about what the
>>>        metadata-storage here was?
>>>>      > ESS/NL-SAS/3way replication?
>>>>      >
>>>>      > I just asked about this in the internal slack channel
>>>        #scale-help today..
>>>>      >
>>>>      >
>>>>      >
>>>>      > -jf
>>>>      >
>>>>      > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
>>>        <_bhartner at us.ibm.com_
>>>>      > <mailto:bhartner at us.ibm.com
>>>        <mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com
>>>        <mailto:bhartner at us.ibm.com>>>>:
>>>>      >
>>>>      >     > "It has a significant performance penalty for
>>>        small files in
>>>>      large
>>>>      >     > block size filesystems"
>>>>      >
>>>>      >     Aaron,
>>>>      >
>>>>      >     Below are mdtest results for a test we ran for
>>>        CORAL - file
>>>>      size was
>>>>      >     32k.
>>>>      >
>>>>      >     We have not gone back and ran the test on a file
>>>        system formatted
>>>>      >     without > 32 subblocks. We'll do that at some point...
>>>>      >
>>>>      >     -Bill
>>>>      >
>>>>      >     -- started at 10/28/2017 17:51:38 --
>>>>      >
>>>>      >     mdtest-1.9.3 was launched with 228 total task(s)
>>>        on 12 node(s)
>>>>      >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>>>>      >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w
>>>        32768 -C -F -r
>>>>      -p 360
>>>>      >     -u -y
>>>>      >     Path: /ibm/fs2-16m-10
>>>>      >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used
>>>        Inodes: 0.0%
>>>>      >
>>>>      >     228 tasks, 67239936 files
>>>>      >
>>>>      >     SUMMARY: (of 3 iterations)
>>>>      >     Operation Max Min Mean Std Dev
>>>>      >     --------- --- --- ---- -------
>>>>      >     File creation : 51953.498 50558.517 51423.221 616.643
>>>>      >     File stat : 0.000 0.000 0.000 0.000
>>>>      >     File read : 0.000 0.000 0.000 0.000
>>>>      >     File removal : 96746.376 92149.535 94658.774 1900.187
>>>>      >     Tree creation : 1.588 0.070 0.599 0.700
>>>>      >     Tree removal : 0.213 0.034 0.097 0.082
>>>>      >
>>>>      >     -- finished at 10/28/2017 19:51:54 --
>>>>      >
>>>>      >     Bill Hartner
>>>>      >     IBM Systems
>>>>      >     Scalable I/O Development
>>>>      >     Austin, Texas_
>>>>      >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
>>>        <mailto:bhartner at us.ibm.com>
>>>>      <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>
>>>>      >     home office 512-784-0980 <tel:(512)%20784-0980>
>>>        <tel:512-784-0980 <tel:(512)%20784-0980>>
>>>>      >
>>>>      >     _
>>>>      >     __gpfsug-discuss-bounces at spectrumscale.org_
>>>>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>>> wrote on
>>>>      >     11/29/2017 04:41:48 PM:
>>>>      >
>>>>      >     > From: Aaron Knister <_aaron.knister at gmail.com_
>>>>      >     <mailto:aaron.knister at gmail.com
>>>        <mailto:aaron.knister at gmail.com>
>>>        <mailto:aaron.knister at gmail.com
>>>        <mailto:aaron.knister at gmail.com>>>>
>>>>      >
>>>>      >
>>>>      >     > To: gpfsug main discussion list
>>>>      >     <_gpfsug-discuss at spectrumscale.org_
>>>>      >     <mailto:gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>>>>
>>>>      >
>>>>      >     > Date: 11/29/2017 04:42 PM
>>>>      >
>>>>      >
>>>>      >     > Subject: Re: [gpfsug-discuss] Online data
>>>        migration tool
>>>>      >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>>>>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>>>
>>>>      >
>>>>      >     >
>>>>      >
>>>>      >     > Thanks, Nikhil. Most of that was consistent with
>>>        my understnading,
>>>>      >     > however I was under the impression that the >32
>>>        subblocks code is
>>>>      >     > required to achieve the touted 50k file
>>>        creates/second that Sven has
>>>>      >     > talked about a bunch of times:
>>>>      >     >
>>>>      >     >
>>>>      >   
>>>>    
>>>          _https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>>>>      >     >
>>>>      >   
>>>>    
>>>          _https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=>
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>>>>      >     > _Sven_Oehme_-_News_from_Research.pdf
>>>>      >     >
>>>        _https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>>>>      >     >
>>>        _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>>>>      >
>>>>      >
>>>>      >     > from those presentations regarding 32 subblocks:
>>>>      >     >
>>>>      >     > "It has a significant performance penalty for
>>>        small files in large
>>>>      >     > block size filesystems"
>>>>      >
>>>>      >     > although I'm not clear on the specific
>>>        definition of "large". Many
>>>>      >     > filesystems I encounter only have a 1M block
>>>        size so it may not
>>>>      >     > matter there, although that same presentation
>>>        clearly shows the
>>>>      >     > benefit of larger block sizes which is yet
>>>        *another* thing for which
>>>>      >     > a migration tool would be helpful.
>>>>      >
>>>>      >     > -Aaron
>>>>      >     >
>>>>      >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>>>>      >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
>>>        <mailto:nikhilk at us.ibm.com>
>>>>      <mailto:nikhilk at us.ibm.com
>>>        <mailto:nikhilk at us.ibm.com>>>> wrote:
>>>>      >
>>>>      >     > Hi,
>>>>      >     >
>>>>      >     > I would like to clarify migration path to 5.0.0
>>>        from 4.X.X
>>>>      clusters.
>>>>      >     > For all Spectrum Scale clusters that are
>>>        currently at 4.X.X,
>>>>      it is
>>>>      >     > possible to migrate to 5.0.0 with no offline
>>>        data migration
>>>>      and no
>>>>      >     > need to move data. Once these clusters are at
>>>        5.0.0, they will
>>>>      >     > benefit from the performance improvements, new
>>>        features (such as
>>>>      >     > file audit logging), and various enhancements
>>>        that are
>>>>      included in
>>>>      >     5.0.0.
>>>>      >     >
>>>>      >     > That being said, there is one enhancement that
>>>        will not be
>>>>      applied
>>>>      >     > to these clusters, and that is the increased
>>>        number of
>>>>      sub-blocks
>>>>      >     > per block for small file allocation. This means
>>>        that for file
>>>>      >     > systems with a large block size and a lot of
>>>        small files, the
>>>>      >     > overall space utilization will be the same it
>>>        currently is
>>>>      in 4.X.X.
>>>>      >     > Since file systems created at 4.X.X and earlier
>>>        used a block
>>>>      size
>>>>      >     > that kept this allocation in mind, there should
>>>        be very little
>>>>      >     > impact on existing file systems.
>>>>      >     >
>>>>      >     > Outside of that one particular function, the
>>>        remainder of the
>>>>      >     > performance improvements, metadata improvements,
>>>        updated
>>>>      >     > compatibility, new functionality, and all of the
>>>        other
>>>>      enhancements
>>>>      >     > will be immediately available to you once you
>>>        complete the
>>>>      upgrade
>>>>      >     > to 5.0.0 -- with no need to reformat, move data,
>>>        or take
>>>>      your data
>>>>      >     offline.
>>>>      >     >
>>>>      >     > I hope that clarifies things a little and makes
>>>        the upgrade path
>>>>      >     > more accessible.
>>>>      >     >
>>>>      >     > Please let me know if there are any other
>>>        questions or concerns.
>>>>      >     >
>>>>      >     > Thank you,
>>>>      >     > Nikhil Khandelwal
>>>>      >     > Spectrum Scale Development
>>>>      >     > Client Adoption
>>>>      >     >
>>>>      >     > _______________________________________________
>>>>      >     > gpfsug-discuss mailing list
>>>>      >     > gpfsug-discuss at _spectrumscale.org_
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>>>      >     > _https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>>>>      >
>>>>      >     > _______________________________________________
>>>>      >     > gpfsug-discuss mailing list
>>>>      >     > gpfsug-discuss at _spectrumscale.org_
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>>>      >
>>>>      >     > _https://urldefense.proofpoint.com/v2/url?_
>>>>      <https://urldefense.proofpoint.com/v2/url?_>
>>>>      >     >
>>>>      >   
>>>         u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>>>>      >     >
>>>>      >   
>>>         siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>>>>      >
>>>>      >     _______________________________________________
>>>>      >     gpfsug-discuss mailing list
>>>>      >     gpfsug-discuss at _spectrumscale.org_
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>>>>      >     __https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>>>>      >     gpfsug-discuss mailing list
>>>>      >     gpfsug-discuss at spectrumscale.org
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>>      >   
>>>>    
>>>          https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>>>>      >
>>>>      >
>>>>      >
>>>>      >
>>>>      > _______________________________________________
>>>>      > gpfsug-discuss mailing list
>>>>      > gpfsug-discuss at spectrumscale.org
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>>      > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=>
>>>>      >
>>>> 
>>>>      --
>>>>      Aaron Knister
>>>>      NASA Center for Climate Simulation (Code 606.2)
>>>>      Goddard Space Flight Center
>>>>      (301) 286-2776 <tel:(301)%20286-2776>
>>>        <tel:%28301%29%20286-2776>
>>>>      _______________________________________________
>>>>      gpfsug-discuss mailing list
>>>>      gpfsug-discuss at spectrumscale.org
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>>      https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=>
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> gpfsug-discuss mailing list
>>>> gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>>> 
>>> 
>>>        --
>>>        Aaron Knister
>>>        NASA Center for Climate Simulation (Code 606.2)
>>>        Goddard Space Flight Center
>>>        (301) 286-2776 <tel:(301)%20286-2776>
>>>        _______________________________________________
>>>        gpfsug-discuss mailing list
>>>        gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>        https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>> 
>>    _______________________________________________
>>    gpfsug-discuss mailing list
>>    gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>    https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>> 
>> 
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>> 
> 
> -- 
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
> 
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/facaf6ea/attachment-0001.htm>

From kh.atmane at gmail.com  Thu Dec 21 12:41:49 2017
From: kh.atmane at gmail.com (atmane)
Date: Thu, 21 Dec 2017 13:41:49 +0100
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 49
In-Reply-To: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
References: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
Message-ID: <op.zblj74uypgw25x@pc-atm>

Hi
This Is My Config

mmlsconfig | grep verbsRdma
verbsRdma enable
mmlsconfig | grep verbsPorts
verbsPorts mlx4_0/1 mlx4_0/2 mlx4_1/1 mlx4_1/2
ibstat | grep -E "CA|Port"

CA 'mlx4_0'
         CA type: MT4099
         Port 1:
                 Port GUID: 0x0002c903001a1571
         Port 2:
                 Port GUID: 0x0002c903001a1572
CA 'mlx4_1'
         CA type: MT4099
         Port 1:
                 Port GUID: 0x0002c903001a15c1
         Port 2:
                 Port GUID: 0x0002c903001a15c2
ethtool ib0 |grep "Link detected"
Link detected: yes
ethtool ib1 |grep "Link detected"
Link detected: yes
ethtool ib2 |grep "Link detected"
Link detected: yes
ethtool ib3 |grep "Link detected"
Link detected: yes
ifconfig ib0 |grep inet
inet 192.168.100.3 netmask 255.255.255.0  broadcast 192.168.100.255
ifconfig ib1 |grep inet
Infiniband hardware address can be incorrect! Please read BUGS section in  
ifconfig(8).
ifconfig ib2 |grep inet
Infiniband hardware address can be incorrect! Please read BUGS section in  
ifconfig(8).
ifconfig ib3 |grep inet
Infiniband hardware address can be incorrect! Please read BUGS section in  
ifconfig(8).
ip addr show ib1
8: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
1024
     link/infiniband  
a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:72 brd  
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
9: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
1024
     link/infiniband  
a0:00:02:20:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c1 brd  
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

10: ib3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP  
qlen 1024
     link/infiniband  
a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c2 brd  
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff






Le Thu, 21 Dec 2017 13:00:01 +0100,  
<gpfsug-discuss-request at spectrumscale.org> a ?crit:

> Send gpfsug-discuss mailing list submissions to
> 	gpfsug-discuss at spectrumscale.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
> 	gpfsug-discuss-request at spectrumscale.org
>
> You can reach the person managing the list at
> 	gpfsug-discuss-owner at spectrumscale.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: more than one mlx connectx-4 adapter in same	host
>       (Frank Kraemer)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 21 Dec 2017 08:07:24 +0100
> From: "Frank Kraemer" <kraemerf at de.ibm.com>
> To: gpfsug-discuss at spectrumscale.org
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in
> 	same	host
> Message-ID:
> 	<OFE99AF1B2.2CCA57A0-ONC12581FD.00272158-C12581FD.00272158 at notes.na.collabserv.com>
> 	
> Content-Type: text/plain; charset="us-ascii"
>
> David,
>
>> We're trying to get 40 gbe connection between Mellanox switches and  
>> Cisco
>> switches down at the other end of the machine room
>> The BiDi part seems to be the best given about 30m  run on multimode.
>> However Mellanox support says it's not supported.
>> Want to use this to get close to IB speeds for GPFS on nodes that aren't
>> on the IB fabric.
>> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
>> switches are different brands?
>
> maybe that's a good reason to get in contact with the team from  
> Interoptic.
> They claim a good expertise for these kind of problems - feedback is  
> good.
> http://packetpushers.net/podcast/podcasts/show-360-all-about-optics-interoptic-sponsored/
> https://interoptic.com/
>
> Frank Kraemer
> IBM Consulting IT Specialist  / Client Technical Architect
> Am Weiher 24, 65451 Kelsterbach
> mailto:kraemerf at de.ibm.com
> voice: +49-(0)171-3043699 / +4970342741078
> IBM Germany
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:  
> <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171221/31aeae52/attachment-0001.html>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 71, Issue 49
> **********************************************


-- 
Utilisant le logiciel de courrier d'Opera : http://www.opera.com/mail/


From a.khiredine at meteo.dz  Thu Dec 21 14:02:40 2017
From: a.khiredine at meteo.dz (atmane khiredine)
Date: Thu, 21 Dec 2017 14:02:40 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 49
In-Reply-To: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
References: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
Message-ID: <4B32CB5C696F2849BDEF7DF9EACE884B72ACDD56@SDEB-EXC02.meteo.dz>



Atmane Khiredine
HPC System Administrator | Office National de la M?t?orologie
T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz

________________________________________
De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org]
Envoy? : jeudi 21 d?cembre 2017 13:00
? : gpfsug-discuss at spectrumscale.org
Objet : gpfsug-discuss Digest, Vol 71, Issue 49

Send gpfsug-discuss mailing list submissions to
        gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
        gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
        gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Re: more than one mlx connectx-4 adapter in same  host
      (Frank Kraemer)

  This Is My Config
    mmlsconfig | grep verbsRdma
    verbsRdma enable
    mmlsconfig | grep verbsPorts
    verbsPorts mlx4_0/1 mlx4_0/2 mlx4_1/1 mlx4_1/2
    ibstat | grep -E "CA|Port"
    CA 'mlx4_0'
             CA type: MT4099
             Port 1:
                     Port GUID: 0x0002c903001a1571
             Port 2:
                     Port GUID: 0x0002c903001a1572
    CA 'mlx4_1'
             CA type: MT4099
             Port 1:
                     Port GUID: 0x0002c903001a15c1
             Port 2:
                     Port GUID: 0x0002c903001a15c2
    ethtool ib0 |grep "Link detected"
    Link detected: yes
    ethtool ib1 |grep "Link detected"
    Link detected: yes
 

    ethtool ib2 |grep "Link detected"
    Link detected: yes
    ethtool ib3 |grep "Link detected"
    Link detected: yes
    ifconfig ib0 |grep inet
    inet 192.168.100.3 netmask 255.255.255.0  broadcast 192.168.100.255
    ifconfig ib1 |grep inet
    Infiniband hardware address can be incorrect! Please read BUGS section in  
    ifconfig(8).
    ifconfig ib2 |grep inet
    Infiniband hardware address can be incorrect! Please read BUGS section in  
    ifconfig(8).
    ifconfig ib3 |grep inet
    Infiniband hardware address can be incorrect! Please read BUGS section in  
    ifconfig(8).
    ip addr show ib1
    8: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
    1024
         link/infiniband  
    a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:72 brd  
    00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    9: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
    1024
         link/infiniband  
    a0:00:02:20:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c1 brd  
    00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
 
    10: ib3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP  
    qlen 1024
         link/infiniband  
    a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c2 brd  
    00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

Message: 1
Date: Thu, 21 Dec 2017 08:07:24 +0100
From: "Frank Kraemer" <kraemerf at de.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in
        same    host
Message-ID:
        <OFE99AF1B2.2CCA57A0-ONC12581FD.00272158-C12581FD.00272158 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

David,

> We're trying to get 40 gbe connection between Mellanox switches and Cisco
> switches down at the other end of the machine room
> The BiDi part seems to be the best given about 30m  run on multimode.
> However Mellanox support says it's not supported.
> Want to use this to get close to IB speeds for GPFS on nodes that aren't
> on the IB fabric.
> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
> switches are different brands?

maybe that's a good reason to get in contact with the team from Interoptic.
They claim a good expertise for these kind of problems - feedback is good.
http://packetpushers.net/podcast/podcasts/show-360-all-about-optics-interoptic-sponsored/
https://interoptic.com/

Frank Kraemer
IBM Consulting IT Specialist  / Client Technical Architect
Am Weiher 24, 65451 Kelsterbach
mailto:kraemerf at de.ibm.com
voice: +49-(0)171-3043699 / +4970342741078
IBM Germany
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171221/31aeae52/attachment-0001.html>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 71, Issue 49
**********************************************


From oehmes at gmail.com  Thu Dec 21 16:38:27 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Thu, 21 Dec 2017 16:38:27 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
References: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
	<OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
Message-ID: <CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>

Daniel,

while this might be easier to think about it, its not true :-)
lets just use an example.  a disk drive can do 100 io's per second with
128kb random writes and 80 iops with 256kb writes . now lets do the math
with a 8+2p setup for each of the 2 cases. this means you can do 100 times
1mb writes (8*128k) or 80 times 2 mb writes so 100 MB/sec or 160 MB/sec
with the exact same drives. given you can fit 2 times as many subblocks
into the 2mb block you would gain 60% of speed by just going to this larger
size. so if you now go to a 16MB blocksize and you have just 50 iops @ 2MB
each you can write ~800 MB/sec with the exact same setup and same size
small writes, that's a factor of 8 .
so i/o size AND number of subblocks matter.

Sven



On Thu, Dec 21, 2017 at 12:22 PM Daniel Kidger <daniel.kidger at uk.ibm.com>
wrote:

> My suggestion is that it is better to not think of the performance coming
> from having more than 32 sub-blocks but instead that the performance comes
> from smaller sub-blocks. The fact that there are now more of them in say a
> 4MB blocksize filesytem is just a side effect.
>
> Daniel
> [image: /spectrum_storage-banne]
>
>
> [image: Spectrum Scale Logo]
>
>
> *Dr Daniel Kidger*
> IBM Technical Sales Specialist
> Software Defined Solution Sales
>
> + <+%2044-7818%20522%20266> 44-(0)7818 522 266 <+%2044-7818%20522%20266>
> daniel.kidger at uk.ibm.com
>
> On 19 Dec 2017, at 21:32, Aaron Knister <aaron.s.knister at nasa.gov> wrote:
>
> Thanks, Sven. Understood!
>
> On 12/19/17 3:20 PM, Sven Oehme wrote:
>
> Hi,
>
>
> the zero padding was never promoted into a GA stream, it was an
>
> experiment to proof we are on the right track when we eliminate the
>
> overhead from client to NSD Server, but also showed that alone is not
>
> good enough. the work for the client is the same compared to the >32
>
> subblocks, but the NSD Server has more work as it can't pack as many
>
> subblocks and therefore files into larger blocks, so you need to do more
>
> writes to store the same number of files.
>
> thats why there is the additional substantial improvement  when we then
>
> went to >32 subblocks.
>
>
> sven
>
>
> On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
>
> SCIENCE CORP] <aaron.s.knister at nasa.gov
>
> <mailto:aaron.s.knister at nasa.gov <aaron.s.knister at nasa.gov>>> wrote:
>
>
>    Thanks Sven! That makes sense to me and is what I thought was the
>
>    case which is why I was confused when I saw the reply to the thread
>
>    that said the >32 subblocks code had no performance impact.
>
>
>    A couple more question for you? in your presentation there?s a
>
>    benchmark that shows the file create performance without the zero
>
>    padding. Since you mention this is done for security reasons was
>
>    that feature ever promoted to a GA Scale release? I?m also wondering
>
>    if you could explain the performance difference between the no zero
>
>    padding code and the > 32 subblock code since given your the example
>
>    of 32K files and 16MB block size I figure both cases ought to write
>
>    the same amount to disk.
>
>
>    Thanks!
>
>
>    -Aaron
>
>
>
>
>
>
>    On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com
>
>    <mailto:oehmes at gmail.com <oehmes at gmail.com>>> wrote:
>
>    i thought i answered that already, but maybe i just thought about
>
>    answering it and then forgot about it :-D
>
>
>    so yes more than 32 subblocks per block significant increase the
>
>    performance of filesystems with small files, for the sake of the
>
>    argument let's say 32k in a large block filesystem again for sake
>
>    of argument say 16MB.
>
>
>    you probably ask why ?
>
>
>    if you create a file and write 32k into it in a pre 5.0.0 Version
>
>    16 MB filesystem your client actually doesn't write 32k to the NSD
>
>    Server, it writes 512k, because thats the subblock size and we
>
>    need to write the full subblock (for security reasons). so first
>
>    you waste significant memory on the client to cache that zero
>
>    padding, you waste network bandwidth and you waste NSD Server
>
>    cache because you store it there too. this means you overrun the
>
>    cache more quickly, means you start doing read/modify writes
>
>    earlier on all your nice large raid tracks... i guess you get the
>
>    story by now.
>
>
>    in fact,  if you have a good raid code that can drive really a lot
>
>    of bandwidth out of individual drives like a GNR system you get
>
>    more performance for small file writes as larger your blocksize
>
>    is, because we can 'pack' more files into larger i/os and
>
>    therefore turn a small file create workload into a bandwidth
>
>    workload, essentially exactly what we did and i demonstrated in
>
>    the CORAL presentation .
>
>
>    hope that makes this crystal clear now .
>
>
>    sven
>
>
>
>
>    On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister
>
>    <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov
> <aaron.s.knister at nasa.gov>>> wrote:
>
>
>        Thanks, Alex. I'm all too familiar with the trade offs between
>
>        large
>
>        blocks and small files and we do use pretty robust SSD storage
>
>        for our
>
>        metadata. We support a wide range of workloads and we have
>
>        some folks
>
>        with many small (<1M) files and other folks with many large
>
>        (>256MB) files.
>
>
>        My point in this thread is that IBM has said over and over
>
>        again in
>
>        presentations that there is a significant performance gain
>
>        with the >32
>
>        subblocks code on filesystems with large block sizes (although
>
>        to your
>
>        point I'm not clear on exactly what large means since I didn't
>
>        define
>
>        large in this context). Therefore given that the >32 subblock
>
>        code gives
>
>        a significant performance gain one could reasonably assume
>
>        that having a
>
>        filesystem with >32 subblocks is required to see this gain
>
>        (rather than
>
>        just running the >32 subblocks code on an fs w/o > 32 subblocks).
>
>
>        This lead me to ask about a migration tool because in my mind
>
>        if there's
>
>        a performance gain from having >32 subblocks on the FS I'd
>
>        like that
>
>        feature and having to manually copy 10's of PB to new hardware
>
>        to get
>
>        this performance boost is unacceptable. However, IBM can't
>
>        seem to make
>
>        up their mind about whether or not the >32 subblocks code
>
>        *actually*
>
>        provides a performance increase. This seems like a pretty
>
>        straightforward question.
>
>
>        -Aaron
>
>
>        On 12/15/17 3:48 PM, Alex Chekholko wrote:
>
> Hey Aaron,
>
>
> Can you define your sizes for "large blocks" and "small
>
>        files"?  If you
>
> dial one up and the other down, your performance will be
>
>        worse.  And in
>
> any case it's a pathological corner case so it shouldn't
>
>        matter much for
>
> your workflow, unless you've designed your system with the
>
>        wrong values.
>
>
> For example, for bioinformatics workloads, I prefer to use 256KB
>
> filesystem block size, and I'd consider 4MB+ to be "large
>
>        block size",
>
> which would make the filesystem obviously unsuitable for
>
>        processing
>
> millions of 8KB files.
>
>
> You can make a histogram of file sizes in your existing
>
>        filesystems and
>
> then make your subblock size (1/32 of block size) on the
>
>        smaller end of
>
> that.   Also definitely use the "small file in inode"
>
>        feature and put
>
> your metadata on SSD.
>
>
> Regards,
>
> Alex
>
>
> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
>
> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov
> <aaron.s.knister at nasa.gov>>
>
>        <mailto:aaron.s.knister at nasa.gov <aaron.s.knister at nasa.gov>
>
>        <mailto:aaron.s.knister at nasa.gov <aaron.s.knister at nasa.gov>>>>
> wrote:
>
>
>      Thanks, Bill.
>
>
>      I still don't feel like I've got an clear answer from
>
>        IBM and frankly
>
>      the core issue of a lack of migration tool was totally
>
>        dodged.
>
>
>      Again in Sven's presentation from SSUG @ SC17
>
>
>
>         (
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=
>
>        <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=
> >)
>
>      he mentions "It has a significant performance penalty
>
>        for small files in
>
>      large block size filesystems" and the demonstrates that
>
>        with several
>
>      mdtest runs (which show the effect with and without the >32
>
>      subblocks code):
>
>
>
>      4.2.1 base code - SUMMARY: (of 3 iterations)
>
>      File creation : Mean = 2237.644
>
>
>      zero-end-of-file-padding (4.2.2 + ifdef for zero
>
>        padding):  SUMMARY: (of
>
>      3 iterations)
>
>      File creation : Mean = 12866.842
>
>
>      more sub blocks per block (4.2.2 + morethan32subblock code):
>
>      File creation : Mean = 40316.721
>
>
>      Can someone (ideally Sven) give me a straight answer as
>
>        to whether or
>
>      not the > 32 subblock code actually makes a performance
>
>        difference for
>
>      small files in large block filesystems? And if not, help
>
>        me understand
>
>      why his slides and provided benchmark data have
>
>        consistently indicated
>
>      it does?
>
>
>      -Aaron
>
>
>      On 12/1/17 11:44 AM, Bill Hartner wrote:
>
>      > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported
>
>        some of the
>
>      > results @ user group meeting.
>
>      >
>
>      > -Bill
>
>      >
>
>      > Bill Hartner
>
>      > IBM Systems
>
>      > Scalable I/O Development
>
>      > Austin, Texas
>
>      > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com
> <bhartner at us.ibm.com>>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com> <
> mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>>
>
>      > home office 512-784-0980 <(512)%20784-0980> <tel:(512)%20784-0980>
>
>        <tel:512-784-0980 <512-784-0980> <tel:(512)%20784-0980>>
>
>      >
>
>      >
>
>      > Inactive hide details for Jan-Frode Myklebust
>
>        ---12/01/2017 06:53:44
>
>      > AM---Bill, could you say something about what the
>
>        metadataJan-Frode
>
>      > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you
>
>        say something
>
>      > about what the metadata-storage here was?
>
>        ESS/NL-SAS/3way replication?
>
>      >
>
>      > From: Jan-Frode Myklebust <janfrode at tanso.net
>
>        <mailto:janfrode at tanso.net <janfrode at tanso.net>> <
> mailto:janfrode at tanso.net <janfrode at tanso.net>
>
>        <mailto:janfrode at tanso.net <janfrode at tanso.net>>>>
>
>      > To: gpfsug main discussion list
>
>        <gpfsug-discuss at spectrumscale.org
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>>>
>
>      > Date: 12/01/2017 06:53 AM
>
>      > Subject: Re: [gpfsug-discuss] Online data migration tool
>
>      > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>>
>
>      >
>
>      >
>
>
>
>
>         ------------------------------------------------------------------------
>
>      >
>
>      >
>
>      >
>
>      > Bill, could you say something about what the
>
>        metadata-storage here was?
>
>      > ESS/NL-SAS/3way replication?
>
>      >
>
>      > I just asked about this in the internal slack channel
>
>        #scale-help today..
>
>      >
>
>      >
>
>      >
>
>      > -jf
>
>      >
>
>      > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
>
>        <_bhartner at us.ibm.com_
>
>      > <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>> <
> mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>>>>:
>
>      >
>
>      >     > "It has a significant performance penalty for
>
>        small files in
>
>      large
>
>      >     > block size filesystems"
>
>      >
>
>      >     Aaron,
>
>      >
>
>      >     Below are mdtest results for a test we ran for
>
>        CORAL - file
>
>      size was
>
>      >     32k.
>
>      >
>
>      >     We have not gone back and ran the test on a file
>
>        system formatted
>
>      >     without > 32 subblocks. We'll do that at some point...
>
>      >
>
>      >     -Bill
>
>      >
>
>      >     -- started at 10/28/2017 17:51:38 --
>
>      >
>
>      >     mdtest-1.9.3 was launched with 228 total task(s)
>
>        on 12 node(s)
>
>      >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>
>      >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w
>
>        32768 -C -F -r
>
>      -p 360
>
>      >     -u -y
>
>      >     Path: /ibm/fs2-16m-10
>
>      >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used
>
>        Inodes: 0.0%
>
>      >
>
>      >     228 tasks, 67239936 files
>
>      >
>
>      >     SUMMARY: (of 3 iterations)
>
>      >     Operation Max Min Mean Std Dev
>
>      >     --------- --- --- ---- -------
>
>      >     File creation : 51953.498 50558.517 51423.221 616.643
>
>      >     File stat : 0.000 0.000 0.000 0.000
>
>      >     File read : 0.000 0.000 0.000 0.000
>
>      >     File removal : 96746.376 92149.535 94658.774 1900.187
>
>      >     Tree creation : 1.588 0.070 0.599 0.700
>
>      >     Tree removal : 0.213 0.034 0.097 0.082
>
>      >
>
>      >     -- finished at 10/28/2017 19:51:54 --
>
>      >
>
>      >     Bill Hartner
>
>      >     IBM Systems
>
>      >     Scalable I/O Development
>
>      >     Austin, Texas_
>
>      >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
> <bhartner at us.ibm.com>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>
>
>      <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com> <
> mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>>>
>
>      >     home office 512-784-0980 <(512)%20784-0980>
> <tel:(512)%20784-0980>
>
>        <tel:512-784-0980 <512-784-0980> <tel:(512)%20784-0980>>
>
>      >
>
>      >     _
>
>      >     __gpfsug-discuss-bounces at spectrumscale.org_
>
>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>>> wrote on
>
>      >     11/29/2017 04:41:48 PM:
>
>      >
>
>      >     > From: Aaron Knister <_aaron.knister at gmail.com_
>
>      >     <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>
>
>        <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>>
>
>        <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>
>
>        <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>>>>>
>
>      >
>
>      >
>
>      >     > To: gpfsug main discussion list
>
>      >     <_gpfsug-discuss at spectrumscale.org_
>
>      >     <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>>>>
>
>      >
>
>      >     > Date: 11/29/2017 04:42 PM
>
>      >
>
>      >
>
>      >     > Subject: Re: [gpfsug-discuss] Online data
>
>        migration tool
>
>      >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>
>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>>>
>
>      >
>
>      >     >
>
>      >
>
>      >     > Thanks, Nikhil. Most of that was consistent with
>
>        my understnading,
>
>      >     > however I was under the impression that the >32
>
>        subblocks code is
>
>      >     > required to achieve the touted 50k file
>
>        creates/second that Sven has
>
>      >     > talked about a bunch of times:
>
>      >     >
>
>      >     >
>
>      >
>
>
>
>          _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=
> >
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >>
>
>      >     >
>
>      >
>
>
>
>          _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=
> >
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >>
>
>      >     > _Sven_Oehme_-_News_from_Research.pdf
>
>      >     >
>
>        _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=
>
>      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=
> >
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >>
>
>      >     >
>
>        _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>
>      >
>
>      >
>
>      >     > from those presentations regarding 32 subblocks:
>
>      >     >
>
>      >     > "It has a significant performance penalty for
>
>        small files in large
>
>      >     > block size filesystems"
>
>      >
>
>      >     > although I'm not clear on the specific
>
>        definition of "large". Many
>
>      >     > filesystems I encounter only have a 1M block
>
>        size so it may not
>
>      >     > matter there, although that same presentation
>
>        clearly shows the
>
>      >     > benefit of larger block sizes which is yet
>
>        *another* thing for which
>
>      >     > a migration tool would be helpful.
>
>      >
>
>      >     > -Aaron
>
>      >     >
>
>      >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>
>      >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
> <nikhilk at us.ibm.com>
>
>        <mailto:nikhilk at us.ibm.com <nikhilk at us.ibm.com>>
>
>      <mailto:nikhilk at us.ibm.com <nikhilk at us.ibm.com>
>
>        <mailto:nikhilk at us.ibm.com <nikhilk at us.ibm.com>>>>> wrote:
>
>      >
>
>      >     > Hi,
>
>      >     >
>
>      >     > I would like to clarify migration path to 5.0.0
>
>        from 4.X.X
>
>      clusters.
>
>      >     > For all Spectrum Scale clusters that are
>
>        currently at 4.X.X,
>
>      it is
>
>      >     > possible to migrate to 5.0.0 with no offline
>
>        data migration
>
>      and no
>
>      >     > need to move data. Once these clusters are at
>
>        5.0.0, they will
>
>      >     > benefit from the performance improvements, new
>
>        features (such as
>
>      >     > file audit logging), and various enhancements
>
>        that are
>
>      included in
>
>      >     5.0.0.
>
>      >     >
>
>      >     > That being said, there is one enhancement that
>
>        will not be
>
>      applied
>
>      >     > to these clusters, and that is the increased
>
>        number of
>
>      sub-blocks
>
>      >     > per block for small file allocation. This means
>
>        that for file
>
>      >     > systems with a large block size and a lot of
>
>        small files, the
>
>      >     > overall space utilization will be the same it
>
>        currently is
>
>      in 4.X.X.
>
>      >     > Since file systems created at 4.X.X and earlier
>
>        used a block
>
>      size
>
>      >     > that kept this allocation in mind, there should
>
>        be very little
>
>      >     > impact on existing file systems.
>
>      >     >
>
>      >     > Outside of that one particular function, the
>
>        remainder of the
>
>      >     > performance improvements, metadata improvements,
>
>        updated
>
>      >     > compatibility, new functionality, and all of the
>
>        other
>
>      enhancements
>
>      >     > will be immediately available to you once you
>
>        complete the
>
>      upgrade
>
>      >     > to 5.0.0 -- with no need to reformat, move data,
>
>        or take
>
>      your data
>
>      >     offline.
>
>      >     >
>
>      >     > I hope that clarifies things a little and makes
>
>        the upgrade path
>
>      >     > more accessible.
>
>      >     >
>
>      >     > Please let me know if there are any other
>
>        questions or concerns.
>
>      >     >
>
>      >     > Thank you,
>
>      >     > Nikhil Khandelwal
>
>      >     > Spectrum Scale Development
>
>      >     > Client Adoption
>
>      >     >
>
>      >     > _______________________________________________
>
>      >     > gpfsug-discuss mailing list
>
>      >     > gpfsug-discuss at _spectrumscale.org_
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
>
>      >     > _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
>
>      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
> >
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>
>
>      >
>
>      >     > _______________________________________________
>
>      >     > gpfsug-discuss mailing list
>
>      >     > gpfsug-discuss at _spectrumscale.org_
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/6513c936/attachment-0001.htm>

From valdis.kletnieks at vt.edu  Thu Dec 21 20:35:26 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Thu, 21 Dec 2017 15:35:26 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>
References: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
	<OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
	<CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>
Message-ID: <37655.1513888526@turing-police.cc.vt.edu>

On Thu, 21 Dec 2017 16:38:27 +0000, Sven Oehme said:

> size. so if you now go to a 16MB blocksize and you have just 50 iops @ 2MB
> each you can write ~800 MB/sec with the exact same setup and same size
> small writes, that's a factor of 8 .

That's assuming your metadata storage is able to handle open/read/write/close
on enough small files per second to push 800MB/sec.  If you're talking 128K subblocks,
you're going to need some 6,400 small files per second to fill that pipe...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/385aeb17/attachment-0001.sig>

From oehmes at gmail.com  Fri Dec 22 00:02:43 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Fri, 22 Dec 2017 00:02:43 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <37655.1513888526@turing-police.cc.vt.edu>
References: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
	<OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
	<CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>
	<37655.1513888526@turing-police.cc.vt.edu>
Message-ID: <CALssuR0Pk=ck9rT-OTvp3SEUroDTDj2bcxFGDpHzRX1wztSjHQ@mail.gmail.com>

thats not how GPFS aehm Scale works :-)
each client has pre-allocated inodes in memory and creating files is a
matter of spooling records. yes, eventually you need to destage this to the
disk, but that happens only every few seconds and given this i/os are
usually very colocated so good storage cache technology can reduce i/os to
physical media significant.

to proof the point look at this numbers :
-- started at 10/17/2017 14:29:13 --

mdtest-1.9.3 was launched with 110 total task(s) on 11 node(s)
Command line used: /ghome/oehmes/mpi/bin/mdtest-pcmpi9131-existingdir -d
/ibm/fs2-16m-09/shared/mdtest-ec -i 1 -n 10000 -F -w 0 -Z -p 8 -N 11 -u
Path: /ibm/fs2-16m-09/shared
FS: 128.1 TiB   Used FS: 0.2%   Inodes: 476.8 Mi   Used Inodes: 0.0%

110 tasks, 1100000 files

SUMMARY: (of 1 iterations)
   Operation                      Max            Min           Mean
Std Dev
   ---------                      ---            ---           ----
-------
   File creation     :     444221.343     444221.343     444221.343
  0.000
   File stat         :    6704498.841    6704498.841    6704498.841
  0.000
   File read         :    3859105.596    3859105.596    3859105.596
  0.000
   File removal      :     409336.606     409336.606     409336.606
  0.000
   Tree creation     :          5.344          5.344          5.344
  0.000
   Tree removal      :          1.145          1.145          1.145
  0.000

-- finished at 10/17/2017 14:29:27 --

this is a run against a 16mb blocksize filesystem with only spinning disks
(just one GL6 ESS) , not a single SSD and as you can see , this system on
11 nodes produces 444k creates / second something far above and beyond of
what drives can do.

and yes i know this stuff is all very complicated and not easy to explain
:-)

sven


On Thu, Dec 21, 2017 at 8:35 PM <valdis.kletnieks at vt.edu> wrote:

> On Thu, 21 Dec 2017 16:38:27 +0000, Sven Oehme said:
>
> > size. so if you now go to a 16MB blocksize and you have just 50 iops @
> 2MB
> > each you can write ~800 MB/sec with the exact same setup and same size
> > small writes, that's a factor of 8 .
>
> That's assuming your metadata storage is able to handle
> open/read/write/close
> on enough small files per second to push 800MB/sec.  If you're talking
> 128K subblocks,
> you're going to need some 6,400 small files per second to fill that pipe...
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/a66aca84/attachment-0001.htm>

From damir.krstic at gmail.com  Fri Dec 22 17:44:50 2017
From: damir.krstic at gmail.com (Damir Krstic)
Date: Fri, 22 Dec 2017 17:44:50 +0000
Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group without
	takeover
Message-ID: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>

It's been a very frustrating couple of months with our 2 ESS systems. IBM
tells us we had blueflame bug and they came on site and updated our ESS to
the latest version back in middle of November. Wednesday night one of the
NSD servers in one of our ESS building blocks kernel panicked. No idea why
and none of the logs are insightful. We have a PMR open with IBM. I am not
very confident we will get to the bottom of what's causing kernel panics on
our IO servers. The system has gone down over 4 times now in 2 months.

When we tried brining it back up, it rejoined the recovery group and the IO
on the entire cluster locked up until we were able to find couple of
compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on
those nodes resolved the issue of the filesystem locking up.

So far we have never been successful in brining back an IO server and not
having a filesystem lock up until we find a node with pending state with
tscomm. Anyway, the system was stable for few minutes until the same IO
server that went down on Wednesday night went into an arbitrating mode. It
never recovered. We stopped gpfs on that server and IO recovered again. We
left gpfs down and cluster seems to be OK.

My question is, is there a way of brining back the IO server into the mix
without the recoverygroup takeover happening? Could I just start a gpfs and
have it back in the mix as a backup server for the recoverygroup and if so,
how do you do that. Right now that server is designated as primary server
for the recovery group. I would like to have both IO servers in the mix for
redundancy purposes.

This ESS situation is beyond frustrating and I don't see end in sight.

Any help is appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/4d155e13/attachment-0001.htm>

From janfrode at tanso.net  Fri Dec 22 17:55:39 2017
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 22 Dec 2017 17:55:39 +0000
Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group
	without takeover
In-Reply-To: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
References: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
Message-ID: <CAHwPatgUFx4GHitKsmiSWZJC=20FCVBoMgmaDnTK-8zUbj4J6g@mail.gmail.com>

Can?t you just reverse the mmchrecoverygroup --servers order, before
starting the io-server?


-jf
fre. 22. des. 2017 kl. 18:45 skrev Damir Krstic <damir.krstic at gmail.com>:

> It's been a very frustrating couple of months with our 2 ESS systems. IBM
> tells us we had blueflame bug and they came on site and updated our ESS to
> the latest version back in middle of November. Wednesday night one of the
> NSD servers in one of our ESS building blocks kernel panicked. No idea why
> and none of the logs are insightful. We have a PMR open with IBM. I am not
> very confident we will get to the bottom of what's causing kernel panics on
> our IO servers. The system has gone down over 4 times now in 2 months.
>
> When we tried brining it back up, it rejoined the recovery group and the
> IO on the entire cluster locked up until we were able to find couple of
> compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on
> those nodes resolved the issue of the filesystem locking up.
>
> So far we have never been successful in brining back an IO server and not
> having a filesystem lock up until we find a node with pending state with
> tscomm. Anyway, the system was stable for few minutes until the same IO
> server that went down on Wednesday night went into an arbitrating mode. It
> never recovered. We stopped gpfs on that server and IO recovered again. We
> left gpfs down and cluster seems to be OK.
>
> My question is, is there a way of brining back the IO server into the mix
> without the recoverygroup takeover happening? Could I just start a gpfs and
> have it back in the mix as a backup server for the recoverygroup and if so,
> how do you do that. Right now that server is designated as primary server
> for the recovery group. I would like to have both IO servers in the mix for
> redundancy purposes.
>
> This ESS situation is beyond frustrating and I don't see end in sight.
>
> Any help is appreciated.
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/d7a7cb53/attachment-0001.htm>

From alex at calicolabs.com  Fri Dec 22 18:01:58 2017
From: alex at calicolabs.com (Alex Chekholko)
Date: Fri, 22 Dec 2017 10:01:58 -0800
Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group
	without takeover
In-Reply-To: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
References: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
Message-ID: <CANcy_Pb8JnZG+nvhuDGFb_X+aKPerzwu74iM7VmVwvFckak9zQ@mail.gmail.com>

Hi Damir,

I'm not sure whether this applies to you, but this was my experience.

GPFS absolutely depends on a reliable network interconnect.  If anything
goes wrong on the network layer, GPFS may not be able to recover.

Do you have visibility and monitoring on all the low-level network counters
on all the relevant network interfaces?

e.g. if one of your clients is connected to a switch port that is flaky in
some way and some GPFS message goes unacked, you can get unusual client
state and then the whole cluster hangs...

In my case years ago we ended up replacing some flaky HP switches and all
our GPFS troubles went away!  And we were able to show those switches had
some issues when doing non-GPFS testing with iperf, etc.

Regards,
Alex

On Fri, Dec 22, 2017 at 9:44 AM, Damir Krstic <damir.krstic at gmail.com>
wrote:

> It's been a very frustrating couple of months with our 2 ESS systems. IBM
> tells us we had blueflame bug and they came on site and updated our ESS to
> the latest version back in middle of November. Wednesday night one of the
> NSD servers in one of our ESS building blocks kernel panicked. No idea why
> and none of the logs are insightful. We have a PMR open with IBM. I am not
> very confident we will get to the bottom of what's causing kernel panics on
> our IO servers. The system has gone down over 4 times now in 2 months.
>
> When we tried brining it back up, it rejoined the recovery group and the
> IO on the entire cluster locked up until we were able to find couple of
> compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on
> those nodes resolved the issue of the filesystem locking up.
>
> So far we have never been successful in brining back an IO server and not
> having a filesystem lock up until we find a node with pending state with
> tscomm. Anyway, the system was stable for few minutes until the same IO
> server that went down on Wednesday night went into an arbitrating mode. It
> never recovered. We stopped gpfs on that server and IO recovered again. We
> left gpfs down and cluster seems to be OK.
>
> My question is, is there a way of brining back the IO server into the mix
> without the recoverygroup takeover happening? Could I just start a gpfs and
> have it back in the mix as a backup server for the recoverygroup and if so,
> how do you do that. Right now that server is designated as primary server
> for the recovery group. I would like to have both IO servers in the mix for
> redundancy purposes.
>
> This ESS situation is beyond frustrating and I don't see end in sight.
>
> Any help is appreciated.
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/bded79d3/attachment-0001.htm>

From makaplan at us.ibm.com  Fri Dec  1 04:25:30 2017
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Thu, 30 Nov 2017 23:25:30 -0500
Subject: [gpfsug-discuss] FIle system vs Database
In-Reply-To: <OFDAB5F371.63BBCEDD-ON002581E8.00776AE7@LocalDomain>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com><OF1A4C52AF.364AA2DD-ON002581E8.00592509@LocalDomain><OF0CB3DA65.211AD6C3-ON852581E8.006590C3-852581E8.0065FFAE@notes.na.collabserv.com><OF777157A3.FE35AFDD-ON002581E8.0066BB88@LocalDomain><OFC357735B.03D413FB-ON852581E8.0072148B-852581E8.0072A3C9@notes.na.collabserv.com>
	<OFDAB5F371.63BBCEDD-ON002581E8.00776AE7@LocalDomain>
Message-ID: <OF4ADF66F6.43003461-ON852581E9.0017BDD1-852581E9.00184F30@notes.na.collabserv.com>

Indeed, for a very large directory you might get some speedup using

samples/ilm/mmfind directory -ls -maxdepth 1

There are some caveats, the same as those for the command upon which 
mmfind rests, mmapplypolicy.




From:   Skylar Thompson <skylar2 at u.washington.edu>
To:     gpfsug-discuss at spectrumscale.org
Date:   11/30/2017 04:44 PM
Subject:        Re: [gpfsug-discuss] FIle system vs Database
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Interesting, thanks for the information Marc. Could there be an 
improvement
for something like "ls -l some-dir" using the API, though? Instead of 
getdents +
stat for every file (entering and leaving kernel mode many times), could 
it be done
in one operation with one context switch?




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171130/4e738bf8/attachment-0002.htm>

From chris.schlipalius at pawsey.org.au  Fri Dec  1 05:32:28 2017
From: chris.schlipalius at pawsey.org.au (Chris Schlipalius)
Date: Fri, 01 Dec 2017 13:32:28 +0800
Subject: [gpfsug-discuss] 2018 March 26th Singapore Spectrum Scale User
 Group event announced - Call for user stories
In-Reply-To: <A675FAE6-BAE2-4E17-AC2B-7EF1607CC7C9@pawsey.org.au>
References: <A675FAE6-BAE2-4E17-AC2B-7EF1607CC7C9@pawsey.org.au>
Message-ID: <391567D9-DC51-46B9-8E0E-D8EE13F6B814@pawsey.org.au>

Hello,
This is an announcement for the event held just before SCA18 and  general call out for user use cases or presentations that you would like to present, at the inaugural Spectrum Scale Usergroup Singapore on the Monday 26th March 2018, Sentosa, Singapore.
This is being held in conjunction with SCA18  https://sc-asia.org/
     
All current Singapore Spectrum Scale User Group event details can be found here: http://goo.gl/dXtqvS
Feel free to circulate this event link to all that may need it.
Please reserve your tickets now, and if you would like to speak, please email me the title and duration of the talk and speakers name details so I can add this to the agenda on Eventbrite and promote this in this discussion list and spectrumscale.org website.
     
Accommodation.
Once numbers of attendees are known I can obtain delegate accommodation custom rate for guests staying at the resort for this event. So please reserve a ticket in the Eventbrite link above ASAP.
We are looking forwards to a great new Usergroup next year in a fabulous venue.
Thanks again to NSCC and IBM for helping to arrange the venue and event booking.
     
Regards,
Chris Schlipalius
     
Team Lead, Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO)
12 Burvill Court
Kensington  WA  6151
Australia
     
Tel  +61 8 6436 8815  
Email  chris.schlipalius at pawsey.org.au

Regards,
Chris Schlipalius
 
Team Lead, Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO)
12 Burvill Court
Kensington  WA  6151
Australia
 
Tel  +61 8 6436 8815  
Email  chris.schlipalius at pawsey.org.au
Web  www.pawsey.org.au <http://www.pawsey.org.au/>
 
 
 

 
 
Web  www.pawsey.org.au <http://www.pawsey.org.au/>
     
     
    




From bhartner at us.ibm.com  Fri Dec  1 12:44:38 2017
From: bhartner at us.ibm.com (Bill Hartner)
Date: Fri, 1 Dec 2017 06:44:38 -0600
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
Message-ID: <OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>

> "It has a significant performance penalty for small files in large
> block size filesystems"

Aaron,

Below are mdtest results for a test we ran for CORAL - file size was 32k.

We have not gone back and ran the test on a file system formatted without >
32 subblocks.  We'll do that at some point...

-Bill

-- started at 10/28/2017 17:51:38 --

mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
Command line used: /tmp/mdtest-binary-dir/mdtest
-d /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360 -u
-y
Path: /ibm/fs2-16m-10
FS: 128.1 TiB   Used FS: 0.3%   Inodes: 476.8 Mi   Used Inodes: 0.0%

228 tasks, 67239936 files

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean
Std Dev
   ---------                      ---            ---           ----
-------
   File creation     :      51953.498      50558.517      51423.221
616.643
   File stat         :          0.000          0.000          0.000
0.000
   File read         :          0.000          0.000          0.000
0.000
   File removal      :      96746.376      92149.535      94658.774
1900.187
   Tree creation     :          1.588          0.070          0.599
0.700
   Tree removal      :          0.213          0.034          0.097
0.082

-- finished at 10/28/2017 19:51:54 --

Bill Hartner
IBM Systems
Scalable I/O Development
Austin, Texas
bhartner at us.ibm.com
home office 512-784-0980


gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:

> From: Aaron Knister <aaron.knister at gmail.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 11/29/2017 04:42 PM
> Subject: Re: [gpfsug-discuss] Online data migration tool
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
> Thanks, Nikhil. Most of that was consistent with my understnading,
> however I was under the impression that the >32 subblocks code is
> required to achieve the touted 50k file creates/second that Sven has
> talked about a bunch of times:
>
>
http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf

> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
> _Sven_Oehme_-_News_from_Research.pdf
> http://files.gpfsug.org/presentations/2016/SC16/12_-
> _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf

> from those presentations regarding 32 subblocks:
>
> "It has a significant performance penalty for small files in large
> block size filesystems"

> although I'm not clear on the specific definition of "large". Many
> filesystems I encounter only have a 1M block size so it may not
> matter there, although that same presentation clearly shows the
> benefit of larger block sizes which is yet *another* thing for which
> a migration tool would be helpful.

> -Aaron
>
> On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com>
wrote:
> Hi,
>
> I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
> For all Spectrum Scale clusters that are currently at 4.X.X, it is
> possible to migrate to 5.0.0 with no offline data migration and no
> need to move data. Once these clusters are at 5.0.0, they will
> benefit from the performance improvements, new features (such as
> file audit logging), and various enhancements that are included in 5.0.0.
>
> That being said, there is one enhancement that will not be applied
> to these clusters, and that is the increased number of sub-blocks
> per block for small file allocation. This means that for file
> systems with a large block size and a lot of small files, the
> overall space utilization will be the same it currently is in 4.X.X.
> Since file systems created at 4.X.X and earlier used a block size
> that kept this allocation in mind, there should be very little
> impact on existing file systems.
>
> Outside of that one particular function, the remainder of the
> performance improvements, metadata improvements, updated
> compatibility, new functionality, and all of the other enhancements
> will be immediately available to you once you complete the upgrade
> to 5.0.0 -- with no need to reformat, move data, or take your data
offline.
>
> I hope that clarifies things a little and makes the upgrade path
> more accessible.
>
> Please let me know if there are any other questions or concerns.
>
> Thank you,
> Nikhil Khandelwal
> Spectrum Scale Development
> Client Adoption
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-

>
siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/18d266d7/attachment-0002.htm>

From janfrode at tanso.net  Fri Dec  1 12:53:21 2017
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 01 Dec 2017 12:53:21 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
Message-ID: <CAHwPatj2=BUa1zHO453BBtAWvmwhxXoBqSByoEDYKg_PvZEa-A@mail.gmail.com>

Bill, could you say something about what the metadata-storage here was?
ESS/NL-SAS/3way replication?

I just asked about this in the internal slack channel #scale-help today..



-jf

fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <bhartner at us.ibm.com>:

> > "It has a significant performance penalty for small files in large
> > block size filesystems"
>
> Aaron,
>
> Below are mdtest results for a test we ran for CORAL - file size was 32k.
>
> We have not gone back and ran the test on a file system formatted without
> > 32 subblocks. We'll do that at some point...
>
> -Bill
>
> -- started at 10/28/2017 17:51:38 --
>
> mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> Command line used: /tmp/mdtest-binary-dir/mdtest -d
> /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360 -u -y
> Path: /ibm/fs2-16m-10
> FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>
> 228 tasks, 67239936 files
>
> SUMMARY: (of 3 iterations)
> Operation Max Min Mean Std Dev
> --------- --- --- ---- -------
> File creation : 51953.498 50558.517 51423.221 616.643
> File stat : 0.000 0.000 0.000 0.000
> File read : 0.000 0.000 0.000 0.000
> File removal : 96746.376 92149.535 94658.774 1900.187
> Tree creation : 1.588 0.070 0.599 0.700
> Tree removal : 0.213 0.034 0.097 0.082
>
> -- finished at 10/28/2017 19:51:54 --
>
> Bill Hartner
> IBM Systems
> Scalable I/O Development
> Austin, Texas
> bhartner at us.ibm.com
> home office 512-784-0980
>
>
> gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:
>
> > From: Aaron Knister <aaron.knister at gmail.com>
>
>
> > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>
> > Date: 11/29/2017 04:42 PM
>
>
> > Subject: Re: [gpfsug-discuss] Online data migration tool
> > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
> >
>
> > Thanks, Nikhil. Most of that was consistent with my understnading,
> > however I was under the impression that the >32 subblocks code is
> > required to achieve the touted 50k file creates/second that Sven has
> > talked about a bunch of times:
> >
> >
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf
> > http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
> > _Sven_Oehme_-_News_from_Research.pdf
> > http://files.gpfsug.org/presentations/2016/SC16/12_-
> > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>
>
> > from those presentations regarding 32 subblocks:
> >
> > "It has a significant performance penalty for small files in large
> > block size filesystems"
>
> > although I'm not clear on the specific definition of "large". Many
> > filesystems I encounter only have a 1M block size so it may not
> > matter there, although that same presentation clearly shows the
> > benefit of larger block sizes which is yet *another* thing for which
> > a migration tool would be helpful.
>
> > -Aaron
> >
> > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com>
> wrote:
>
> > Hi,
> >
> > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
> > For all Spectrum Scale clusters that are currently at 4.X.X, it is
> > possible to migrate to 5.0.0 with no offline data migration and no
> > need to move data. Once these clusters are at 5.0.0, they will
> > benefit from the performance improvements, new features (such as
> > file audit logging), and various enhancements that are included in 5.0.0.
> >
> > That being said, there is one enhancement that will not be applied
> > to these clusters, and that is the increased number of sub-blocks
> > per block for small file allocation. This means that for file
> > systems with a large block size and a lot of small files, the
> > overall space utilization will be the same it currently is in 4.X.X.
> > Since file systems created at 4.X.X and earlier used a block size
> > that kept this allocation in mind, there should be very little
> > impact on existing file systems.
> >
> > Outside of that one particular function, the remainder of the
> > performance improvements, metadata improvements, updated
> > compatibility, new functionality, and all of the other enhancements
> > will be immediately available to you once you complete the upgrade
> > to 5.0.0 -- with no need to reformat, move data, or take your data
> offline.
> >
> > I hope that clarifies things a little and makes the upgrade path
> > more accessible.
> >
> > Please let me know if there are any other questions or concerns.
> >
> > Thank you,
> > Nikhil Khandelwal
> > Spectrum Scale Development
> > Client Adoption
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
>
> > https://urldefense.proofpoint.com/v2/url?
> >
> u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
> >
> siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/a6a697b3/attachment-0002.htm>

From luis.bolinches at fi.ibm.com  Fri Dec  1 14:00:19 2017
From: luis.bolinches at fi.ibm.com (Luis Bolinches)
Date: Fri, 1 Dec 2017 14:00:19 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF9BFF17D3.E70240F6-ON002581E9.0046DD04@LocalDomain>
Message-ID: <OFECC4DC82.CE3786C2-ON002581E9.004CEEB3-1512136819057@notes.na.collabserv.com>

For Hana 2.0 only SP1 and 2 are supported

--
Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations
Luis Bolinches
Consultant IT Specialist
Mobile Phone: +358503112585
https://www.youracclaim.com/user/luis-bolinches

"If you always give you will always have" --  Anonymous

> On 1 Dec 2017, at 14.53, Jan-Frode Myklebust <janfrode at tanso.net> wrote:
> 
> Bill, could you say something about what the metadata-storage here was? ESS/NL-SAS/3way replication?
> 
> I just asked about this in the internal slack channel #scale-help today..
> 
> 
> 
> -jf
> 
>> fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <bhartner at us.ibm.com>:
>> > "It has a significant performance penalty for small files in large 
>> > block size filesystems"
>> 
>> 
>> Aaron,
>> 
>> Below are mdtest results for a test we ran for CORAL - file size was 32k.
>> 
>> We have not gone back and ran the test on a file system formatted without > 32 subblocks. We'll do that at some point...
>> 
>> -Bill
>> 
>> -- started at 10/28/2017 17:51:38 --
>> 
>> mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>> Command line used: /tmp/mdtest-binary-dir/mdtest -d /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360 -u -y
>> Path: /ibm/fs2-16m-10
>> FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>> 
>> 228 tasks, 67239936 files
>> 
>> SUMMARY: (of 3 iterations)
>> Operation Max Min Mean Std Dev
>> --------- --- --- ---- -------
>> File creation : 51953.498 50558.517 51423.221 616.643
>> File stat : 0.000 0.000 0.000 0.000
>> File read : 0.000 0.000 0.000 0.000
>> File removal : 96746.376 92149.535 94658.774 1900.187
>> Tree creation : 1.588 0.070 0.599 0.700
>> Tree removal : 0.213 0.034 0.097 0.082
>> 
>> -- finished at 10/28/2017 19:51:54 --
>> 
>> Bill Hartner
>> IBM Systems
>> Scalable I/O Development
>> Austin, Texas
>> bhartner at us.ibm.com
>> home office 512-784-0980
>> 
>> 
>> gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:
>> 
>> > From: Aaron Knister <aaron.knister at gmail.com>
>> 
>> 
>> > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> 
>> > Date: 11/29/2017 04:42 PM
>> 
>> 
>> > Subject: Re: [gpfsug-discuss] Online data migration tool
>> > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> 
>> > 
>> 
>> > Thanks, Nikhil. Most of that was consistent with my understnading, 
>> > however I was under the impression that the >32 subblocks code is 
>> > required to achieve the touted 50k file creates/second that Sven has
>> > talked about a bunch of times:
>> > 
>> > http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf
>> > http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
>> > _Sven_Oehme_-_News_from_Research.pdf
>> > http://files.gpfsug.org/presentations/2016/SC16/12_-
>> > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>> 
>> 
>> > from those presentations regarding 32 subblocks:
>> > 
>> > "It has a significant performance penalty for small files in large 
>> > block size filesystems"
>> 
>> > although I'm not clear on the specific definition of "large". Many 
>> > filesystems I encounter only have a 1M block size so it may not 
>> > matter there, although that same presentation clearly shows the 
>> > benefit of larger block sizes which is yet *another* thing for which
>> > a migration tool would be helpful.
>> 
>> > -Aaron
>> > 
>> > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com> wrote:
>> 
>> > Hi,
>> > 
>> > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
>> > For all Spectrum Scale clusters that are currently at 4.X.X, it is 
>> > possible to migrate to 5.0.0 with no offline data migration and no 
>> > need to move data. Once these clusters are at 5.0.0, they will 
>> > benefit from the performance improvements, new features (such as 
>> > file audit logging), and various enhancements that are included in 5.0.0.
>> > 
>> > That being said, there is one enhancement that will not be applied 
>> > to these clusters, and that is the increased number of sub-blocks 
>> > per block for small file allocation. This means that for file 
>> > systems with a large block size and a lot of small files, the 
>> > overall space utilization will be the same it currently is in 4.X.X.
>> > Since file systems created at 4.X.X and earlier used a block size 
>> > that kept this allocation in mind, there should be very little 
>> > impact on existing file systems.
>> > 
>> > Outside of that one particular function, the remainder of the 
>> > performance improvements, metadata improvements, updated 
>> > compatibility, new functionality, and all of the other enhancements 
>> > will be immediately available to you once you complete the upgrade 
>> > to 5.0.0 -- with no need to reformat, move data, or take your data offline.
>> > 
>> > I hope that clarifies things a little and makes the upgrade path 
>> > more accessible.
>> > 
>> > Please let me know if there are any other questions or concerns.
>> > 
>> > Thank you,
>> > Nikhil Khandelwal
>> > Spectrum Scale Development
>> > Client Adoption
>> > 
>> > _______________________________________________
>> > gpfsug-discuss mailing list
>> > gpfsug-discuss at spectrumscale.org
>> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>> 
>> 
>> > _______________________________________________
>> > gpfsug-discuss mailing list
>> > gpfsug-discuss at spectrumscale.org
>> 
>> > https://urldefense.proofpoint.com/v2/url?
>> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>> > siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Ellei edell? ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3 
Registered in Finland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/e09df482/attachment-0002.htm>

From bhartner at us.ibm.com  Fri Dec  1 16:44:23 2017
From: bhartner at us.ibm.com (Bill Hartner)
Date: Fri, 1 Dec 2017 10:44:23 -0600
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com><OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain><OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
Message-ID: <OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>


ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the results @
user group meeting.

-Bill

Bill Hartner
IBM Systems
Scalable I/O Development
Austin, Texas
bhartner at us.ibm.com
home office 512-784-0980




From:	Jan-Frode Myklebust <janfrode at tanso.net>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	12/01/2017 06:53 AM
Subject:	Re: [gpfsug-discuss] Online data migration tool
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Bill, could you say something about what the metadata-storage here was?
ESS/NL-SAS/3way replication?

I just asked about this in the internal slack channel #scale-help today..



-jf

fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <bhartner at us.ibm.com>:
  > "It has a significant performance penalty for small files in large
  > block size filesystems"



  Aaron,

  Below are mdtest results for a test we ran for CORAL - file size was 32k.

  We have not gone back and ran the test on a file system formatted without
  > 32 subblocks. We'll do that at some point...

  -Bill

  -- started at 10/28/2017 17:51:38 --

  mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
  Command line used: /tmp/mdtest-binary-dir/mdtest
  -d /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360
  -u -y
  Path: /ibm/fs2-16m-10
  FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%

  228 tasks, 67239936 files

  SUMMARY: (of 3 iterations)
  Operation Max Min Mean Std Dev
  --------- --- --- ---- -------
  File creation : 51953.498 50558.517 51423.221 616.643
  File stat : 0.000 0.000 0.000 0.000
  File read : 0.000 0.000 0.000 0.000
  File removal : 96746.376 92149.535 94658.774 1900.187
  Tree creation : 1.588 0.070 0.599 0.700
  Tree removal : 0.213 0.034 0.097 0.082

  -- finished at 10/28/2017 19:51:54 --

  Bill Hartner
  IBM Systems
  Scalable I/O Development
  Austin, Texas
  bhartner at us.ibm.com
  home office 512-784-0980


  gpfsug-discuss-bounces at spectrumscale.org wrote on 11/29/2017 04:41:48 PM:

  > From: Aaron Knister <aaron.knister at gmail.com>



  > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>


  > Date: 11/29/2017 04:42 PM



  > Subject: Re: [gpfsug-discuss] Online data migration tool
  > Sent by: gpfsug-discuss-bounces at spectrumscale.org


  >


  > Thanks, Nikhil. Most of that was consistent with my understnading,
  > however I was under the impression that the >32 subblocks code is
  > required to achieve the touted 50k file creates/second that Sven has
  > talked about a bunch of times:
  >
  >
  http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf

  > http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-
  > _Sven_Oehme_-_News_from_Research.pdf
  > http://files.gpfsug.org/presentations/2016/SC16/12_-
  > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf



  > from those presentations regarding 32 subblocks:
  >
  > "It has a significant performance penalty for small files in large
  > block size filesystems"

  > although I'm not clear on the specific definition of "large". Many
  > filesystems I encounter only have a 1M block size so it may not
  > matter there, although that same presentation clearly shows the
  > benefit of larger block sizes which is yet *another* thing for which
  > a migration tool would be helpful.

  > -Aaron
  >
  > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal <nikhilk at us.ibm.com>
  wrote:


  > Hi,
  >
  > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
  > For all Spectrum Scale clusters that are currently at 4.X.X, it is
  > possible to migrate to 5.0.0 with no offline data migration and no
  > need to move data. Once these clusters are at 5.0.0, they will
  > benefit from the performance improvements, new features (such as
  > file audit logging), and various enhancements that are included in
  5.0.0.
  >
  > That being said, there is one enhancement that will not be applied
  > to these clusters, and that is the increased number of sub-blocks
  > per block for small file allocation. This means that for file
  > systems with a large block size and a lot of small files, the
  > overall space utilization will be the same it currently is in 4.X.X.
  > Since file systems created at 4.X.X and earlier used a block size
  > that kept this allocation in mind, there should be very little
  > impact on existing file systems.
  >
  > Outside of that one particular function, the remainder of the
  > performance improvements, metadata improvements, updated
  > compatibility, new functionality, and all of the other enhancements
  > will be immediately available to you once you complete the upgrade
  > to 5.0.0 -- with no need to reformat, move data, or take your data
  offline.
  >
  > I hope that clarifies things a little and makes the upgrade path
  > more accessible.
  >
  > Please let me know if there are any other questions or concerns.
  >
  > Thank you,
  > Nikhil Khandelwal
  > Spectrum Scale Development
  > Client Adoption
  >
  > _______________________________________________
  > gpfsug-discuss mailing list
  > gpfsug-discuss at spectrumscale.org
  > http://gpfsug.org/mailman/listinfo/gpfsug-discuss



  > _______________________________________________
  > gpfsug-discuss mailing list
  > gpfsug-discuss at spectrumscale.org


  > https://urldefense.proofpoint.com/v2/url?
  >
  u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-

  >
  siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=


  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at spectrumscale.org
  http://gpfsug.org/mailman/listinfo/gpfsug-discuss
  _______________________________________________
  gpfsug-discuss mailing list
  gpfsug-discuss at spectrumscale.org
  https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/60b7fce0/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/60b7fce0/attachment-0002.gif>

From r.sobey at imperial.ac.uk  Fri Dec  1 16:57:46 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Fri, 1 Dec 2017 16:57:46 +0000
Subject: [gpfsug-discuss] Policy rule to list files based on access time
Message-ID: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Hi all,

I'm getting some strange results using a policy to list all the files (well, I'm only interested in the total size that GPFS chooses) that have not been accessed for more than 1/2/3 etc years.

The policy rule I'm using is:

RULE 'list_old' LIST 'atime_gt_730'
WHERE DAYS(CURRENT_TIMESTAMP) -DAYS(ACCESS_TIME) > 730

..for anything over 2 years. My filesystem is 3.5 years old and we first started migrating to it about three years ago, yet although GPFS is choosing ~400000 files that match, the total size is just north of 11000KB:

[I] Summary of Rule Applicability and File Choices:
Rule#      Hit_Cnt          KB_Hit          Chosen       KB_Chosen          KB_Ill     Rule
     0       453765           11776          453765           11776               0     RULE 'list_old' LIST 'atime_gt_730' WHERE(.)

I'm having a hard time believing half a million files total 11MB especially given the very random spread of usage the system sees.

The mmapplypolicy command I'm running is:

/usr/lpp/mmfs/bin/mmapplypolicy /gpfs/path -f /gpfs/policies/ -I defer --scope filesystem -P ./list_old_allfs.txt

The directory 'path' above is the root of the main CES share which encompasses all my filesets.

This is my first real foray into using the policy engine so I might be doing something silly, or misinterpreting the results, or there is simply a far easier way to achieve my goal.

Can anyone help?

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/69ada640/attachment-0002.htm>

From alex at calicolabs.com  Fri Dec  1 17:18:57 2017
From: alex at calicolabs.com (Alex Chekholko)
Date: Fri, 1 Dec 2017 09:18:57 -0800
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <CANcy_PbUS93Y7TMxL2f+LcH7_4fJk0SkpJED54n3h=2Ju5Lzfg@mail.gmail.com>

Hi Richard,

Either add something like "-L 2" to your mmapplypolicy command or add an
'EXEC ...' to your policy file to have it print out the actual files that
matched. There is an example exec script somewhere in the samples folder

I have found that in the past when I doubted the policy results, there was
something I misunderstood. So just go ahead and examine the metadata on the
actual files that match.

Perhaps at some point in the past, you had some kind of application scan
through the filesystem and thus all the access times got updated to newer
than you expect.

For the file count/size numbers, I don't find that unusual; there are all
kinds of user applications that have pathological patterns like 450k files
totaling 11MB.  But worth double-checking.  You can run some example
policies like the one you ran, but list only >10TB files or list only files
of 0 size; you never know what you might find.

Regards,
Alex


On Fri, Dec 1, 2017 at 8:57 AM, Sobey, Richard A <r.sobey at imperial.ac.uk>
wrote:

> Hi all,
>
>
>
> I?m getting some strange results using a policy to list all the files
> (well, I?m only interested in the total size that GPFS chooses) that have
> not been accessed for more than 1/2/3 etc years.
>
>
>
> The policy rule I?m using is:
>
>
>
> RULE 'list_old' LIST 'atime_gt_730'
>
> WHERE DAYS(CURRENT_TIMESTAMP) -DAYS(ACCESS_TIME) > 730
>
>
>
> ..for anything over 2 years. My filesystem is 3.5 years old and we first
> started migrating to it about three years ago, yet although GPFS is
> choosing ~400000 files that match, the total size is just north of 11000KB:
>
>
>
> [I] Summary of Rule Applicability and File Choices:
>
> Rule#      Hit_Cnt          KB_Hit          Chosen
> KB_Chosen          KB_Ill     Rule
>
>      0       453765           11776          453765
> 11776               0     RULE 'list_old' LIST 'atime_gt_730' WHERE(.)
>
>
>
> I?m having a hard time believing half a million files total 11MB
> especially given the very random spread of usage the system sees.
>
>
>
> The mmapplypolicy command I?m running is:
>
>
>
> /usr/lpp/mmfs/bin/mmapplypolicy /gpfs/path -f /gpfs/policies/ -I defer
> --scope filesystem -P ./list_old_allfs.txt
>
>
>
> The directory ?path? above is the root of the main CES share which
> encompasses all my filesets.
>
>
>
> This is my first real foray into using the policy engine so I might be
> doing something silly, or misinterpreting the results, or there is simply a
> far easier way to achieve my goal.
>
>
>
> Can anyone help?
>
>
>
> Thanks
>
> Richard
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171201/1395f933/attachment-0002.htm>

From ewahl at osc.edu  Fri Dec  1 17:59:21 2017
From: ewahl at osc.edu (Edward Wahl)
Date: Fri, 1 Dec 2017 12:59:21 -0500
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <20171201125921.507043a3@osc.edu>


You don't specify a choice algorithm?  Might want to change your script with a
quick thing like this to view all the candidates.  I use a similar 120day and
I've looked at atime/ctimes of them before and was happy with the output.
You can then quickly size the output with a quick sed/awk/whatever.

SHOW (varchar(file_size) || '  ' || varchar(USER_ID))
WHERE DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME) >= 180 

and add "--choice-algorithm exact" to the command line to see what files match. 

I can't recall what the default is. Might already be exact?  Oops, man pages
says yes.  Might then run all 3 and see if the output is much different. 

Ed

On Fri, 1 Dec 2017 16:57:46 +0000
"Sobey, Richard A" <r.sobey at imperial.ac.uk> wrote:

> Hi all,
> 
> I'm getting some strange results using a policy to list all the files (well,
> I'm only interested in the total size that GPFS chooses) that have not been
> accessed for more than 1/2/3 etc years.
> 
> The policy rule I'm using is:
> 
> RULE 'list_old' LIST 'atime_gt_730'
> WHERE DAYS(CURRENT_TIMESTAMP) -DAYS(ACCESS_TIME) > 730
> 
> ..for anything over 2 years. My filesystem is 3.5 years old and we first
> started migrating to it about three years ago, yet although GPFS is choosing
> ~400000 files that match, the total size is just north of 11000KB:
> 
> [I] Summary of Rule Applicability and File Choices:
> Rule#      Hit_Cnt          KB_Hit          Chosen       KB_Chosen
> KB_Ill     Rule 0       453765           11776          453765
> 11776               0     RULE 'list_old' LIST 'atime_gt_730' WHERE(.)
> 
> I'm having a hard time believing half a million files total 11MB especially
> given the very random spread of usage the system sees.
> 
> The mmapplypolicy command I'm running is:
> 
> /usr/lpp/mmfs/bin/mmapplypolicy /gpfs/path -f /gpfs/policies/ -I defer
> --scope filesystem -P ./list_old_allfs.txt
> 
> The directory 'path' above is the root of the main CES share which
> encompasses all my filesets.
> 
> This is my first real foray into using the policy engine so I might be doing
> something silly, or misinterpreting the results, or there is simply a far
> easier way to achieve my goal.
> 
> Can anyone help?
> 
> Thanks
> Richard



-- 

Ed Wahl
Ohio Supercomputer Center
614-292-9302


From makaplan at us.ibm.com  Sat Dec  2 18:03:25 2017
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Sat, 2 Dec 2017 13:03:25 -0500
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>
Message-ID: <OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>

You might want to look at FILE_SIZE.  KB_ALLOCATED will be 0 if the file 
data fits into the inode.

You might also want to use SIZE(FILE_SIZE) in the policy LIST rule, this 
will cause the KB_HIT and KB_CHOSEN numbers to be the sum of FILE_SIZEs 
instead of the default SIZE(KB_ALLOCATED).

--marc of GPFS

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171202/1703340b/attachment-0002.htm>

From r.sobey at imperial.ac.uk  Sat Dec  2 18:53:14 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Sat, 2 Dec 2017 18:53:14 +0000
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>,
	<OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>
Message-ID: <DB6PR0601MB210125AFF8783FFFAB2BF3E4DF3E0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Thank you all for your replies. I will take a look at them on Monday. And reply individually where I need to..

Richard

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Marc A Kaplan <makaplan at us.ibm.com>
Sent: Saturday, December 2, 2017 6:03:25 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Policy rule to list files based on access time

You might want to look at FILE_SIZE.  KB_ALLOCATED will be 0 if the file data fits into the inode.

You might also want to use SIZE(FILE_SIZE) in the policy LIST rule, this will cause the KB_HIT and KB_CHOSEN numbers to be the sum of FILE_SIZEs instead of the default SIZE(KB_ALLOCATED).

--marc of GPFS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171202/815f8412/attachment-0002.htm>

From bipcuds at gmail.com  Mon Dec  4 02:49:06 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Sun, 3 Dec 2017 21:49:06 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file quota/limit
	possible to set?
Message-ID: <CAAxuGpEFH7mi80Aa2u_LbaHP+3JU1y4qKQPFOB+HohCD76XZ2Q@mail.gmail.com>

From: Keith Ball <kball at redlineperf.com>
To: gpfsug-discuss at spectrumscale.org
Cc:
Bcc:
Date: Sun, 3 Dec 2017 16:13:13 -0500
Subject: Smallest block quota/limit and file quota/limit possible to set?
HI All,

We have a system where all users have their own private group as well.
However, for a given fileset (we are using --perfileset-quota), we would
like to ONLY allow users who also belong to just a few central groups to be
able to write to the fileset.

That is, user "userA" has its own "groupA", but we only want the user to be
able to write to the fileset if:
 - userA belongs to one of the groups (e.g. group1, group2, group3) that
have explicitly set quotas
 - The group(s) in question are within quota/limits.

In general, we do not want any users that do NOT belong to one of the three
groups with enabled quotas to be able to write anything at all to the
fileset.

Is there a way to set a ZERO quota for block/file in GPFS, that means what
it actually should mean? i.e. "Your limit is 0 file = you cannot create
files in this fileset". Creating some kind of "supergroup" owner of the
fileset (with entitled users as members of the group) could work, but that
will only work for *one* group.

If we cannot set the block and file limits to zero, what *are* the smallest
block and fie limits? In GPFS 3.5, they seem to be 1760MB for block. Is
there a smallest quota for files? (blocksize is 16MB, which will be reduced
to 4MB probably, in a subsequent cluster).

Many Thanks,
  Keith



-- 
Keith D. Ball, PhD
RedLine Performance Solutions, LLC
web:  http://www.redlineperf.com/
email: kball at redlineperf.com <aqualkenbush at redlineperf.com>
cell: 540-557-7851 <(540)%20557-7851>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171203/8c867510/attachment-0002.htm>

From scale at us.ibm.com  Mon Dec  4 03:49:38 2017
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Mon, 4 Dec 2017 09:19:38 +0530
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit	possible to set?
In-Reply-To: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
Message-ID: <OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>

Hi Keith,

You can use ACLs for fine grained permissions. A quota limit of 0 in GPFS 
implies no limits.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   Keith Ball <bipcuds at gmail.com>
To:     gpfsug-discuss at spectrumscale.org
Date:   12/04/2017 08:19 AM
Subject:        [gpfsug-discuss] Smallest block quota/limit and file 
quota/limit     possible to set?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



From: Keith Ball <kball at redlineperf.com>
To: gpfsug-discuss at spectrumscale.org
Cc: 
Bcc: 
Date: Sun, 3 Dec 2017 16:13:13 -0500
Subject: Smallest block quota/limit and file quota/limit possible to set?
HI All,

We have a system where all users have their own private group as well. 
However, for a given fileset (we are using --perfileset-quota), we would 
like to ONLY allow users who also belong to just a few central groups to 
be able to write to the fileset. 

That is, user "userA" has its own "groupA", but we only want the user to 
be able to write to the fileset if:
 - userA belongs to one of the groups (e.g. group1, group2, group3) that 
have explicitly set quotas
 - The group(s) in question are within quota/limits.

In general, we do not want any users that do NOT belong to one of the 
three groups with enabled quotas to be able to write anything at all to 
the fileset.

Is there a way to set a ZERO quota for block/file in GPFS, that means what 
it actually should mean? i.e. "Your limit is 0 file = you cannot create 
files in this fileset". Creating some kind of "supergroup" owner of the 
fileset (with entitled users as members of the group) could work, but that 
will only work for *one* group. 

If we cannot set the block and file limits to zero, what *are* the 
smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB for 
block. Is there a smallest quota for files? (blocksize is 16MB, which will 
be reduced to 4MB probably, in a subsequent cluster).

Many Thanks,
  Keith 



-- 
Keith D. Ball, PhD
RedLine Performance Solutions, LLC
web:  http://www.redlineperf.com/
email: kball at redlineperf.com
cell: 540-557-7851_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=N0b8UJpG1zFuuQjvCE1SaaIG2EUDv4v783wDc3TbnyM&s=vuNQN4VJonV9nmcTEOKKr-hL2LAWSOUV7HKAQBKWcK4&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/0c8ffd87/attachment-0002.htm>

From z.han at imperial.ac.uk  Mon Dec  4 09:52:31 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Mon, 4 Dec 2017 09:52:31 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or above?
Message-ID: <alpine.LFD.2.20.1712040941230.2395@laptop>

Hi All,

Any one is using a Linux kernel 3.12.x or above
to run gpfs 4.2.3-4.2? I mean you've compiled
your own kernel without paying for a professional
service.

We're stuck by CentOS/RHEL's distributed kernel
as the PCI passthrough is required for VMs. Your
comments or suggestions are much appreciated.

Kind regards,

Zong-Pei



From S.J.Thompson at bham.ac.uk  Mon Dec  4 10:29:27 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Mon, 4 Dec 2017 10:29:27 +0000
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712040941230.2395@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
Message-ID: <B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>

The FAQ at:

https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux

Lists support with (e.g. Ubutu 16.04.2) with kernel 4.4.0-62, so likely it would work with a build your own kernel, but that doesn?t mean it is **supported**

Simon

On 04/12/2017, 09:52, "gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk" <gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk> wrote:

    Hi All,
    
    Any one is using a Linux kernel 3.12.x or above
    to run gpfs 4.2.3-4.2? I mean you've compiled
    your own kernel without paying for a professional
    service.
    
    We're stuck by CentOS/RHEL's distributed kernel
    as the PCI passthrough is required for VMs. Your
    comments or suggestions are much appreciated.
    
    Kind regards,
    
    Zong-Pei
    
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
    


From z.han at imperial.ac.uk  Mon Dec  4 10:37:39 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Mon, 4 Dec 2017 10:37:39 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
	<B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>
Message-ID: <alpine.LFD.2.20.1712041036340.2395@laptop>

Thanks, Simon.

Only on clients (oh so-called clients!) ;)


On Mon, 4 Dec 2017, Simon Thompson (IT Research Support) wrote:
> Date: Mon, 4 Dec 2017 10:29:27 +0000
> From: "Simon Thompson (IT Research Support)" <S.J.Thompson at bham.ac.uk>
> Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
>     above?
> 
> The FAQ at:
>
> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux
>
> Lists support with (e.g. Ubutu 16.04.2) with kernel 4.4.0-62, so likely it would work with a build your own kernel, but that doesn?t mean it is **supported**
>
> Simon
>
> On 04/12/2017, 09:52, "gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk" <gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk> wrote:
>
>    Hi All,
>
>    Any one is using a Linux kernel 3.12.x or above
>    to run gpfs 4.2.3-4.2? I mean you've compiled
>    your own kernel without paying for a professional
>    service.
>
>    We're stuck by CentOS/RHEL's distributed kernel
>    as the PCI passthrough is required for VMs. Your
>    comments or suggestions are much appreciated.
>
>    Kind regards,
>
>    Zong-Pei
>
>    _______________________________________________
>    gpfsug-discuss mailing list
>    gpfsug-discuss at spectrumscale.org
>    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From ulmer at ulmer.org  Mon Dec  4 13:46:31 2017
From: ulmer at ulmer.org (Stephen Ulmer)
Date: Mon, 4 Dec 2017 08:46:31 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit	possible to set?
In-Reply-To: <OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
Message-ID: <8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>

I don?t understand why not having permission(s) doesn?t prevent the user from writing into the fileset...

As described, your case is about not wanting userA to be able to write to a fileset if userA isn?t in some groups. Don?t put them in those groups. That?s not even Spectrum Scale specific, it?s about generic *nix permissions.

What am I missing? I don?t understand why you would want to use quota to enforce permissions. (There could be a legitimate reason here, but I don?t understand it.)

Liberty,

-- 
Stephen Ulmer

Sent from a mobile device; please excuse autocorrect silliness.

> On Dec 3, 2017, at 10:49 PM, IBM Spectrum Scale <scale at us.ibm.com> wrote:
> 
> Hi Keith,
> 
> You can use ACLs for fine grained permissions. A quota limit of 0 in GPFS implies no limits.
> 
> Regards, The Spectrum Scale (GPFS) team
> 
> ------------------------------------------------------------------------------------------------------------------
> If you feel that your question can benefit other users of  Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. 
> 
> If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact  1-800-237-5511 in the United States or your local IBM Service Center in other countries. 
> 
> The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.
> 
> 
> 
> From:        Keith Ball <bipcuds at gmail.com>
> To:        gpfsug-discuss at spectrumscale.org
> Date:        12/04/2017 08:19 AM
> Subject:        [gpfsug-discuss] Smallest block quota/limit and file quota/limit        possible to set?
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> 
> 
> 
> From: Keith Ball <kball at redlineperf.com>
> To: gpfsug-discuss at spectrumscale.org
> Cc: 
> Bcc: 
> Date: Sun, 3 Dec 2017 16:13:13 -0500
> Subject: Smallest block quota/limit and file quota/limit possible to set?
> HI All,
> 
> We have a system where all users have their own private group as well. However, for a given fileset (we are using --perfileset-quota), we would like to ONLY allow users who also belong to just a few central groups to be able to write to the fileset. 
> 
> That is, user "userA" has its own "groupA", but we only want the user to be able to write to the fileset if:
>  - userA belongs to one of the groups (e.g. group1, group2, group3) that have explicitly set quotas
>  - The group(s) in question are within quota/limits.
> 
> In general, we do not want any users that do NOT belong to one of the three groups with enabled quotas to be able to write anything at all to the fileset.
> 
> Is there a way to set a ZERO quota for block/file in GPFS, that means what it actually should mean? i.e. "Your limit is 0 file = you cannot create files in this fileset". Creating some kind of "supergroup" owner of the fileset (with entitled users as members of the group) could work, but that will only work for *one* group. 
> 
> If we cannot set the block and file limits to zero, what *are* the smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB for block. Is there a smallest quota for files? (blocksize is 16MB, which will be reduced to 4MB probably, in a subsequent cluster).
> 
> Many Thanks,
>   Keith 
> 
> 
> 
> -- 
> Keith D. Ball, PhD
> RedLine Performance Solutions, LLC
> web:  http://www.redlineperf.com/
> email: kball at redlineperf.com
> cell: 540-557-7851_______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=N0b8UJpG1zFuuQjvCE1SaaIG2EUDv4v783wDc3TbnyM&s=vuNQN4VJonV9nmcTEOKKr-hL2LAWSOUV7HKAQBKWcK4&e=
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/9b42e1fc/attachment-0002.htm>

From valdis.kletnieks at vt.edu  Mon Dec  4 16:27:38 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Mon, 04 Dec 2017 11:27:38 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit possible to set?
In-Reply-To: <8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
Message-ID: <19755.1512404858@turing-police.cc.vt.edu>

On Mon, 04 Dec 2017 08:46:31 -0500, Stephen Ulmer said:

> As described, your case is about not wanting userA to be able to write to a
> fileset if userA isn???t in some groups. Don???t put them in those groups. That???s
> not even Spectrum Scale specific, it???s about generic *nix permissions.

The problem is that doesn't stop userA from scribbling into a world-writable directory,
and they may have some of those because more than one group needs to write in
a specific directory and ACLs are problematic for other reasons....
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/18b3756a/attachment-0002.sig>

From stuartb at 4gh.net  Mon Dec  4 16:33:17 2017
From: stuartb at 4gh.net (Stuart Barkley)
Date: Mon, 4 Dec 2017 11:33:17 -0500 (EST)
Subject: [gpfsug-discuss] Smallest block quota/limit and file
 quota/limit possible to set?
In-Reply-To: <8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>
Message-ID: <alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>

We have a /projects filesystem where individual projects can "buy" a
specific amount of disk space.  We enforce this purchase limit by
creating a specific group for the allocation, adding designated users
to the group and setting a group quota.

This works fine as long as the users properly use setgid directories
and keep proper group ownership of the files and directories.

However, for various reasons our users keep creating files and
directories with incorrect group ownership.  In most cases this is
accidental and eventually causes problems when other group members
need to access the files.  In abusive cases (not yet seen) people
could use this to exceed project disk space allocations.

To address this problem we have default quotas set to about 2GB (the
smallest we seem to be able to set).  This prevents users for
consuming too much unpurchased disk space.  However, this continues to
allow users to create files and directories with incorrect group
ownership and it takes users a while to discover their error.  User
education and cleanup becomes a problem long after the user thinks
things are working.

We would like to have groups without quota definitions to not be able
to create any files.  This would prevent accidental file creation at
the first attempt.

Stuart Barkley

On Mon, 4 Dec 2017 at 08:46 -0000, Stephen Ulmer wrote:

> I don?t understand why not having permission(s) doesn?t prevent the
> user from writing into the fileset...
>
> As described, your case is about not wanting userA to be able to
> write to a fileset if userA isn?t in some groups. Don?t put them in
> those groups. That?s not even Spectrum Scale specific, it?s about
> generic *nix permissions.
>
> What am I missing? I don?t understand why you would want to use
> quota to enforce permissions. (There could be a legitimate reason
> here, but I don?t understand it.)
>
> Liberty,
>
> --
> Stephen Ulmer
>
> Sent from a mobile device; please excuse autocorrect silliness.
>
> > On Dec 3, 2017, at 10:49 PM, IBM Spectrum Scale <scale at us.ibm.com> wrote:
> >
> > Hi Keith,
> >
> > You can use ACLs for fine grained permissions. A quota limit of 0
> > in GPFS implies no limits.
> >
> > Regards, The Spectrum Scale (GPFS) team
> >
> > From:        Keith Ball <bipcuds at gmail.com>
> > To:        gpfsug-discuss at spectrumscale.org
> > Date:        12/04/2017 08:19 AM
> > Subject:        [gpfsug-discuss] Smallest block quota/limit and file quota/limit        possible to set?
> > Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> >
> > HI All,
> >
> > We have a system where all users have their own private group as
> > well. However, for a given fileset (we are using
> > --perfileset-quota), we would like to ONLY allow users who also
> > belong to just a few central groups to be able to write to the
> > fileset.
> >
> > That is, user "userA" has its own "groupA", but we only want the
> > user to be able to write to the fileset if:
> >  - userA belongs to one of the groups (e.g. group1, group2,
> >  group3) that have explicitly set quotas
> >  - The group(s) in question are within quota/limits.
> >
> > In general, we do not want any users that do NOT belong to one of
> > the three groups with enabled quotas to be able to write anything
> > at all to the fileset.
> >
> > Is there a way to set a ZERO quota for block/file in GPFS, that
> > means what it actually should mean? i.e. "Your limit is 0 file =
> > you cannot create files in this fileset". Creating some kind of
> > "supergroup" owner of the fileset (with entitled users as members
> > of the group) could work, but that will only work for *one* group.
> >
> > If we cannot set the block and file limits to zero, what *are* the
> > smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB
> > for block. Is there a smallest quota for files? (blocksize is
> > 16MB, which will be reduced to 4MB probably, in a subsequent
> > cluster).
> >
> > Many Thanks,
> >   Keith


From S.J.Thompson at bham.ac.uk  Mon Dec  4 17:08:19 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Mon, 4 Dec 2017 17:08:19 +0000
Subject: [gpfsug-discuss] Smallest block quota/limit and file
 quota/limit possible to set?
In-Reply-To: <alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>,
	<alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>

Stuart,

Have you looked at using filesets instead an using fileset quotas to achieve this?

This is what we do and the max number of filesets (currently) isn't an issue for us.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of stuartb at 4gh.net [stuartb at 4gh.net]
Sent: 04 December 2017 16:33
To: gpfsug main discussion list
Cc: gpfsug-discuss-bounces at spectrumscale.org
Subject: Re: [gpfsug-discuss] Smallest block quota/limit and file quota/limit possible to set?

We have a /projects filesystem where individual projects can "buy" a
specific amount of disk space.  We enforce this purchase limit by
creating a specific group for the allocation, adding designated users
to the group and setting a group quota.

This works fine as long as the users properly use setgid directories
and keep proper group ownership of the files and directories.

However, for various reasons our users keep creating files and
directories with incorrect group ownership.  In most cases this is
accidental and eventually causes problems when other group members
need to access the files.  In abusive cases (not yet seen) people
could use this to exceed project disk space allocations.

To address this problem we have default quotas set to about 2GB (the
smallest we seem to be able to set).  This prevents users for
consuming too much unpurchased disk space.  However, this continues to
allow users to create files and directories with incorrect group
ownership and it takes users a while to discover their error.  User
education and cleanup becomes a problem long after the user thinks
things are working.

We would like to have groups without quota definitions to not be able
to create any files.  This would prevent accidental file creation at
the first attempt.

Stuart Barkley

On Mon, 4 Dec 2017 at 08:46 -0000, Stephen Ulmer wrote:

> I don?t understand why not having permission(s) doesn?t prevent the
> user from writing into the fileset...
>
> As described, your case is about not wanting userA to be able to
> write to a fileset if userA isn?t in some groups. Don?t put them in
> those groups. That?s not even Spectrum Scale specific, it?s about
> generic *nix permissions.
>
> What am I missing? I don?t understand why you would want to use
> quota to enforce permissions. (There could be a legitimate reason
> here, but I don?t understand it.)
>
> Liberty,
>
> --
> Stephen Ulmer
>
> Sent from a mobile device; please excuse autocorrect silliness.
>
> > On Dec 3, 2017, at 10:49 PM, IBM Spectrum Scale <scale at us.ibm.com> wrote:
> >
> > Hi Keith,
> >
> > You can use ACLs for fine grained permissions. A quota limit of 0
> > in GPFS implies no limits.
> >
> > Regards, The Spectrum Scale (GPFS) team
> >
> > From:        Keith Ball <bipcuds at gmail.com>
> > To:        gpfsug-discuss at spectrumscale.org
> > Date:        12/04/2017 08:19 AM
> > Subject:        [gpfsug-discuss] Smallest block quota/limit and file quota/limit        possible to set?
> > Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> >
> > HI All,
> >
> > We have a system where all users have their own private group as
> > well. However, for a given fileset (we are using
> > --perfileset-quota), we would like to ONLY allow users who also
> > belong to just a few central groups to be able to write to the
> > fileset.
> >
> > That is, user "userA" has its own "groupA", but we only want the
> > user to be able to write to the fileset if:
> >  - userA belongs to one of the groups (e.g. group1, group2,
> >  group3) that have explicitly set quotas
> >  - The group(s) in question are within quota/limits.
> >
> > In general, we do not want any users that do NOT belong to one of
> > the three groups with enabled quotas to be able to write anything
> > at all to the fileset.
> >
> > Is there a way to set a ZERO quota for block/file in GPFS, that
> > means what it actually should mean? i.e. "Your limit is 0 file =
> > you cannot create files in this fileset". Creating some kind of
> > "supergroup" owner of the fileset (with entitled users as members
> > of the group) could work, but that will only work for *one* group.
> >
> > If we cannot set the block and file limits to zero, what *are* the
> > smallest block and fie limits? In GPFS 3.5, they seem to be 1760MB
> > for block. Is there a smallest quota for files? (blocksize is
> > 16MB, which will be reduced to 4MB probably, in a subsequent
> > cluster).
> >
> > Many Thanks,
> >   Keith
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From stuartb at 4gh.net  Mon Dec  4 20:24:50 2017
From: stuartb at 4gh.net (Stuart Barkley)
Date: Mon, 4 Dec 2017 15:24:50 -0500 (EST)
Subject: [gpfsug-discuss] Smallest block quota/limit and file
 quota/limit possible to set?
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>,
	<alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
Message-ID: <alpine.BSF.2.21.1712041521500.34093@freeman.4gh.net>

On Mon, 4 Dec 2017 at 12:08 -0000, Simon Thompson (IT Research Support) wrote:

> Have you looked at using filesets instead an using fileset quotas to
> achieve this?
>
> This is what we do and the max number of filesets (currently) isn't
> an issue for us.

This is a possibility.  We will be moving to a new infrastructure in a
couple of months and I'll have our administrator and contractor
discuss this possibility.

Thanks,
Stuart
-- 
I've never been lost; I was once bewildered for three days, but never lost!
                                        --  Daniel Boone


From valdis.kletnieks at vt.edu  Mon Dec  4 20:44:34 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Mon, 04 Dec 2017 15:44:34 -0500
Subject: [gpfsug-discuss] Smallest block quota/limit and file
	quota/limit possible to set?
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
References: <OF8182F1B4.05AEE513-ON002581EC.000F8022@LocalDomain>
	<OF9112D31A.207179CF-ON852581EC.00147F44-652581EC.00150622@notes.na.collabserv.com>
	<8EB56403-70D4-42D5-88F9-32A0D4D68A67@ulmer.org>,
	<alpine.BSF.2.21.1712041049350.34093@freeman.4gh.net>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A072@EX13.adf.bham.ac.uk>
Message-ID: <50130.1512420274@turing-police.cc.vt.edu>

On Mon, 04 Dec 2017 17:08:19 +0000, "Simon Thompson (IT Research Support)" said:
> Have you looked at using filesets instead an using fileset quotas to achieve this?

Note that fileset quotas aren't able to represent "No Storage Allowed" either....
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171204/c92bcc95/attachment-0002.sig>

From Greg.Lehmann at csiro.au  Mon Dec  4 23:28:05 2017
From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au)
Date: Mon, 4 Dec 2017 23:28:05 +0000
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712041036340.2395@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
	<B89C6202-6723-4030-BFE3-B7AA59FCB956@bham.ac.uk>
	<alpine.LFD.2.20.1712041036340.2395@laptop>
Message-ID: <e65b855db0924e62abed07a24be62251@exch1-cdc.nexus.csiro.au>

We run GPFS client SW on SLES 12 SP2 which has a 4.4 kernel. It is only at 4.2.3-1 at present.

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of z.han at imperial.ac.uk
Sent: Monday, 4 December 2017 8:38 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or above?

Thanks, Simon.

Only on clients (oh so-called clients!) ;)


On Mon, 4 Dec 2017, Simon Thompson (IT Research Support) wrote:
> Date: Mon, 4 Dec 2017 10:29:27 +0000
> From: "Simon Thompson (IT Research Support)" <S.J.Thompson at bham.ac.uk>
> Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
>     above?
> 
> The FAQ at:
>
> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux
>
> Lists support with (e.g. Ubutu 16.04.2) with kernel 4.4.0-62, so likely it would work with a build your own kernel, but that doesn?t mean it is **supported**
>
> Simon
>
> On 04/12/2017, 09:52, "gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk" <gpfsug-discuss-bounces at spectrumscale.org on behalf of z.han at imperial.ac.uk> wrote:
>
>    Hi All,
>
>    Any one is using a Linux kernel 3.12.x or above
>    to run gpfs 4.2.3-4.2? I mean you've compiled
>    your own kernel without paying for a professional
>    service.
>
>    We're stuck by CentOS/RHEL's distributed kernel
>    as the PCI passthrough is required for VMs. Your
>    comments or suggestions are much appreciated.
>
>    Kind regards,
>
>    Zong-Pei
>
>    _______________________________________________
>    gpfsug-discuss mailing list
>    gpfsug-discuss at spectrumscale.org
>    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

From r.sobey at imperial.ac.uk  Tue Dec  5 10:38:37 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 5 Dec 2017 10:38:37 +0000
Subject: [gpfsug-discuss] Hidden fileset
Message-ID: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Quick question: is there a way to make a fileset hidden so it doesn't appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

I want to migrate a load of files in but I don't want the customer to access them prematurely.

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/ab0283f4/attachment-0002.htm>

From robert.horton at icr.ac.uk  Tue Dec  5 10:47:51 2017
From: robert.horton at icr.ac.uk (Robert Horton)
Date: Tue, 5 Dec 2017 10:47:51 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <1512470871.4350.27.camel@icr.ac.uk>

On Tue, 2017-12-05 at 10:38 +0000, Sobey, Richard A wrote:
Quick question: is there a way to make a fileset hidden so it doesn?t appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

Can you link it somewhere obscure, put the data in, then re-link where you actually want it?

Rob

--

Robert Horton | Research Data Storage Lead
The Institute of Cancer Research | 237 Fulham Road | London | SW3 6JB
T +44 (0)20 7153 5350 | E robert.horton at icr.ac.uk<mailto:dugan.witherick at icr.ac.uk> | W www.icr.ac.uk | Twitter @ICR_London
Facebook: www.facebook.com/theinstituteofcancerresearch

Making the discoveries that defeat cancer

The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.

This e-mail message is confidential and for use by the addressee only.  If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/d2edfd94/attachment-0002.htm>

From jonathan.buzzard at strath.ac.uk  Tue Dec  5 10:56:19 2017
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Tue, 05 Dec 2017 10:56:19 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <1512470871.4350.27.camel@icr.ac.uk>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<1512470871.4350.27.camel@icr.ac.uk>
Message-ID: <1512471379.18554.162.camel@strath.ac.uk>

On Tue, 2017-12-05 at 10:47 +0000, Robert Horton wrote:
> On Tue, 2017-12-05 at 10:38 +0000, Sobey, Richard A wrote:
> > Quick question: is there a way to make a fileset hidden so it
> > doesn?t appear when browsing the filesystem through a CES SMB
> > export? Could I do something clever with the directory mask?
> 
> Can you link it somewhere obscure, put the data in, then re-link
> where you actually want it?
> 

Would not setting the DOS hidden attribute on the directory do the
trick? That's assuming the CES SMB is doing everything correctly, no
experience with that but if it is that will do the trick.

Obviously if a user has Explorer set to view hidden files and folders
they will see it.

The other option would be to do something like

chown root:root /path/to/fileset
chmod 700 /path/to/fileset

Then from memory Samba won't show it because you don't have permissions
to access it.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG



From chair at spectrumscale.org  Tue Dec  5 10:58:33 2017
From: chair at spectrumscale.org (Simon Thompson)
Date: Tue, 05 Dec 2017 10:58:33 +0000
Subject: [gpfsug-discuss] SSUG @ CIUK next week [Tuesday 12th December 2017]
Message-ID: <00CFF55D-4D04-4F8D-90D9-F26AF7DA3B1D@spectrumscale.org>

Hi All,

 

Just a reminder that the last #SSUG of 2017 is happening next week as part of Computing Insights UK (CIUK) in Manchester. The agenda for the session is:

 

2pm Introductions and welcome, Simon Thompson & Claire O?Toole (SSUG Committee) 
2:10pm Spectrum Scale 5.0, Daniel Kidger (IBM) 
2:40pm Deep Dive: Blueprints, Ulf Troppens (IBM) 
3:40pm User Talk: Institute of Cancer Research, Robert Horton (ICR) 
4:00pm User Talk: Red Bull Technology, John Brookes (Red Bull Technology) 
4:20pm Questions and discussion 
4:50pm Close

 

To attend, you must be registered as a delegate of CIUK:

https://eventbooking.stfc.ac.uk/news-events/ciuk-2017

 

Simon

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/22096afa/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Tue Dec  5 11:04:40 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Tue, 5 Dec 2017 11:04:40 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A3A3@EX13.adf.bham.ac.uk>

Don't give them permissions on the fileset and use the smb option:

"hideunreadable=yes"

https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htmhttps://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htm

This enables access based enumeration.

Simon
________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sobey, Richard A [r.sobey at imperial.ac.uk]
Sent: 05 December 2017 10:38
To: 'gpfsug-discuss at spectrumscale.org'
Subject: ?spam? [gpfsug-discuss] Hidden fileset

Quick question: is there a way to make a fileset hidden so it doesn?t appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

I want to migrate a load of files in but I don?t want the customer to access them prematurely.

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/4ae420b2/attachment-0002.htm>

From r.sobey at imperial.ac.uk  Tue Dec  5 11:14:03 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 5 Dec 2017 11:14:03 +0000
Subject: [gpfsug-discuss] Hidden fileset
In-Reply-To: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A3A3@EX13.adf.bham.ac.uk>
References: <DB6PR0601MB21018269F75664188F9501D3DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A2A3A3@EX13.adf.bham.ac.uk>
Message-ID: <DB6PR0601MB2101298220A37AA14C2DA71EDF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Holy.. why did I not know about that option ?

Will look into that.

And Robert ? yes linking it somewhere else is my current option but it just makes TSM back it all up again when it gets linked to the correct path. Next on the agenda is how to exclude certain directories from the backup.

Thanks
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support)
Sent: 05 December 2017 11:05
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Hidden fileset

Don't give them permissions on the fileset and use the smb option:

"hideunreadable=yes"

https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htmhttps://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_mmsmb.htm

This enables access based enumeration.

Simon
________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Sobey, Richard A [r.sobey at imperial.ac.uk]
Sent: 05 December 2017 10:38
To: 'gpfsug-discuss at spectrumscale.org'
Subject: ?spam? [gpfsug-discuss] Hidden fileset
Quick question: is there a way to make a fileset hidden so it doesn?t appear when browsing the filesystem through a CES SMB export? Could I do something clever with the directory mask?

I want to migrate a load of files in but I don?t want the customer to access them prematurely.

Thanks
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/135ccbd2/attachment-0002.htm>

From andreas.mattsson at maxiv.lu.se  Tue Dec  5 13:15:38 2017
From: andreas.mattsson at maxiv.lu.se (Andreas Mattsson)
Date: Tue, 5 Dec 2017 13:15:38 +0000
Subject: [gpfsug-discuss] Infiniband connection rejected,
	ibv_create_qp err 13
Message-ID: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>

Hi.

Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message "ibv_create_qp err 13"?
I'm having issues with this on a IBM ESS system.

The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to.
The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs.
MPI-jobs on the client nodes can communicate over the infiniband fabric without issues.
Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue.
When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted interfaces.

I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this.

Any help or suggestions is appreciated.
Regards,
Andreas Mattsson
_____________________________________________
[cid:part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer

MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/0e987953/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5610 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/0e987953/attachment-0002.png>

From aaron.s.knister at nasa.gov  Tue Dec  5 13:23:43 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Tue, 5 Dec 2017 13:23:43 +0000
Subject: [gpfsug-discuss] Infiniband connection rejected,
 ibv_create_qp err 13
In-Reply-To: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>
References: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>
Message-ID: <957FBC22-DDAF-4BEF-95DA-1DB7B9AB2CCA@nasa.gov>



Looks like 13 is EPERM which means apparently permissions didn?t exist to create the QP of the desired type which is odd since mmfsd runs as root. Is there any remote chance SELinux is enabled (e.g. sestatus)? Although I?d think mmfsd would run unconfined in the default policy, but maybe it didn?t transition correctly.

On December 5, 2017 at 08:16:49 EST, Andreas Mattsson <andreas.mattsson at maxiv.lu.se> wrote:

Hi.



Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message ?ibv_create_qp err 13??

I?m having issues with this on a IBM ESS system.



The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to.

The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs.

MPI-jobs on the client nodes can communicate over the infiniband fabric without issues.

Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue.

When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted interfaces.



I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this.



Any help or suggestions is appreciated.

Regards,

Andreas Mattsson

_____________________________________________

[mid:d8d07f7e01ec4fcca5ae124f40c2d457 at maxiv.lu.se/part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer



MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/1d5fad85/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5610 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/1d5fad85/attachment-0002.png>

From andreas.mattsson at maxiv.lu.se  Tue Dec  5 14:22:11 2017
From: andreas.mattsson at maxiv.lu.se (Andreas Mattsson)
Date: Tue, 5 Dec 2017 14:22:11 +0000
Subject: [gpfsug-discuss] Infiniband connection rejected,
 ibv_create_qp err 13
In-Reply-To: <957FBC22-DDAF-4BEF-95DA-1DB7B9AB2CCA@nasa.gov>
References: <d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se>
	<957FBC22-DDAF-4BEF-95DA-1DB7B9AB2CCA@nasa.gov>
Message-ID: <869901bdbfd347969d21d619204a9b87@maxiv.lu.se>

Thanks!
I?ve double checked the selinux state, and it is disabled on both all the ESS nodes and on all the client nodes.
mmfsd is running as root on all nodes as well.
It seem a bit strange that this would be an issue of permissions though. I forgot to state this in my original question, but the issue comes and goes, and can affect some clients while not affecting others at the same time, and which clients are affected at any given time changes as well.
Just a thought; How does infiniband queue pairs react to time skew between nodes?

For future reference, where did you find the specification of ibv_create_qp error 13?
I must have been looking in all the wrong places, because I?ve been unable to find the meaning of this error.

Regards,
Andreas
_____________________________________________
[cid:part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer

MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>

Fr?n: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] F?r Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]
Skickat: den 5 december 2017 14:24
Till: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
?mne: Re: [gpfsug-discuss] Infiniband connection rejected, ibv_create_qp err 13



Looks like 13 is EPERM which means apparently permissions didn?t exist to create the QP of the desired type which is odd since mmfsd runs as root. Is there any remote chance SELinux is enabled (e.g. sestatus)? Although I?d think mmfsd would run unconfined in the default policy, but maybe it didn?t transition correctly.
On December 5, 2017 at 08:16:49 EST, Andreas Mattsson <andreas.mattsson at maxiv.lu.se<mailto:andreas.mattsson at maxiv.lu.se>> wrote:

Hi.



Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message ?ibv_create_qp err 13??

I?m having issues with this on a IBM ESS system.



The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to.

The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs.

MPI-jobs on the client nodes can communicate over the infiniband fabric without issues.

Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue.

When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted interfaces.



I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this.



Any help or suggestions is appreciated.

Regards,

Andreas Mattsson

_____________________________________________

[mid:d8d07f7e01ec4fcca5ae124f40c2d457 at maxiv.lu.se/part1.08040705.03090509 at maxiv.lu.se]

Andreas Mattsson
Systems Engineer



MAX IV Laboratory
Lund University
P.O. Box 118, SE-221 00 Lund, Sweden
Visiting address: Fotongatan 2, 225 94 Lund
Mobile: +46 706 64 95 44
andreas.mattsson at maxiv.se<mailto:andreas.mattsson at maxiv.se>
www.maxiv.se<http://www.maxiv.se/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/9909e177/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5610 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/9909e177/attachment-0002.png>

From r.sobey at imperial.ac.uk  Tue Dec  5 14:40:33 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 5 Dec 2017 14:40:33 +0000
Subject: [gpfsug-discuss] Policy rule to list files based on access time
In-Reply-To: <DB6PR0601MB210125AFF8783FFFAB2BF3E4DF3E0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <DB6PR0601MB210119F1CDA7E13C6FECB923DF390@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<OF44BD7BA5.7E0CCB1A-ON002581E9.005F2AF2@LocalDomain>,
	<OFF9B65025.E670F3E6-ON852581EA.00626D6D-852581EA.00633103@notes.na.collabserv.com>
	<DB6PR0601MB210125AFF8783FFFAB2BF3E4DF3E0@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <DB6PR0601MB210100429C099C1A2BFBD016DF3D0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Thanks all, managed to rerun the policy with the additional list options (kb_allocated etc) and verified that indeed there are at least 60000 files with the same name that are 0 bytes, and more besides.

Now my paranoia is setting in that somehow these files are all corrupted ?

Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 02 December 2017 18:53
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Policy rule to list files based on access time

Thank you all for your replies. I will take a look at them on Monday. And reply individually where I need to..
Richard
Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Marc A Kaplan <makaplan at us.ibm.com>
Sent: Saturday, December 2, 2017 6:03:25 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Policy rule to list files based on access time

You might want to look at FILE_SIZE.  KB_ALLOCATED will be 0 if the file data fits into the inode.

You might also want to use SIZE(FILE_SIZE) in the policy LIST rule, this will cause the KB_HIT and KB_CHOSEN numbers to be the sum of FILE_SIZEs instead of the default SIZE(KB_ALLOCATED).

--marc of GPFS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/9593f4f0/attachment-0002.htm>

From Mark.Bush at siriuscom.com  Tue Dec  5 22:43:35 2017
From: Mark.Bush at siriuscom.com (Mark Bush)
Date: Tue, 5 Dec 2017 22:43:35 +0000
Subject: [gpfsug-discuss] CES and local workstations
Message-ID: <DM5PR05MB35774E1C84500C5EBAA9A5E0E43D0@DM5PR05MB3577.namprd05.prod.outlook.com>

I have a windows 10 machine that is part of my local domain.  I have a separate SpecScale test cluster that has local (not part of my AD domain) ldap and CES (NFS/SMB) running.  I cannot get my local workstation to connect to a SMB share at all.  When I get the logon prompt I'm using IBM NAS\Administrator and the password for that user but it never lets me in.  Is this just defaulting to regular linux SAMBA authentication mechanisms at this point?  Of is the SAMBA code in the gpfs packages unique somehow in this?  I've confirmed that my mmuserauth service appears correct and that ldap is working properly. Any clues?

Frustrated.



Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/72015aba/attachment-0002.htm>

From daniel.kidger at uk.ibm.com  Tue Dec  5 22:44:31 2017
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Tue, 5 Dec 2017 22:44:31 +0000
Subject: [gpfsug-discuss] mmauth/mmremotecluster wonkyness?
In-Reply-To: <OF7BE5A1DA.D530374F-ON002581E8.005B1ECF@LocalDomain>
Message-ID: <OFE0EA91FB.30381E41-ON002581ED.007CED24-1512513871966@notes.na.collabserv.com>

If you think about it, if a CES node opens a file for an NFS client and then a client in a remote MC mounted cluster opens the same file then that MC client node needs to talk to that CES node to negotiate the lock.

Daniel


 

 
 	
Dr Daniel Kidger 
IBM Technical Sales Specialist
Software Defined Solution Sales

+ 44-(0)7818 522 266 
daniel.kidger at uk.ibm.com

> On 30 Nov 2017, at 17:35, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:
> 
> It?s my understanding and experience that all member nodes of two clusters that are multi-clustered must be able to (and will eventually given enough time/activity) make connections to any and all nodes in both clusters. Even if you don?t designate the 2 protocol nodes as contact nodes I would expect to see connections from remote clusters to the protocol nodes just because of the nature of the beast. If you don?t want remote nodes to make connections to the protocol nodes then I believe you would need to put the protocol nodes in their own cluster. CES/CNFS hasn?t always supported this but I think it is now supported, at least with NFS. 
> 
> 
> 
> 
> 
>> On November 30, 2017 at 11:28:03 EST, valdis.kletnieks at vt.edu wrote:
>> We have a 10-node cluster running gpfs 4.2.2.3, where 8 nodes are GPFS contact 
>> nodes for 2 filesystems, and 2 are protocol nodes doingNFS exports of the 
>> filesystems. 
>> 
>> But we see some nodes in remote clusters trying to GPFS connect to 
>> the 2 protocol nodes anyhow. 
>> 
>> My reading of the manpages is that the remote cluster is responsible 
>> for setting '-n contactNodes' when they do the 'mmremotecluster add', 
>> and there's no way to sanity check or enforce that at the local end, and 
>> fail/flag connections to unintended non-contact nodes if the remote 
>> admin forgets/botches the -n. 
>> 
>> Is that actually correct? If so, is it time for an RFE? 
>> _______________________________________________ 
>> gpfsug-discuss mailing list 
>> gpfsug-discuss at spectrumscale.org 
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171205/da71f602/attachment-0002.htm>

From z.han at imperial.ac.uk  Wed Dec  6 14:04:00 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Wed, 6 Dec 2017 14:04:00 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712040941230.2395@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
Message-ID: <alpine.LFD.2.20.1712061358300.10433@laptop>

I'm pleased to report that kernel 4.4.103 has been
successfully compiled and works with gpfs 4.2.3-4.2,
after a few failed attempts for other 3/4 kernels!

The only minor issue is the iDRAC virtual console
stops working, but fortunately the serial connection
is fine. Any expert advice will be warmly received.

Kind regards,

Zong-Pei


From z.han at imperial.ac.uk  Wed Dec  6 14:05:46 2017
From: z.han at imperial.ac.uk (z.han at imperial.ac.uk)
Date: Wed, 6 Dec 2017 14:05:46 +0000 (GMT)
Subject: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
 above?
In-Reply-To: <alpine.LFD.2.20.1712061358300.10433@laptop>
References: <alpine.LFD.2.20.1712040941230.2395@laptop>
	<alpine.LFD.2.20.1712061358300.10433@laptop>
Message-ID: <alpine.LFD.2.20.1712061404430.10433@laptop>

PS

On CentOS Linux release 7.4.1708 (Core)


On Wed, 6 Dec 2017, z.han at imperial.ac.uk wrote:
> Date: Wed, 6 Dec 2017 14:04:00 +0000
> From: z.han at imperial.ac.uk
> Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Can gpfs 4.2.3-4.2 work for kernel 3.12.x or
>     above?
> 
> I'm pleased to report that kernel 4.4.103 has been
> successfully compiled and works with gpfs 4.2.3-4.2,
> after a few failed attempts for other 3/4 kernels!
>
> The only minor issue is the iDRAC virtual console
> stops working, but fortunately the serial connection
> is fine. Any expert advice will be warmly received.
>
> Kind regards,
>
> Zong-Pei
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>


From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Dec  6 21:56:33 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 6 Dec 2017 21:56:33 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
Message-ID: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>

Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/886bd1a4/attachment-0002.htm>

From jdkwedar at gmail.com  Wed Dec  6 22:10:11 2017
From: jdkwedar at gmail.com (Joshua Kwedar (froz1))
Date: Wed, 6 Dec 2017 17:10:11 -0500
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
Message-ID: <2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)

chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]

Josh K

> On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu> wrote:
> 
> Hi All,
> 
> So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(
> 
> Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.
> 
> Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)
> 
> Thanks?
> 
> ?
> Kevin Buterbaugh - Senior System Administrator
> Vanderbilt University - Advanced Computing Center for Research and Education
> Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/c23c3d52/attachment-0002.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Dec  6 22:13:46 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 6 Dec 2017 22:13:46 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
Message-ID: <26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID  {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]


Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:

Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/28c88bcc/attachment-0002.htm>

From jdkwedar at gmail.com  Wed Dec  6 22:32:18 2017
From: jdkwedar at gmail.com (Joshua Kwedar)
Date: Wed, 06 Dec 2017 17:32:18 -0500
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
Message-ID: <D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>

Hmm.. odd.? 

 

Here?s what the lsuser output should look like.? 

 

# /usr/lpp/mmfs/gui/cli/lsuser

Name? Long name Password status Group names???????????????? Failed login attempts

admin?????????? active????????? Administrator,SecurityAdmin 0

EFSSG1000I The command completed successfully.

 

Can you try something like?

 

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin

 

 

 

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

 

All, 

 

Sorry - should?ve mentioned that:

 

/root

root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231

EFSSG0001C Cannot validate option: login

/root

root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y

lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:

/root

root at testnsd1# 

 

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

 

Kevin



On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com> wrote:

 

The GUI password can be changed via command line using chuser.

 

/usr/lpp/mmfs/gui/cli/chuser



Usage is as follows (where userID = admin)



chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]


Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu> wrote:

Hi All, 

 

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

 

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

 

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

 

Thanks?

 

?

Kevin Buterbaugh - Senior System Administrator

Vanderbilt University - Advanced Computing Center for Research and Education

Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633

 

 

 

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

 

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/aec01adf/attachment-0002.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Wed Dec  6 22:41:12 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Wed, 6 Dec 2017 22:41:12 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
Message-ID: <D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin

On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin


On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser


Usage is as follows (where userID = admin)



chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171206/531a9a8d/attachment-0002.htm>

From r.sobey at imperial.ac.uk  Thu Dec  7 11:57:07 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 7 Dec 2017 11:57:07 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>,
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
Message-ID: <DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin

On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin


On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser


Usage is as follows (where userID = admin)



chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171207/c3bdf293/attachment-0002.htm>

From r.sobey at imperial.ac.uk  Thu Dec  7 12:10:52 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Thu, 7 Dec 2017 12:10:52 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>,
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
	<DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Sorry I need to learn to read... didn't see the "object 'Administrator' does not exist" error.

That said, my workaround for the problem of forgetting the password was to create a new "admin2" user and use that to reset the password on admin itself.

[root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin
EFSSG0019I The user admin2 has been successfully created.
EFSSG1000I The command completed successfully.


Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 07 December 2017 11:57
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.
Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I'll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it's subsequently been updated to GPFS 4.2.3.5 ... not sure that's relevant but wanted to mention it just in case.

Thanks!

Kevin


On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here's what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like...

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should've mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it's like the login doesn't exist ... but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I've forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM's, re-installing, and starting over from scratch?  If that's what I have to do, it's no big deal as this is just our little 6-node test cluster, but I thought I'd ask before going down that route.

Oh, and if someone has a way to accomplish this that they'd rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won't tell if you won't tell (and hopefully Michael Flynn won't tell either!).... ;-)

Thanks...

-
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171207/9e93a421/attachment-0002.htm>

From bipcuds at gmail.com  Thu Dec  7 22:48:49 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Thu, 7 Dec 2017 17:48:49 -0500
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <CAAxuGpHj+4z6cLff69Z25CqfUaTWFjoV9vNGWhHSEUqWgWOZ9w@mail.gmail.com>

Hi All,

In working with per-fileset quotas (not user/group/fileset quotas at the
filesystem level), I have the follwing issues/questions.

1.) Setting grace periods. I notice that some of the group quotas in a
specific fileset have a grace period (or remaining grace period) of X days,
while others report "none":

# mmrepquota -g --block-size G hpc-fs:fileset01
                         Block Limits
|                     File Limits
Name       fileset    type             GB      quota      limit
in_doubt    grace |    files   quota    limit in_doubt    grace
groupa      fileset01 GRP            2257          2          2
0   4 days |      143     100      100        0   4 days
root       fileset01 GRP             710          0          0
0     none |    15578       0        0        0     none
groupb   fileset01 GRP            2106        400        400          0   4
days |        1 1048576  1048576        0     none
...

How can I set a grace period of "none" on group quotas? mmsetquota does not
appear (from the man pages) to provide any way to set grace periods for
per-fileset quotas:

mmsetquota Device ??grace {user | group | fileset}
           {[??block GracePeriod] [??files GracePeriod]}

How can I set to "none" or "0days"? (i.e. no grace period given if over
quota). Or, for that matter, set grace periods for any duration at all?


2.) How to remove any explicit quota settings for (not just deactivating
default quota settings) at the per-fileset level. The mmdefquotaoff docs
seem to suggest that the ?-d? option will not remove explicit per-fileset
quota settings if they are non-zero (so really, what use is the -d option
then?)

Many Thanks!
  Keith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171207/b790fd92/attachment-0002.htm>

From Robert.Oesterlin at nuance.com  Fri Dec  8 12:39:58 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Fri, 8 Dec 2017 12:39:58 +0000
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <D7F39BE9-B69E-4AB4-8E6E-7B8FA022EE9D@nuance.com>

1) One way you can set the grace period to none by setting the soft and hard quotas to the same value.

2) By removing any quota setting, do you mean unlimited quota? Which is just setting it (quota) to zero. (meaning unlimited)

Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Keith Ball <bipcuds at gmail.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Thursday, December 7, 2017 at 4:49 PM
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] [gpfsug-discuss] Working with per-fileset quotas

How can I set a grace period of "none" on group quotas? mmsetquota does not appear (from the man pages) to provide any way to set grace periods for per-fileset quotas:

mmsetquota Device ??grace {user | group | fileset}
           {[??block GracePeriod] [??files GracePeriod]}
How can I set to "none" or "0days"? (i.e. no grace period given if over quota). Or, for that matter, set grace periods for any duration at all?


2.) How to remove any explicit quota settings for (not just deactivating default quota settings) at the per-fileset level. The mmdefquotaoff docs seem to suggest that the ?-d? option will not remove explicit per-fileset quota settings if they are non-zero (so really, what use is the -d option then?)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/c006d850/attachment-0002.htm>

From truongv at us.ibm.com  Fri Dec  8 15:41:04 2017
From: truongv at us.ibm.com (Truong Vu)
Date: Fri, 8 Dec 2017 10:41:04 -0500
Subject: [gpfsug-discuss] Working with per-fileset quotas
In-Reply-To: <OF2C4BFBB9.F213F6C2-ON002581F0.0041F153@LocalDomain>
References: <OF2C4BFBB9.F213F6C2-ON002581F0.0041F153@LocalDomain>
Message-ID: <OF95F138D7.BB03BC26-ON852581F0.00560D00-852581F0.00562863@notes.na.collabserv.com>


1) That is correct.  The grace period can't be set for per-fileset
quota.  As you pointed out, you can only change the grace period for
user, group or fileset.

If you want a particular fileset to have no grace period, you can
set the hard limit to be the same as the soft limit.

When the grace column shows "none", this means the soft limit has
not been reached.  Once the soft limit is reached, the grace period
is start counting.

2) To remove explicit quota settings, you need to set the limit to 0.




From:	gpfsug-discuss-request at spectrumscale.org
To:	gpfsug-discuss at spectrumscale.org
Date:	12/08/2017 07:00 AM
Subject:	gpfsug-discuss Digest, Vol 71, Issue 19
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Send gpfsug-discuss mailing list submissions to
		 gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit

https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=5OySIaqfU0j1miWKKp6aydLjiGbE8z5pDz5JGveRRlQ&s=HKXwjIgPAsNTzNSL3-FrvHAXVyvZdzGYugbZgJ3FvMI&e=

or, via email, send a message with subject or body 'help' to
		 gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
		 gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Working with per-fileset quotas (Keith Ball)


----------------------------------------------------------------------

Message: 1
Date: Thu, 7 Dec 2017 17:48:49 -0500
From: Keith Ball <bipcuds at gmail.com>
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID:
		 <CAAxuGpHj
+4z6cLff69Z25CqfUaTWFjoV9vNGWhHSEUqWgWOZ9w at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi All,

In working with per-fileset quotas (not user/group/fileset quotas at the
filesystem level), I have the follwing issues/questions.

1.) Setting grace periods. I notice that some of the group quotas in a
specific fileset have a grace period (or remaining grace period) of X days,
while others report "none":

# mmrepquota -g --block-size G hpc-fs:fileset01
                         Block Limits
|                     File Limits
Name       fileset    type             GB      quota      limit
in_doubt    grace |    files   quota    limit in_doubt    grace
groupa      fileset01 GRP            2257          2          2
0   4 days |      143     100      100        0   4 days
root       fileset01 GRP             710          0          0
0     none |    15578       0        0        0     none
groupb   fileset01 GRP            2106        400        400          0   4
days |        1 1048576  1048576        0     none
...

How can I set a grace period of "none" on group quotas? mmsetquota does not
appear (from the man pages) to provide any way to set grace periods for
per-fileset quotas:

mmsetquota Device ??grace {user | group | fileset}
           {[??block GracePeriod] [??files GracePeriod]}

How can I set to "none" or "0days"? (i.e. no grace period given if over
quota). Or, for that matter, set grace periods for any duration at all?


2.) How to remove any explicit quota settings for (not just deactivating
default quota settings) at the per-fileset level. The mmdefquotaoff docs
seem to suggest that the ?-d? option will not remove explicit per-fileset
quota settings if they are non-zero (so really, what use is the -d option
then?)

Many Thanks!
  Keith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20171207_b790fd92_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=5OySIaqfU0j1miWKKp6aydLjiGbE8z5pDz5JGveRRlQ&s=-DNcYGwFlUrOQZsZ9yQAYBdVv46u1xxeA4wJ2-VxQ_A&e=
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=5OySIaqfU0j1miWKKp6aydLjiGbE8z5pDz5JGveRRlQ&s=HKXwjIgPAsNTzNSL3-FrvHAXVyvZdzGYugbZgJ3FvMI&e=



End of gpfsug-discuss Digest, Vol 71, Issue 19
**********************************************



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/790f55b2/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/790f55b2/attachment-0002.gif>

From eric.wonderley at vt.edu  Fri Dec  8 16:10:16 2017
From: eric.wonderley at vt.edu (J. Eric Wonderley)
Date: Fri, 8 Dec 2017 11:10:16 -0500
Subject: [gpfsug-discuss] mm'add|del'node with ccr enabled
Message-ID: <CABOSGQf2JTQdp3E+8dkFXY07x5u9SnePh0yhAgyD6YMty2WhHA@mail.gmail.com>

Hello:

If I recall correctly this does not work...correct?  I think the last time
I attempted this was gpfs version <=4.1.  I think I attempted to add a
quorum node.

The process was that I remember doing was mmshutdown -a, mmchcluster
--ccr-disable, mmaddnode yadayada, mmchcluster --ccr-enable, mmstartup.

I think with ccr disabled mmaddnode can be run with gpfs up.  We would like
to run with ccr enabled but it does make adding/removing nodes unpleasant.

Would this be required of a non-quorum node?

Any changes concerning this with gpfs version >=4.2?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/850462fc/attachment-0002.htm>

From Robert.Oesterlin at nuance.com  Fri Dec  8 16:17:13 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Fri, 8 Dec 2017 16:17:13 +0000
Subject: [gpfsug-discuss] mm'add|del'node with ccr enabled
Message-ID: <A1F99FA0-3794-4989-A1A2-D81A038765B5@nuance.com>

Yes, there were issues with this in pre 4.2? versions of GPFS. My clusters are at 4.2.2 with CCR (working my way to 4.2.3-5) and I?ve been using this process on my quorum nodes when I need to update them:

mmchnode ?quorum -N new-node (designate a new quorum node, going from 3 to 4)
mmchnode ?nonquorum -N old-node (remove existing node to be updated)

I can then shutdown ?old-node? and then when I bring it back up, I reverse the process. I?ve had no issues using this process.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "J. Eric Wonderley" <eric.wonderley at vt.edu>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Friday, December 8, 2017 at 10:11 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] [gpfsug-discuss] mm'add|del'node with ccr enabled

Hello:

If I recall correctly this does not work...correct?  I think the last time I attempted this was gpfs version <=4.1.  I think I attempted to add a quorum node.

The process was that I remember doing was mmshutdown -a, mmchcluster --ccr-disable, mmaddnode yadayada, mmchcluster --ccr-enable, mmstartup.

I think with ccr disabled mmaddnode can be run with gpfs up.  We would like to run with ccr enabled but it does make adding/removing nodes unpleasant.

Would this be required of a non-quorum node?

Any changes concerning this with gpfs version >=4.2?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171208/da944b42/attachment-0002.htm>

From TROPPENS at de.ibm.com  Sat Dec  9 00:11:27 2017
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Sat, 9 Dec 2017 01:11:27 +0100
Subject: [gpfsug-discuss] Genomics Blueprint - SSUG @ CIUK next week
 [Tuesday 12th December 2017]
In-Reply-To: <OF0DB03636.FFEC975E-ON002581ED.003C507F@LocalDomain>
References: <OF0DB03636.FFEC975E-ON002581ED.003C507F@LocalDomain>
Message-ID: <OF3EFB1F2F.FBD960FA-ONC12581F1.00006B84-C12581F1.00010C76@notes.na.collabserv.com>

We just published the first blueprint:

IBM Spectrum Scale Best Practices for Genomics Medicine Workloads
http://www.redbooks.ibm.com/abstracts/redp5479.html

This blueprint is optimized for genomics workloads but most recommendations
apply to other workloads.

Looking forward to receive first-hand feedback at CIUK.


--
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery
Consulting IT Specialist
Author "Storage Networks Explained"

IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294




From:	Simon Thompson <chair at spectrumscale.org>
To:	"gpfsug-discuss at spectrumscale.org"
            <gpfsug-discuss at spectrumscale.org>
Date:	05.12.2017 11:58
Subject:	[gpfsug-discuss] SSUG @ CIUK next week [Tuesday 12th December
            2017]
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hi All,

Just a reminder that the last #SSUG of 2017 is happening next week as part
of Computing Insights UK (CIUK) in Manchester. The agenda for the session
is:

2pm Introductions and welcome, Simon Thompson & Claire O?Toole (SSUG
Committee)
2:10pm Spectrum Scale 5.0, Daniel Kidger (IBM)
2:40pm Deep Dive: Blueprints, Ulf Troppens (IBM)
3:40pm User Talk: Institute of Cancer Research, Robert Horton (ICR)
4:00pm User Talk: Red Bull Technology, John Brookes (Red Bull Technology)
4:20pm Questions and discussion
4:50pm Close

To attend, you must be registered as a delegate of CIUK:
https://eventbooking.stfc.ac.uk/news-events/ciuk-2017

Simon_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/9d16277d/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/9d16277d/attachment-0002.gif>

From Renar.Grunenberg at huk-coburg.de  Sat Dec  9 10:58:52 2017
From: Renar.Grunenberg at huk-coburg.de (Grunenberg, Renar)
Date: Sat, 9 Dec 2017 10:58:52 +0000
Subject: [gpfsug-discuss] mm'add|del'node with ccr enabled
In-Reply-To: <CABOSGQf2JTQdp3E+8dkFXY07x5u9SnePh0yhAgyD6YMty2WhHA@mail.gmail.com>
References: <CABOSGQf2JTQdp3E+8dkFXY07x5u9SnePh0yhAgyD6YMty2WhHA@mail.gmail.com>
Message-ID: <f926f5c352f146ceb49f5e14fb1e76fd@SMXRF105.msg.hukrf.de>

Hallo Eric,
our experiences are add and delete new/old nodes is working only if this node is no quorum node in an ccr cluster, no problem. There are no mmshutdown steps necessary. We are on 4.2.3.6. I think this is already available since >4.2. If you want to add a new quorum node, than you must put this node first as a client and after that you can change this node to a quorum node.


Renar Grunenberg
Abteilung Informatik ? Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon:        09561 96-44110
Telefax:        09561 96-44104
E-Mail: Renar.Grunenberg at huk-coburg.de
Internet:       www.huk.de
________________________________
HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Her?y, Dr. J?rg Rheinl?nder (stv.), Sarah R?ssler, Daniel Thomas (stv.).
________________________________
Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.

This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this information in error) please notify the
sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.
________________________________
Von: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von J. Eric Wonderley
Gesendet: Freitag, 8. Dezember 2017 17:10
An: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Betreff: [gpfsug-discuss] mm'add|del'node with ccr enabled

Hello:

If I recall correctly this does not work...correct?  I think the last time I attempted this was gpfs version <=4.1.  I think I attempted to add a quorum node.

The process was that I remember doing was mmshutdown -a, mmchcluster --ccr-disable, mmaddnode yadayada, mmchcluster --ccr-enable, mmstartup.

I think with ccr disabled mmaddnode can be run with gpfs up.  We would like to run with ccr enabled but it does make adding/removing nodes unpleasant.

Would this be required of a non-quorum node?

Any changes concerning this with gpfs version >=4.2?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/16c21559/attachment-0002.htm>

From bipcuds at gmail.com  Sat Dec  9 21:49:54 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Sat, 9 Dec 2017 16:49:54 -0500
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <CAAxuGpHH6e2K=mBTZQ=tA7wc-UMYmg7_8F9hvrh8Kn7Gh78_rA@mail.gmail.com>

I meant more specifically, remove any indication that quotas have been set,
for anyone (or everyone). That way, I could have a script to clean out old
quota definitions, then set both default and explicit quotas, without any
old settings lurking around. Is there a way to get rid of (e.g. zero out)
all existing explicit PER-FILESET quota definitions, and start over
(without having to look at mmrepquota and zero them out one by one)?


So I see your point about soft=lard limit => no grace period. I guess
what's odd is that I see any indication that the grace period is not
"none"; what grace period would be assigned to a per-fileset quota if I
cannot even view or set it?

Thanks,
   Keith


From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Working with per-fileset quotas
>
> 2) By removing any quota setting, do you mean unlimited quota? Which is
> just setting it (quota) to zero. (meaning unlimited)
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 8 Dec 2017 10:41:04 -0500
> From: "Truong Vu" <truongv at us.ibm.com>
> To: gpfsug-discuss at spectrumscale.org
> Subject: Re: [gpfsug-discuss] Working with per-fileset quotas
> Message-ID:
>         <OF95F138D7.BB03BC26-ON852581F0.00560D00-852581F0.00562863@
> notes.na.collabserv.com>
>
> Content-Type: text/plain; charset="us-ascii"
>
>
> 1) That is correct.  The grace period can't be set for per-fileset
> quota.  As you pointed out, you can only change the grace period for
> user, group or fileset.
>
> If you want a particular fileset to have no grace period, you can
> set the hard limit to be the same as the soft limit.
>
> When the grace column shows "none", this means the soft limit has
> not been reached.  Once the soft limit is reached, the grace period
> is start counting.
>
> 2) To remove explicit quota settings, you need to set the limit to 0.
>
>
>
>
> From:   gpfsug-discuss-request at spectrumscale.org
> To:     gpfsug-discuss at spectrumscale.org
> Date:   12/08/2017 07:00 AM
> Subject:        gpfsug-discuss Digest, Vol 71, Issue 19
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
>
>
>    1. Working with per-fileset quotas (Keith Ball)
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 7 Dec 2017 17:48:49 -0500
> From: Keith Ball <bipcuds at gmail.com>
> To: gpfsug-discuss at spectrumscale.org
> Subject: [gpfsug-discuss] Working with per-fileset quotas
> Message-ID:
>                  <CAAxuGpHj
> +4z6cLff69Z25CqfUaTWFjoV9vNGWhHSEUqWgWOZ9w at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi All,
>
> In working with per-fileset quotas (not user/group/fileset quotas at the
> filesystem level), I have the follwing issues/questions.
>
> 1.) Setting grace periods. I notice that some of the group quotas in a
> specific fileset have a grace period (or remaining grace period) of X days,
> while others report "none":
>
> # mmrepquota -g --block-size G hpc-fs:fileset01
>                          Block Limits
> |                     File Limits
> Name       fileset    type             GB      quota      limit
> in_doubt    grace |    files   quota    limit in_doubt    grace
> groupa      fileset01 GRP            2257          2          2
> 0   4 days |      143     100      100        0   4 days
> root       fileset01 GRP             710          0          0
> 0     none |    15578       0        0        0     none
> groupb   fileset01 GRP            2106        400        400          0   4
> days |        1 1048576  1048576        0     none
> ...
>
> How can I set a grace period of "none" on group quotas? mmsetquota does not
> appear (from the man pages) to provide any way to set grace periods for
> per-fileset quotas:
>
> mmsetquota Device ??grace {user | group | fileset}
>            {[??block GracePeriod] [??files GracePeriod]}
>
> How can I set to "none" or "0days"? (i.e. no grace period given if over
> quota). Or, for that matter, set grace periods for any duration at all?
>
>
> 2.) How to remove any explicit quota settings for (not just deactivating
> default quota settings) at the per-fileset level. The mmdefquotaoff docs
> seem to suggest that the ?-d? option will not remove explicit per-fileset
> quota settings if they are non-zero (so really, what use is the -d option
> then?)
>
> Many Thanks!
>   Keith
>
>

-- 
Keith D. Ball, PhD
RedLine Performance Solutions, LLC
web:  http://www.redlineperf.com/
email: kball at redlineperf.com <aqualkenbush at redlineperf.com>
cell: 540-557-7851 <(540)%20557-7851>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171209/203f7a33/attachment-0002.htm>

From Robert.Oesterlin at nuance.com  Sun Dec 10 01:27:13 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Sun, 10 Dec 2017 01:27:13 +0000
Subject: [gpfsug-discuss] Working with per-fileset quotas
Message-ID: <C1EFE45D-20E7-4689-87A6-7C5B2CE2ED5F@nuance.com>

Looking at the default case in a new file system/fileset, if you did an ?mmrepquota? you?d see everyone have a default quota of zero. Meaning ? any time you set a user/fileset/group quota back to zero, you are removing any trace of the previous quota. On you your specific question - yes, one by one is it, I?m not aware of any other way to do it.

Hard quota set, no soft quota: no grace period
Hard Soft quota set and equal: no grace period
Hard quota set or unset, soft quota set: grace period

The default behavior is no grace period unless soft quota is set, maybe that?s why it?s no displayed?


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Keith Ball <bipcuds at gmail.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Saturday, December 9, 2017 at 3:50 PM
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Working with per-fileset quotas

I meant more specifically, remove any indication that quotas have been set, for anyone (or everyone). That way, I could have a script to clean out old quota definitions, then set both default and explicit quotas, without any old settings lurking around. Is there a way to get rid of (e.g. zero out) all existing explicit PER-FILESET quota definitions, and start over (without having to look at mmrepquota and zero them out one by one)?


So I see your point about soft=lard limit => no grace period. I guess what's odd is that I see any indication that the grace period is not "none"; what grace period would be assigned to a per-fileset quota if I cannot even view or set it?


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171210/5b19346b/attachment-0002.htm>

From john.hearns at asml.com  Wed Dec 13 12:18:01 2017
From: john.hearns at asml.com (John Hearns)
Date: Wed, 13 Dec 2017 12:18:01 +0000
Subject: [gpfsug-discuss] Am I a gateway or not?
Message-ID: <HE1PR02MB14506B5498911ABC8CE08D4F88350@HE1PR02MB1450.eurprd02.prod.outlook.com>

Firstly, I hope everyone at the Manchester meeting is enjoying it and that the snow did not hamper too many travel plans.
I like Manchester - I shall tell the tale of blowing the 100Amp fuse in the University server room some other time.
(Yes - a fuse. Not a circuit breaker).

I am writing some monitoring scripts for AFM - actually to interface with Bright.
I would like the script to detect if the node which is running it is a Gateway node or has the AFM component.
If not, the script silently exits.
This is not a Callback script - I can write these and indeed have one for the AFM fileset demount event.

In order to determine which components are on a given node I could:

  1.  Run "mmhealth node show" and grep for AFM
  2.  Run mmlscluster and grep for $HOSTNAME

Both methods seem slightly inelegant. IS there a handy-dandy command I can run to see the components on this node?

John Hearns
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/22341ece/attachment-0002.htm>

From scale at us.ibm.com  Wed Dec 13 14:45:52 2017
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Wed, 13 Dec 2017 09:45:52 -0500
Subject: [gpfsug-discuss] Am I a gateway or not?
In-Reply-To: <OF9C6D99B0.DF71D62B-ON002581F5.0043B3C0@LocalDomain>
References: <OF9C6D99B0.DF71D62B-ON002581F5.0043B3C0@LocalDomain>
Message-ID: <OF1EF7D0AE.0DD977EE-ON852581F5.00510567-852581F5.00511AAB@notes.na.collabserv.com>

Hi Abhishek,

Can you please help answer this query.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   John Hearns <john.hearns at asml.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   12/13/2017 07:19 AM
Subject:        [gpfsug-discuss] Am I a gateway or not?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Firstly, I hope everyone at the Manchester meeting is enjoying it and that 
the snow did not hamper too many travel plans.
I like Manchester ? I shall tell the tale of blowing the 100Amp fuse in 
the University server room some other time.
(Yes ? a fuse. Not a circuit breaker).
 
I am writing some monitoring scripts for AFM ? actually to interface with 
Bright.
I would like the script to detect if the node which is running it is a 
Gateway node or has the AFM component.
If not, the script silently exits.
This is not a Callback script ? I can write these and indeed have one for 
the AFM fileset demount event.
 
In order to determine which components are on a given node I could:
a.      Run ?mmhealth node show? and grep for AFM
b.      Run mmlscluster and grep for $HOSTNAME
 
Both methods seem slightly inelegant. IS there a handy-dandy command I can 
run to see the components on this node?
 
John Hearns
-- The information contained in this communication and any attachments is 
confidential and may be privileged, and is for the sole use of the 
intended recipient(s). Any unauthorized review, use, disclosure or 
distribution is prohibited. Unless explicitly stated otherwise in the body 
of this communication or the attachment thereto (if any), the information 
is provided on an AS-IS basis without any express or implied warranties or 
liabilities. To the extent you are relying on this information, you are 
doing so at your own risk. If you are not the intended recipient, please 
notify the sender immediately by replying to this message and destroy all 
copies of this message and any attachments. Neither the sender nor the 
company/group of companies he or she represents shall be liable for the 
proper and complete transmission of the information contained in this 
communication, or for any delay in its receipt. 
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=iZJyYX6s6-fBYfRNYStIrfVhph3NLzwXJNqTZEfKOyA&s=HoPYEIPpBLIFV-JCq69MS026ZLBRxqqvqNZYGH8UmlY&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/92f3f864/attachment-0002.htm>

From abhisdav at in.ibm.com  Wed Dec 13 16:58:39 2017
From: abhisdav at in.ibm.com (Abhishek Dave)
Date: Wed, 13 Dec 2017 22:28:39 +0530
Subject: [gpfsug-discuss] Am I a gateway or not?
In-Reply-To: <OF1EF7D0AE.0DD977EE-ON852581F5.00510567-852581F5.00511998@LocalDomain>
References: <OF9C6D99B0.DF71D62B-ON002581F5.0043B3C0@LocalDomain>
	<OF1EF7D0AE.0DD977EE-ON852581F5.00510567-852581F5.00511998@LocalDomain>
Message-ID: <OF90FA1155.FA40F211-ON652581F5.005C72A9-652581F5.005D42EF@notes.na.collabserv.com>

Hi John,

I would recommend running 'mmlscluster' command on any cluster node to
check if any AFM Gateway Node(s) is available.

This will list hostname(s) which are running as AFM Gateway.

/usr/lpp/mmfs/bin/mmlscluster | grep -w gateway | awk '{print $4}'

e.g.

~bash# /usr/lpp/mmfs/bin/mmlscluster | grep -w gateway | awk '{print $4}'
node1
node2

Thanks,
Abhishek, Dave




From:	IBM Spectrum Scale/Poughkeepsie/IBM
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc:	Abhishek Dave/India/IBM at IBMIN, IBM Spectrum
            Scale/Poughkeepsie/IBM at IBMUS
Date:	12/13/2017 08:16 PM
Subject:	Re: [gpfsug-discuss] Am I a gateway or not?
Sent by:	Haizhu Liu


Hi Abhishek,

Can you please help answer this query.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------

If you feel that your question can benefit other users of  Spectrum Scale
(GPFS), then please post it to the public IBM developerWroks Forum at
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
.

If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact
1-800-237-5511 in the United States or your local IBM Service Center in
other countries.

The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.




From:	John Hearns <john.hearns at asml.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	12/13/2017 07:19 AM
Subject:	[gpfsug-discuss] Am I a gateway or not?
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Firstly, I hope everyone at the Manchester meeting is enjoying it and that
the snow did not hamper too many travel plans.
I like Manchester ? I shall tell the tale of blowing the 100Amp fuse in the
University server room some other time.
(Yes ? a fuse. Not a circuit breaker).

I am writing some monitoring scripts for AFM ? actually to interface with
Bright.
I would like the script to detect if the node which is running it is a
Gateway node or has the AFM component.
If not, the script silently exits.
This is not a Callback script ? I can write these and indeed have one for
the AFM fileset demount event.

In order to determine which components are on a given node I could:
   a.	Run ?mmhealth node show? and grep for AFM
   b.	Run mmlscluster and grep for $HOSTNAME

Both methods seem slightly inelegant. IS there a handy-dandy command I can
run to see the components on this node?

John Hearns


-- The information contained in this communication and any attachments is
confidential and may be privileged, and is for the sole use of the intended
recipient(s). Any unauthorized review, use, disclosure or distribution is
prohibited. Unless explicitly stated otherwise in the body of this
communication or the attachment thereto (if any), the information is
provided on an AS-IS basis without any express or implied warranties or
liabilities. To the extent you are relying on this information, you are
doing so at your own risk. If you are not the intended recipient, please
notify the sender immediately by replying to this message and destroy all
copies of this message and any attachments. Neither the sender nor the
company/group of companies he or she represents shall be liable for the
proper and complete transmission of the information contained in this
communication, or for any delay in its receipt.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=iZJyYX6s6-fBYfRNYStIrfVhph3NLzwXJNqTZEfKOyA&s=HoPYEIPpBLIFV-JCq69MS026ZLBRxqqvqNZYGH8UmlY&e=










-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/3b793bb7/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171213/3b793bb7/attachment-0002.gif>

From chair at spectrumscale.org  Thu Dec 14 11:16:05 2017
From: chair at spectrumscale.org (Simon Thompson)
Date: Thu, 14 Dec 2017 11:16:05 +0000
Subject: [gpfsug-discuss] SSUG17 wraps
Message-ID: <F96F7F48-5592-45D9-9A1D-16B9D8F8D507@spectrumscale.org>

Well, our last event of 2017 took place this year at CIUK in Manchester on 12th December. Claire pointed out that it was exactly 7 years since the user group was first formed and we?ve grown significantly since then. There?s now hundreds of subscribers to the mailing list and regular events taking place across the globe.

 

For those interested in the presentations this year, all the slide decks we have or that have been approved are published on the user group website.

 

Thanks to all the speakers from both IBM and the user community for speaking at our events!

 

And finally, we?ve updated the look of our website and changed to something we hope will be a bit easier for use to manage. We?ve made some changes to the navigation to make things easier to find, so please do check out the website and let myself of Claire know directly if you have any comments on this.

 

We?re also interested in hearing from you if you?d like to write a feature or article for the user group website, perhaps comment on a scale feature you use, or something you?ve struggled with and fixed. Please get in touch if you?d like to do this.

 

We?re already planning for 2018 events and hope we?ll have a successful year next year.

 

Simon Thompson

UK Group Chair

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171214/53dbbc98/attachment-0002.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Thu Dec 14 19:38:52 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Thu, 14 Dec 2017 19:38:52 +0000
Subject: [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5 upgrade
Message-ID: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>

Hi All,

         26 mmbackupDors-20171023.log
         26 mmbackupDors-20171024.log
         26 mmbackupDors-20171025.log
         26 mmbackupDors-20171026.log
    2922752 mmbackupDors-20171027.log
    1375555 mmbackupDors-20171028.log
      59328 mmbackupDors-20171029.log
    2748095 mmbackupDors-20171030.log
     124953 mmbackupDors-20171031.log

That?s ?wc -l? output ? and the difference in size occurred with the GPFS 4.2.3.5 upgrade.  I?m not technically ?responsible? for mmbackup here, so I?m not at all familiar with it.  However, we?ve asked a certain vendor (not IBM) about it and they don?t know either, so I don?t feel too awfully bad.

And we have looked at the man page and didn?t see any obvious options to decrease the verbosity.  We did not make any changes to the backup script itself, so the mmbackup invocation is the same.  Any ideas?  Thanks?

Kevin
?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171214/23f59fce/attachment-0002.htm>

From scale at us.ibm.com  Fri Dec 15 14:32:25 2017
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Fri, 15 Dec 2017 09:32:25 -0500
Subject: [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5
	upgrade
In-Reply-To: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>
References: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>
Message-ID: <OFEF3AE524.5EDB2C57-ON852581F7.004FCC00-852581F7.004FDF6C@notes.na.collabserv.com>

Tru,

Can you please help with this query or forward to the right person. 
Thanks. 

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   12/14/2017 02:39 PM
Subject:        [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5 
upgrade
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi All, 

         26 mmbackupDors-20171023.log
         26 mmbackupDors-20171024.log
         26 mmbackupDors-20171025.log
         26 mmbackupDors-20171026.log
    2922752 mmbackupDors-20171027.log
    1375555 mmbackupDors-20171028.log
      59328 mmbackupDors-20171029.log
    2748095 mmbackupDors-20171030.log
     124953 mmbackupDors-20171031.log

That?s ?wc -l? output ? and the difference in size occurred with the GPFS 
4.2.3.5 upgrade.  I?m not technically ?responsible? for mmbackup here, so 
I?m not at all familiar with it.  However, we?ve asked a certain vendor 
(not IBM) about it and they don?t know either, so I don?t feel too awfully 
bad.

And we have looked at the man page and didn?t see any obvious options to 
decrease the verbosity.  We did not make any changes to the backup script 
itself, so the mmbackup invocation is the same.  Any ideas?  Thanks?

Kevin
?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and 
Education
Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=ZFlCTb2hD9WKda9YJ94wM4Kb-ZTkTg6yuH2H0DabbRs&s=4TRCVgkMd1Ly9WvaBPVWr4ciur_ZYZswzqrlFHl_6EI&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/9212a7f7/attachment-0002.htm>

From makaplan at us.ibm.com  Fri Dec 15 15:14:40 2017
From: makaplan at us.ibm.com (Marc A Kaplan)
Date: Fri, 15 Dec 2017 10:14:40 -0500
Subject: [gpfsug-discuss] mmbackup log file size after GPFS
	4.2.3.5upgrade
In-Reply-To: <OFEF3AE524.5EDB2C57-ON852581F7.004FCC00-852581F7.004FDF6C@notes.na.collabserv.com>
References: <634164E5-4915-4DAA-B399-7620A4DC33C1@vanderbilt.edu>
	<OFEF3AE524.5EDB2C57-ON852581F7.004FCC00-852581F7.004FDF6C@notes.na.collabserv.com>
Message-ID: <OF94C2F295.6B1FFCC4-ON852581F7.005351E8-852581F7.0053BE2A@notes.na.collabserv.com>

It's not clear that this is a problem or malfunction.
Customer should contact IBM support and be ready to transmit copies of the 
cited log files and other mmbackup command output (stdout and stderr 
messages) for analysis.
Also mmsnap output.



From:   "IBM Spectrum Scale" <scale at us.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc:     Truong Vu <truongv at us.ibm.com>
Date:   12/15/2017 09:32 AM
Subject:        Re: [gpfsug-discuss] mmbackup log file size after GPFS 
4.2.3.5 upgrade
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Tru,

Can you please help with this query or forward to the right person. 
Thanks.  

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        12/14/2017 02:39 PM
Subject:        [gpfsug-discuss] mmbackup log file size after GPFS 4.2.3.5 
upgrade
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi All, 

         26 mmbackupDors-20171023.log
         26 mmbackupDors-20171024.log
         26 mmbackupDors-20171025.log
         26 mmbackupDors-20171026.log
    2922752 mmbackupDors-20171027.log
    1375555 mmbackupDors-20171028.log
      59328 mmbackupDors-20171029.log
    2748095 mmbackupDors-20171030.log
     124953 mmbackupDors-20171031.log

That?s ?wc -l? output ? and the difference in size occurred with the GPFS 
4.2.3.5 upgrade.  I?m not technically ?responsible? for mmbackup here, so 
I?m not at all familiar with it.  However, we?ve asked a certain vendor 
(not IBM) about it and they don?t know either, so I don?t feel too awfully 
bad.

And we have looked at the man page and didn?t see any obvious options to 
decrease the verbosity.  We did not make any changes to the backup script 
itself, so the mmbackup invocation is the same.  Any ideas?  Thanks?

Kevin
?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and 
Education
Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=ZFlCTb2hD9WKda9YJ94wM4Kb-ZTkTg6yuH2H0DabbRs&s=4TRCVgkMd1Ly9WvaBPVWr4ciur_ZYZswzqrlFHl_6EI&e=



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=n3w55C-SUpJhWZEaM249Jx7hx44Q8KsZi0-S2WaisJc&s=SFkl9xPj2c7TR7paHtU3Nby1vonHbBBt1Vmv7QdLH9Q&e=





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/f4679d08/attachment-0002.htm>

From Robert.Oesterlin at nuance.com  Fri Dec 15 18:40:28 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Fri, 15 Dec 2017 18:40:28 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0.0 Available on Passport
	Advantage
Message-ID: <985A9D83-4B5E-4234-9884-FFA5AE295063@nuance.com>

Scale 5.0 was released today and is available for download. Time to construct a test cluster!


Bob Oesterlin
Sr Principal Storage Engineer, Nuance
507-269-0413

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/b7a70d41/attachment-0002.htm>

From aaron.s.knister at nasa.gov  Fri Dec 15 19:49:59 2017
From: aaron.s.knister at nasa.gov (Aaron Knister)
Date: Fri, 15 Dec 2017 14:49:59 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
Message-ID: <fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>

Thanks, Bill.

I still don't feel like I've got an clear answer from IBM and frankly
the core issue of a lack of migration tool was totally dodged.

Again in Sven's presentation from SSUG @ SC17
(http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf)
he mentions "It has a significant performance penalty for small files in
large block size filesystems" and the demonstrates that with several
mdtest runs (which show the effect with and without the >32 subblocks code):


4.2.1 base code - SUMMARY: (of 3 iterations)
File creation : Mean = 2237.644

zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY: (of
3 iterations)
File creation : Mean = 12866.842

more sub blocks per block (4.2.2 + morethan32subblock code):
File creation : Mean = 40316.721

Can someone (ideally Sven) give me a straight answer as to whether or
not the > 32 subblock code actually makes a performance difference for
small files in large block filesystems? And if not, help me understand
why his slides and provided benchmark data have consistently indicated
it does?

-Aaron

On 12/1/17 11:44 AM, Bill Hartner wrote:
> ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> results @ user group meeting.
> 
> -Bill
> 
> Bill Hartner
> IBM Systems
> Scalable I/O Development
> Austin, Texas
> bhartner at us.ibm.com
> home office 512-784-0980
> 
> 
> Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
> AM---Bill, could you say something about what the metadataJan-Frode
> Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> about what the metadata-storage here was? ESS/NL-SAS/3way replication?
> 
> From: Jan-Frode Myklebust <janfrode at tanso.net>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 12/01/2017 06:53 AM
> Subject: Re: [gpfsug-discuss] Online data migration tool
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> 
> ------------------------------------------------------------------------
> 
> 
> 
> Bill, could you say something about what the metadata-storage here was?
> ESS/NL-SAS/3way replication?
> 
> I just asked about this in the internal slack channel #scale-help today..
> 
> 
> 
> -jf
> 
> fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
> <mailto:bhartner at us.ibm.com>>:
> 
>     > "It has a significant performance penalty for small files in large
>     > block size filesystems"
> 
>     Aaron,
> 
>     Below are mdtest results for a test we ran for CORAL - file size was
>     32k.
> 
>     We have not gone back and ran the test on a file system formatted
>     without > 32 subblocks. We'll do that at some point...
> 
>     -Bill
> 
>     -- started at 10/28/2017 17:51:38 --
> 
>     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360
>     -u -y
>     Path: /ibm/fs2-16m-10
>     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> 
>     228 tasks, 67239936 files
> 
>     SUMMARY: (of 3 iterations)
>     Operation Max Min Mean Std Dev
>     --------- --- --- ---- -------
>     File creation : 51953.498 50558.517 51423.221 616.643
>     File stat : 0.000 0.000 0.000 0.000
>     File read : 0.000 0.000 0.000 0.000
>     File removal : 96746.376 92149.535 94658.774 1900.187
>     Tree creation : 1.588 0.070 0.599 0.700
>     Tree removal : 0.213 0.034 0.097 0.082
> 
>     -- finished at 10/28/2017 19:51:54 --
> 
>     Bill Hartner
>     IBM Systems
>     Scalable I/O Development
>     Austin, Texas_
>     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com>
>     home office 512-784-0980
> 
>     _
>     __gpfsug-discuss-bounces at spectrumscale.org_
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>?wrote on
>     11/29/2017 04:41:48 PM:
> 
>     > From: Aaron Knister <_aaron.knister at gmail.com_
>     <mailto:aaron.knister at gmail.com>>
> 
> 
>     > To: gpfsug main discussion list
>     <_gpfsug-discuss at spectrumscale.org_
>     <mailto:gpfsug-discuss at spectrumscale.org>>
> 
>     > Date: 11/29/2017 04:42 PM
> 
> 
>     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> 
>     >
> 
>     > Thanks, Nikhil. Most of that was consistent with my understnading,
>     > however I was under the impression that the >32 subblocks code is
>     > required to achieve the touted 50k file creates/second that Sven has
>     > talked about a bunch of times:
>     >
>     >
>     _http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>
>     >
>     _http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_ <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>
>     > _Sven_Oehme_-_News_from_Research.pdf
>     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>
>     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> 
> 
>     > from those presentations regarding 32 subblocks:
>     >
>     > "It has a significant performance penalty for small files in large
>     > block size filesystems"
> 
>     > although I'm not clear on the specific definition of "large". Many
>     > filesystems I encounter only have a 1M block size so it may not
>     > matter there, although that same presentation clearly shows the
>     > benefit of larger block sizes which is yet *another* thing for which
>     > a migration tool would be helpful.
> 
>     > -Aaron
>     >
>     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com>> wrote:
> 
>     > Hi,
>     >
>     > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.
>     > For all Spectrum Scale clusters that are currently at 4.X.X, it is
>     > possible to migrate to 5.0.0 with no offline data migration and no
>     > need to move data. Once these clusters are at 5.0.0, they will
>     > benefit from the performance improvements, new features (such as
>     > file audit logging), and various enhancements that are included in
>     5.0.0.
>     >
>     > That being said, there is one enhancement that will not be applied
>     > to these clusters, and that is the increased number of sub-blocks
>     > per block for small file allocation. This means that for file
>     > systems with a large block size and a lot of small files, the
>     > overall space utilization will be the same it currently is in 4.X.X.
>     > Since file systems created at 4.X.X and earlier used a block size
>     > that kept this allocation in mind, there should be very little
>     > impact on existing file systems.
>     >
>     > Outside of that one particular function, the remainder of the
>     > performance improvements, metadata improvements, updated
>     > compatibility, new functionality, and all of the other enhancements
>     > will be immediately available to you once you complete the upgrade
>     > to 5.0.0 -- with no need to reformat, move data, or take your data
>     offline.
>     >
>     > I hope that clarifies things a little and makes the upgrade path
>     > more accessible.
>     >
>     > Please let me know if there are any other questions or concerns.
>     >
>     > Thank you,
>     > Nikhil Khandelwal
>     > Spectrum Scale Development
>     > Client Adoption
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at _spectrumscale.org_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
>     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
> 
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at _spectrumscale.org_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
> 
>     > _https://urldefense.proofpoint.com/v2/url?_
>     >
>     u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>     >
>     siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> 
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at _spectrumscale.org_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>_
>     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>_______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org
>     https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> 
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776


From alex at calicolabs.com  Fri Dec 15 20:48:16 2017
From: alex at calicolabs.com (Alex Chekholko)
Date: Fri, 15 Dec 2017 12:48:16 -0800
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
Message-ID: <CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>

Hey Aaron,

Can you define your sizes for "large blocks" and "small files"?  If you
dial one up and the other down, your performance will be worse.  And in any
case it's a pathological corner case so it shouldn't matter much for your
workflow, unless you've designed your system with the wrong values.

For example, for bioinformatics workloads, I prefer to use 256KB filesystem
block size, and I'd consider 4MB+ to be "large block size", which would
make the filesystem obviously unsuitable for processing millions of 8KB
files.

You can make a histogram of file sizes in your existing filesystems and
then make your subblock size (1/32 of block size) on the smaller end of
that.   Also definitely use the "small file in inode" feature and put your
metadata on SSD.

Regards,
Alex

On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister <aaron.s.knister at nasa.gov>
wrote:

> Thanks, Bill.
>
> I still don't feel like I've got an clear answer from IBM and frankly
> the core issue of a lack of migration tool was totally dodged.
>
> Again in Sven's presentation from SSUG @ SC17
> (http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf)
> he mentions "It has a significant performance penalty for small files in
> large block size filesystems" and the demonstrates that with several
> mdtest runs (which show the effect with and without the >32 subblocks
> code):
>
>
> 4.2.1 base code - SUMMARY: (of 3 iterations)
> File creation : Mean = 2237.644
>
> zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY: (of
> 3 iterations)
> File creation : Mean = 12866.842
>
> more sub blocks per block (4.2.2 + morethan32subblock code):
> File creation : Mean = 40316.721
>
> Can someone (ideally Sven) give me a straight answer as to whether or
> not the > 32 subblock code actually makes a performance difference for
> small files in large block filesystems? And if not, help me understand
> why his slides and provided benchmark data have consistently indicated
> it does?
>
> -Aaron
>
> On 12/1/17 11:44 AM, Bill Hartner wrote:
> > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> > results @ user group meeting.
> >
> > -Bill
> >
> > Bill Hartner
> > IBM Systems
> > Scalable I/O Development
> > Austin, Texas
> > bhartner at us.ibm.com
> > home office 512-784-0980
> >
> >
> > Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
> > AM---Bill, could you say something about what the metadataJan-Frode
> > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> > about what the metadata-storage here was? ESS/NL-SAS/3way replication?
> >
> > From: Jan-Frode Myklebust <janfrode at tanso.net>
> > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> > Date: 12/01/2017 06:53 AM
> > Subject: Re: [gpfsug-discuss] Online data migration tool
> > Sent by: gpfsug-discuss-bounces at spectrumscale.org
> >
> > ------------------------------------------------------------------------
> >
> >
> >
> > Bill, could you say something about what the metadata-storage here was?
> > ESS/NL-SAS/3way replication?
> >
> > I just asked about this in the internal slack channel #scale-help today..
> >
> >
> >
> > -jf
> >
> > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
> > <mailto:bhartner at us.ibm.com>>:
> >
> >     > "It has a significant performance penalty for small files in large
> >     > block size filesystems"
> >
> >     Aaron,
> >
> >     Below are mdtest results for a test we ran for CORAL - file size was
> >     32k.
> >
> >     We have not gone back and ran the test on a file system formatted
> >     without > 32 subblocks. We'll do that at some point...
> >
> >     -Bill
> >
> >     -- started at 10/28/2017 17:51:38 --
> >
> >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
> >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360
> >     -u -y
> >     Path: /ibm/fs2-16m-10
> >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> >
> >     228 tasks, 67239936 files
> >
> >     SUMMARY: (of 3 iterations)
> >     Operation Max Min Mean Std Dev
> >     --------- --- --- ---- -------
> >     File creation : 51953.498 50558.517 51423.221 616.643
> >     File stat : 0.000 0.000 0.000 0.000
> >     File read : 0.000 0.000 0.000 0.000
> >     File removal : 96746.376 92149.535 94658.774 1900.187
> >     Tree creation : 1.588 0.070 0.599 0.700
> >     Tree removal : 0.213 0.034 0.097 0.082
> >
> >     -- finished at 10/28/2017 19:51:54 --
> >
> >     Bill Hartner
> >     IBM Systems
> >     Scalable I/O Development
> >     Austin, Texas_
> >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com>
> >     home office 512-784-0980
> >
> >     _
> >     __gpfsug-discuss-bounces at spectrumscale.org_
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org> wrote on
> >     11/29/2017 04:41:48 PM:
> >
> >     > From: Aaron Knister <_aaron.knister at gmail.com_
> >     <mailto:aaron.knister at gmail.com>>
> >
> >
> >     > To: gpfsug main discussion list
> >     <_gpfsug-discuss at spectrumscale.org_
> >     <mailto:gpfsug-discuss at spectrumscale.org>>
> >
> >     > Date: 11/29/2017 04:42 PM
> >
> >
> >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> >
> >     >
> >
> >     > Thanks, Nikhil. Most of that was consistent with my understnading,
> >     > however I was under the impression that the >32 subblocks code is
> >     > required to achieve the touted 50k file creates/second that Sven
> has
> >     > talked about a bunch of times:
> >     >
> >     >
> >     _http://files.gpfsug.org/presentations/2017/Manchester/
> 08_Research_Topics.pdf_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__files.gpfsug.org_presentations_2017_Manchester_
> 08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>
> >     >
> >     _http://files.gpfsug.org/presentations/2017/Ehningen/
> 31_-_SSUG17DE_-_ <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__files.gpfsug.org_presentations_2017_Ehningen_
> 31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>
> >     > _Sven_Oehme_-_News_from_Research.pdf
> >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__files.gpfsug.org_presentations_2016_SC16_12-5F-
> 2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW
> UGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--
> uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>
> >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> >
> >
> >     > from those presentations regarding 32 subblocks:
> >     >
> >     > "It has a significant performance penalty for small files in large
> >     > block size filesystems"
> >
> >     > although I'm not clear on the specific definition of "large". Many
> >     > filesystems I encounter only have a 1M block size so it may not
> >     > matter there, although that same presentation clearly shows the
> >     > benefit of larger block sizes which is yet *another* thing for
> which
> >     > a migration tool would be helpful.
> >
> >     > -Aaron
> >     >
> >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
> >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com>> wrote:
> >
> >     > Hi,
> >     >
> >     > I would like to clarify migration path to 5.0.0 from 4.X.X
> clusters.
> >     > For all Spectrum Scale clusters that are currently at 4.X.X, it is
> >     > possible to migrate to 5.0.0 with no offline data migration and no
> >     > need to move data. Once these clusters are at 5.0.0, they will
> >     > benefit from the performance improvements, new features (such as
> >     > file audit logging), and various enhancements that are included in
> >     5.0.0.
> >     >
> >     > That being said, there is one enhancement that will not be applied
> >     > to these clusters, and that is the increased number of sub-blocks
> >     > per block for small file allocation. This means that for file
> >     > systems with a large block size and a lot of small files, the
> >     > overall space utilization will be the same it currently is in
> 4.X.X.
> >     > Since file systems created at 4.X.X and earlier used a block size
> >     > that kept this allocation in mind, there should be very little
> >     > impact on existing file systems.
> >     >
> >     > Outside of that one particular function, the remainder of the
> >     > performance improvements, metadata improvements, updated
> >     > compatibility, new functionality, and all of the other enhancements
> >     > will be immediately available to you once you complete the upgrade
> >     > to 5.0.0 -- with no need to reformat, move data, or take your data
> >     offline.
> >     >
> >     > I hope that clarifies things a little and makes the upgrade path
> >     > more accessible.
> >     >
> >     > Please let me know if there are any other questions or concerns.
> >     >
> >     > Thank you,
> >     > Nikhil Khandelwal
> >     > Spectrum Scale Development
> >     > Client Adoption
> >     >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at _spectrumscale.org_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
> >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=
> DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW
> UGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_
> TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
> >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at _spectrumscale.org_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>
> >
> >     > _https://urldefense.proofpoint.com/v2/url?_
> >     >
> >     u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=
> DwICAg&c=jf_iaSHvJObTbx-
> >     >
> >     siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=
> DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_
> W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> >
> >     _______________________________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at _spectrumscale.org_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >_
> >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=
> DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW
> UGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_
> TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>___________________
> ____________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at spectrumscale.org
> >     https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=
> Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_
> jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/5f33cc4a/attachment-0002.htm>

From aaron.s.knister at nasa.gov  Fri Dec 15 22:47:43 2017
From: aaron.s.knister at nasa.gov (Aaron Knister)
Date: Fri, 15 Dec 2017 17:47:43 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
Message-ID: <03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>

Thanks, Alex. I'm all too familiar with the trade offs between large
blocks and small files and we do use pretty robust SSD storage for our
metadata. We support a wide range of workloads and we have some folks
with many small (<1M) files and other folks with many large (>256MB) files.

My point in this thread is that IBM has said over and over again in
presentations that there is a significant performance gain with the >32
subblocks code on filesystems with large block sizes (although to your
point I'm not clear on exactly what large means since I didn't define
large in this context). Therefore given that the >32 subblock code gives
a significant performance gain one could reasonably assume that having a
filesystem with >32 subblocks is required to see this gain (rather than
just running the >32 subblocks code on an fs w/o > 32 subblocks).

This lead me to ask about a migration tool because in my mind if there's
a performance gain from having >32 subblocks on the FS I'd like that
feature and having to manually copy 10's of PB to new hardware to get
this performance boost is unacceptable. However, IBM can't seem to make
up their mind about whether or not the >32 subblocks code *actually*
provides a performance increase. This seems like a pretty
straightforward question.

-Aaron

On 12/15/17 3:48 PM, Alex Chekholko wrote:
> Hey Aaron,
> 
> Can you define your sizes for "large blocks" and "small files"?? If you
> dial one up and the other down, your performance will be worse.? And in
> any case it's a pathological corner case so it shouldn't matter much for
> your workflow, unless you've designed your system with the wrong values.
> 
> For example, for bioinformatics workloads, I prefer to use 256KB
> filesystem block size, and I'd consider 4MB+ to be "large block size",
> which would make the filesystem obviously unsuitable for processing
> millions of 8KB files.
> 
> You can make a histogram of file sizes in your existing filesystems and
> then make your subblock size (1/32 of block size) on the smaller end of
> that.? ?Also definitely use the "small file in inode" feature and put
> your metadata on SSD.
> 
> Regards,
> Alex
> 
> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
> 
>     Thanks, Bill.
> 
>     I still don't feel like I've got an clear answer from IBM and frankly
>     the core issue of a lack of migration tool was totally dodged.
> 
>     Again in Sven's presentation from SSUG @ SC17
>     (http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
>     he mentions "It has a significant performance penalty for small files in
>     large block size filesystems" and the demonstrates that with several
>     mdtest runs (which show the effect with and without the >32
>     subblocks code):
> 
> 
>     4.2.1 base code - SUMMARY: (of 3 iterations)
>     File creation : Mean = 2237.644
> 
>     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):? SUMMARY: (of
>     3 iterations)
>     File creation : Mean = 12866.842
> 
>     more sub blocks per block (4.2.2 + morethan32subblock code):
>     File creation : Mean = 40316.721
> 
>     Can someone (ideally Sven) give me a straight answer as to whether or
>     not the > 32 subblock code actually makes a performance difference for
>     small files in large block filesystems? And if not, help me understand
>     why his slides and provided benchmark data have consistently indicated
>     it does?
> 
>     -Aaron
> 
>     On 12/1/17 11:44 AM, Bill Hartner wrote:
>     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
>     > results @ user group meeting.
>     >
>     > -Bill
>     >
>     > Bill Hartner
>     > IBM Systems
>     > Scalable I/O Development
>     > Austin, Texas
>     > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
>     > home office 512-784-0980 <tel:512-784-0980>
>     >
>     >
>     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
>     > AM---Bill, could you say something about what the metadataJan-Frode
>     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
>     > about what the metadata-storage here was? ESS/NL-SAS/3way replication?
>     >
>     > From: Jan-Frode Myklebust <janfrode at tanso.net <mailto:janfrode at tanso.net>>
>     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org
>     <mailto:gpfsug-discuss at spectrumscale.org>>
>     > Date: 12/01/2017 06:53 AM
>     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     >
>     >
>     ------------------------------------------------------------------------
>     >
>     >
>     >
>     > Bill, could you say something about what the metadata-storage here was?
>     > ESS/NL-SAS/3way replication?
>     >
>     > I just asked about this in the internal slack channel #scale-help today..
>     >
>     >
>     >
>     > -jf
>     >
>     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
>     > <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>:
>     >
>     >? ? ?> "It has a significant performance penalty for small files in
>     large
>     >? ? ?> block size filesystems"
>     >
>     >? ? ?Aaron,
>     >
>     >? ? ?Below are mdtest results for a test we ran for CORAL - file
>     size was
>     >? ? ?32k.
>     >
>     >? ? ?We have not gone back and ran the test on a file system formatted
>     >? ? ?without > 32 subblocks. We'll do that at some point...
>     >
>     >? ? ?-Bill
>     >
>     >? ? ?-- started at 10/28/2017 17:51:38 --
>     >
>     >? ? ?mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>     >? ? ?Command line used: /tmp/mdtest-binary-dir/mdtest -d
>     >? ? ?/ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
>     -p 360
>     >? ? ?-u -y
>     >? ? ?Path: /ibm/fs2-16m-10
>     >? ? ?FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>     >
>     >? ? ?228 tasks, 67239936 files
>     >
>     >? ? ?SUMMARY: (of 3 iterations)
>     >? ? ?Operation Max Min Mean Std Dev
>     >? ? ?--------- --- --- ---- -------
>     >? ? ?File creation : 51953.498 50558.517 51423.221 616.643
>     >? ? ?File stat : 0.000 0.000 0.000 0.000
>     >? ? ?File read : 0.000 0.000 0.000 0.000
>     >? ? ?File removal : 96746.376 92149.535 94658.774 1900.187
>     >? ? ?Tree creation : 1.588 0.070 0.599 0.700
>     >? ? ?Tree removal : 0.213 0.034 0.097 0.082
>     >
>     >? ? ?-- finished at 10/28/2017 19:51:54 --
>     >
>     >? ? ?Bill Hartner
>     >? ? ?IBM Systems
>     >? ? ?Scalable I/O Development
>     >? ? ?Austin, Texas_
>     >? ? ?__bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
>     <mailto:bhartner at us.ibm.com>>
>     >? ? ?home office 512-784-0980 <tel:512-784-0980>
>     >
>     >? ? ?_
>     >? ? ?__gpfsug-discuss-bounces at spectrumscale.org_
>     >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>?wrote on
>     >? ? ?11/29/2017 04:41:48 PM:
>     >
>     >? ? ?> From: Aaron Knister <_aaron.knister at gmail.com_
>     >? ? ?<mailto:aaron.knister at gmail.com <mailto:aaron.knister at gmail.com>>>
>     >
>     >
>     >? ? ?> To: gpfsug main discussion list
>     >? ? ?<_gpfsug-discuss at spectrumscale.org_
>     >? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>     <mailto:gpfsug-discuss at spectrumscale.org>>>
>     >
>     >? ? ?> Date: 11/29/2017 04:42 PM
>     >
>     >
>     >? ? ?> Subject: Re: [gpfsug-discuss] Online data migration tool
>     >? ? ?> Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>     >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>     >
>     >? ? ?>
>     >
>     >? ? ?> Thanks, Nikhil. Most of that was consistent with my understnading,
>     >? ? ?> however I was under the impression that the >32 subblocks code is
>     >? ? ?> required to achieve the touted 50k file creates/second that Sven has
>     >? ? ?> talked about a bunch of times:
>     >? ? ?>
>     >? ? ?>
>     >? ?
>     ?_http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>     <http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>     >? ? ?>
>     >? ?
>     ?_http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
>     <http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>     >? ? ?> _Sven_Oehme_-_News_from_Research.pdf
>     >? ? ?> _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>     >? ? ?> _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>     >
>     >
>     >? ? ?> from those presentations regarding 32 subblocks:
>     >? ? ?>
>     >? ? ?> "It has a significant performance penalty for small files in large
>     >? ? ?> block size filesystems"
>     >
>     >? ? ?> although I'm not clear on the specific definition of "large". Many
>     >? ? ?> filesystems I encounter only have a 1M block size so it may not
>     >? ? ?> matter there, although that same presentation clearly shows the
>     >? ? ?> benefit of larger block sizes which is yet *another* thing for which
>     >? ? ?> a migration tool would be helpful.
>     >
>     >? ? ?> -Aaron
>     >? ? ?>
>     >? ? ?> On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>     >? ? ?<_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
>     <mailto:nikhilk at us.ibm.com>>> wrote:
>     >
>     >? ? ?> Hi,
>     >? ? ?>
>     >? ? ?> I would like to clarify migration path to 5.0.0 from 4.X.X
>     clusters.
>     >? ? ?> For all Spectrum Scale clusters that are currently at 4.X.X,
>     it is
>     >? ? ?> possible to migrate to 5.0.0 with no offline data migration
>     and no
>     >? ? ?> need to move data. Once these clusters are at 5.0.0, they will
>     >? ? ?> benefit from the performance improvements, new features (such as
>     >? ? ?> file audit logging), and various enhancements that are
>     included in
>     >? ? ?5.0.0.
>     >? ? ?>
>     >? ? ?> That being said, there is one enhancement that will not be
>     applied
>     >? ? ?> to these clusters, and that is the increased number of
>     sub-blocks
>     >? ? ?> per block for small file allocation. This means that for file
>     >? ? ?> systems with a large block size and a lot of small files, the
>     >? ? ?> overall space utilization will be the same it currently is
>     in 4.X.X.
>     >? ? ?> Since file systems created at 4.X.X and earlier used a block
>     size
>     >? ? ?> that kept this allocation in mind, there should be very little
>     >? ? ?> impact on existing file systems.
>     >? ? ?>
>     >? ? ?> Outside of that one particular function, the remainder of the
>     >? ? ?> performance improvements, metadata improvements, updated
>     >? ? ?> compatibility, new functionality, and all of the other
>     enhancements
>     >? ? ?> will be immediately available to you once you complete the
>     upgrade
>     >? ? ?> to 5.0.0 -- with no need to reformat, move data, or take
>     your data
>     >? ? ?offline.
>     >? ? ?>
>     >? ? ?> I hope that clarifies things a little and makes the upgrade path
>     >? ? ?> more accessible.
>     >? ? ?>
>     >? ? ?> Please let me know if there are any other questions or concerns.
>     >? ? ?>
>     >? ? ?> Thank you,
>     >? ? ?> Nikhil Khandelwal
>     >? ? ?> Spectrum Scale Development
>     >? ? ?> Client Adoption
>     >? ? ?>
>     >? ? ?> _______________________________________________
>     >? ? ?> gpfsug-discuss mailing list
>     >? ? ?> gpfsug-discuss at _spectrumscale.org_
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >? ? ?> _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>     >
>     >? ? ?> _______________________________________________
>     >? ? ?> gpfsug-discuss mailing list
>     >? ? ?> gpfsug-discuss at _spectrumscale.org_
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >
>     >? ? ?> _https://urldefense.proofpoint.com/v2/url?_
>     <https://urldefense.proofpoint.com/v2/url?_>
>     >? ? ?>
>     >? ? ?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>     >? ? ?>
>     >? ? ?siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>     >
>     >? ? ?_______________________________________________
>     >? ? ?gpfsug-discuss mailing list
>     >? ? ?gpfsug-discuss at _spectrumscale.org_
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>     >? ? ?__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >? ?
>     ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>     >? ? ?gpfsug-discuss mailing list
>     >? ? ?gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     >? ?
>     ?https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>     >
> 
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776 <tel:%28301%29%20286-2776>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> 
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776


From oehmes at gmail.com  Fri Dec 15 23:06:52 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Fri, 15 Dec 2017 23:06:52 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
Message-ID: <CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>

i thought i answered that already, but maybe i just thought about answering
it and then forgot about it :-D

so yes more than 32 subblocks per block significant increase the
performance of filesystems with small files, for the sake of the argument
let's say 32k in a large block filesystem again for sake of argument say
16MB.

you probably ask why ?

if you create a file and write 32k into it in a pre 5.0.0 Version 16 MB
filesystem your client actually doesn't write 32k to the NSD Server, it
writes 512k, because thats the subblock size and we need to write the full
subblock (for security reasons). so first you waste significant memory on
the client to cache that zero padding, you waste network bandwidth and you
waste NSD Server cache because you store it there too. this means you
overrun the cache more quickly, means you start doing read/modify writes
earlier on all your nice large raid tracks... i guess you get the story by
now.

in fact,  if you have a good raid code that can drive really a lot of
bandwidth out of individual drives like a GNR system you get more
performance for small file writes as larger your blocksize is, because we
can 'pack' more files into larger i/os and therefore turn a small file
create workload into a bandwidth workload, essentially exactly what we did
and i demonstrated in the CORAL presentation .

hope that makes this crystal clear now .

sven



On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister <aaron.s.knister at nasa.gov>
wrote:

> Thanks, Alex. I'm all too familiar with the trade offs between large
> blocks and small files and we do use pretty robust SSD storage for our
> metadata. We support a wide range of workloads and we have some folks
> with many small (<1M) files and other folks with many large (>256MB) files.
>
> My point in this thread is that IBM has said over and over again in
> presentations that there is a significant performance gain with the >32
> subblocks code on filesystems with large block sizes (although to your
> point I'm not clear on exactly what large means since I didn't define
> large in this context). Therefore given that the >32 subblock code gives
> a significant performance gain one could reasonably assume that having a
> filesystem with >32 subblocks is required to see this gain (rather than
> just running the >32 subblocks code on an fs w/o > 32 subblocks).
>
> This lead me to ask about a migration tool because in my mind if there's
> a performance gain from having >32 subblocks on the FS I'd like that
> feature and having to manually copy 10's of PB to new hardware to get
> this performance boost is unacceptable. However, IBM can't seem to make
> up their mind about whether or not the >32 subblocks code *actually*
> provides a performance increase. This seems like a pretty
> straightforward question.
>
> -Aaron
>
> On 12/15/17 3:48 PM, Alex Chekholko wrote:
> > Hey Aaron,
> >
> > Can you define your sizes for "large blocks" and "small files"?  If you
> > dial one up and the other down, your performance will be worse.  And in
> > any case it's a pathological corner case so it shouldn't matter much for
> > your workflow, unless you've designed your system with the wrong values.
> >
> > For example, for bioinformatics workloads, I prefer to use 256KB
> > filesystem block size, and I'd consider 4MB+ to be "large block size",
> > which would make the filesystem obviously unsuitable for processing
> > millions of 8KB files.
> >
> > You can make a histogram of file sizes in your existing filesystems and
> > then make your subblock size (1/32 of block size) on the smaller end of
> > that.   Also definitely use the "small file in inode" feature and put
> > your metadata on SSD.
> >
> > Regards,
> > Alex
> >
> > On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> > <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
> >
> >     Thanks, Bill.
> >
> >     I still don't feel like I've got an clear answer from IBM and frankly
> >     the core issue of a lack of migration tool was totally dodged.
> >
> >     Again in Sven's presentation from SSUG @ SC17
> >     (
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
> >     he mentions "It has a significant performance penalty for small
> files in
> >     large block size filesystems" and the demonstrates that with several
> >     mdtest runs (which show the effect with and without the >32
> >     subblocks code):
> >
> >
> >     4.2.1 base code - SUMMARY: (of 3 iterations)
> >     File creation : Mean = 2237.644
> >
> >     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY:
> (of
> >     3 iterations)
> >     File creation : Mean = 12866.842
> >
> >     more sub blocks per block (4.2.2 + morethan32subblock code):
> >     File creation : Mean = 40316.721
> >
> >     Can someone (ideally Sven) give me a straight answer as to whether or
> >     not the > 32 subblock code actually makes a performance difference
> for
> >     small files in large block filesystems? And if not, help me
> understand
> >     why his slides and provided benchmark data have consistently
> indicated
> >     it does?
> >
> >     -Aaron
> >
> >     On 12/1/17 11:44 AM, Bill Hartner wrote:
> >     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> >     > results @ user group meeting.
> >     >
> >     > -Bill
> >     >
> >     > Bill Hartner
> >     > IBM Systems
> >     > Scalable I/O Development
> >     > Austin, Texas
> >     > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
> >     > home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >
> >     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017
> 06:53:44
> >     > AM---Bill, could you say something about what the metadataJan-Frode
> >     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> >     > about what the metadata-storage here was? ESS/NL-SAS/3way
> replication?
> >     >
> >     > From: Jan-Frode Myklebust <janfrode at tanso.net <mailto:
> janfrode at tanso.net>>
> >     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>
> >     > Date: 12/01/2017 06:53 AM
> >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     > Sent by: gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> >     >
> >     >
> >
>  ------------------------------------------------------------------------
> >     >
> >     >
> >     >
> >     > Bill, could you say something about what the metadata-storage here
> was?
> >     > ESS/NL-SAS/3way replication?
> >     >
> >     > I just asked about this in the internal slack channel #scale-help
> today..
> >     >
> >     >
> >     >
> >     > -jf
> >     >
> >     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
> <_bhartner at us.ibm.com_
> >     > <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>:
> >     >
> >     >     > "It has a significant performance penalty for small files in
> >     large
> >     >     > block size filesystems"
> >     >
> >     >     Aaron,
> >     >
> >     >     Below are mdtest results for a test we ran for CORAL - file
> >     size was
> >     >     32k.
> >     >
> >     >     We have not gone back and ran the test on a file system
> formatted
> >     >     without > 32 subblocks. We'll do that at some point...
> >     >
> >     >     -Bill
> >     >
> >     >     -- started at 10/28/2017 17:51:38 --
> >     >
> >     >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> >     >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
> >     >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
> >     -p 360
> >     >     -u -y
> >     >     Path: /ibm/fs2-16m-10
> >     >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> >     >
> >     >     228 tasks, 67239936 files
> >     >
> >     >     SUMMARY: (of 3 iterations)
> >     >     Operation Max Min Mean Std Dev
> >     >     --------- --- --- ---- -------
> >     >     File creation : 51953.498 50558.517 51423.221 616.643
> >     >     File stat : 0.000 0.000 0.000 0.000
> >     >     File read : 0.000 0.000 0.000 0.000
> >     >     File removal : 96746.376 92149.535 94658.774 1900.187
> >     >     Tree creation : 1.588 0.070 0.599 0.700
> >     >     Tree removal : 0.213 0.034 0.097 0.082
> >     >
> >     >     -- finished at 10/28/2017 19:51:54 --
> >     >
> >     >     Bill Hartner
> >     >     IBM Systems
> >     >     Scalable I/O Development
> >     >     Austin, Texas_
> >     >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
> >     <mailto:bhartner at us.ibm.com>>
> >     >     home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >     _
> >     >     __gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>> wrote on
> >     >     11/29/2017 04:41:48 PM:
> >     >
> >     >     > From: Aaron Knister <_aaron.knister at gmail.com_
> >     >     <mailto:aaron.knister at gmail.com <mailto:
> aaron.knister at gmail.com>>>
> >     >
> >     >
> >     >     > To: gpfsug main discussion list
> >     >     <_gpfsug-discuss at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>>
> >     >
> >     >     > Date: 11/29/2017 04:42 PM
> >     >
> >     >
> >     >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
> >     >
> >     >     >
> >     >
> >     >     > Thanks, Nikhil. Most of that was consistent with my
> understnading,
> >     >     > however I was under the impression that the >32 subblocks
> code is
> >     >     > required to achieve the touted 50k file creates/second that
> Sven has
> >     >     > talked about a bunch of times:
> >     >     >
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >     <
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >>
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
> >     <
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >>
> >     >     > _Sven_Oehme_-_News_from_Research.pdf
> >     >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
> >     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >>
> >     >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> >     >
> >     >
> >     >     > from those presentations regarding 32 subblocks:
> >     >     >
> >     >     > "It has a significant performance penalty for small files in
> large
> >     >     > block size filesystems"
> >     >
> >     >     > although I'm not clear on the specific definition of
> "large". Many
> >     >     > filesystems I encounter only have a 1M block size so it may
> not
> >     >     > matter there, although that same presentation clearly shows
> the
> >     >     > benefit of larger block sizes which is yet *another* thing
> for which
> >     >     > a migration tool would be helpful.
> >     >
> >     >     > -Aaron
> >     >     >
> >     >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
> >     >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
> >     <mailto:nikhilk at us.ibm.com>>> wrote:
> >     >
> >     >     > Hi,
> >     >     >
> >     >     > I would like to clarify migration path to 5.0.0 from 4.X.X
> >     clusters.
> >     >     > For all Spectrum Scale clusters that are currently at 4.X.X,
> >     it is
> >     >     > possible to migrate to 5.0.0 with no offline data migration
> >     and no
> >     >     > need to move data. Once these clusters are at 5.0.0, they
> will
> >     >     > benefit from the performance improvements, new features
> (such as
> >     >     > file audit logging), and various enhancements that are
> >     included in
> >     >     5.0.0.
> >     >     >
> >     >     > That being said, there is one enhancement that will not be
> >     applied
> >     >     > to these clusters, and that is the increased number of
> >     sub-blocks
> >     >     > per block for small file allocation. This means that for file
> >     >     > systems with a large block size and a lot of small files, the
> >     >     > overall space utilization will be the same it currently is
> >     in 4.X.X.
> >     >     > Since file systems created at 4.X.X and earlier used a block
> >     size
> >     >     > that kept this allocation in mind, there should be very
> little
> >     >     > impact on existing file systems.
> >     >     >
> >     >     > Outside of that one particular function, the remainder of the
> >     >     > performance improvements, metadata improvements, updated
> >     >     > compatibility, new functionality, and all of the other
> >     enhancements
> >     >     > will be immediately available to you once you complete the
> >     upgrade
> >     >     > to 5.0.0 -- with no need to reformat, move data, or take
> >     your data
> >     >     offline.
> >     >     >
> >     >     > I hope that clarifies things a little and makes the upgrade
> path
> >     >     > more accessible.
> >     >     >
> >     >     > Please let me know if there are any other questions or
> concerns.
> >     >     >
> >     >     > Thank you,
> >     >     > Nikhil Khandelwal
> >     >     > Spectrum Scale Development
> >     >     > Client Adoption
> >     >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>
> >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >
> >     >     > _https://urldefense.proofpoint.com/v2/url?_
> >     <https://urldefense.proofpoint.com/v2/url?_>
> >     >     >
> >     >
>  u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
> >     >     >
> >     >
>  siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> >     >
> >     >     _______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>_
> >     >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>_______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >
> >     >
> >     >
> >     >
> >     >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >     >
> >
> >     --
> >     Aaron Knister
> >     NASA Center for Climate Simulation (Code 606.2)
> >     Goddard Space Flight Center
> >     (301) 286-2776 <tel:%28301%29%20286-2776>
> >     _______________________________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171215/1d465799/attachment-0002.htm>

From Robert.Oesterlin at nuance.com  Mon Dec 18 19:43:35 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Mon, 18 Dec 2017 19:43:35 +0000
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=> replied to the IBM Spectrum Scale V5.0 announcements<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=> topic thread in the General Parallel File System - Announce (GPFS - Announce)<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=> forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
as updated, April 2017

Changes to this release of the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library include the following:

Added DMPs for TIP events

       A topic is added listing the directed maintenance procedures for TIP events.
       The DMPs help users resolve issues caused due to TIP events.

AFM and AFM DR

       - Compression and snapshot ILM policy supported.
       - A general recommendation added for the Gateway node.
       - Configuration parameters added - afmMaxParallelRecoveries, afmAsyncOpWaitTimeout,
           afmSyncOpWaitTimeout, and afmRevalOpWaitTimeout.
       - Configuration parameters modified - afmRPO and afmHashVersion.

Authentication: Authentication packages

       Updated the authentication page to include packages specific to Ubuntu.

Authentication: AD-based authentication

       New information is added on NFS with server-side group
       lookup and Active Directory authentication.

Authentication: Primary group selection configurable for AD + RFC2307 based authentication

       Ability to choose primary group as set in "UNIX attributes" of a user on
       Active Directory introduced with AD + RFC2307 based authentication scheme.
       Earlier, Windows primary group was by default selected as the primary group.

Big data and analytics
       - The GPFS Ambari integration package is now called the IBM Spectrum
         Scale Ambari management pack (in short, management pack or MPack).
       - IBM Spectrum Scale Ambari management pack version 2.4.2.1 with
         HDFS Transparency version 2.7.3.1 supports BI 4.2/BI 4.2.5 IOP migration
         to HDP 2.6.2.
       - Supports the remote mount configuration in Ambari.
       - Supports the multiple file systems configuration. In management
         pack version 2.4.2.1, the current limit is two file systems.
       - The Short circuit write is supported for better performance.
       - In HDFS Transparency, the Ranger performance is enhanced.

Changes to IBM Spectrum Scale management API

   Added the following new commands:

       GET /perfmon/data
       GET /filesystems/{filesystemName}/afm/state
       DELETE /nodes/{name}
       POST /nodes
       GET /nodeclasses
       POST /nodeclasses
       DELETE /nodeclasses/{nodeclassName}
       GET /nodeclasses/{nodeclassName}
       PUT /nodeclasses/{nodeclassName}
       DELETE /jobs/jobId
       POST /filesystems/{filesystemName}/filesets/{filesetName}/psnaps
       DELETE /filesystems/{filesystemName}/filesets/{filesetName}/psnaps/{snapshotName}
       GET /thresholds
       GET /thresholds/{name}
       POST /thresholds
       DELETE /thresholds/{name}

IBM Spectrum Scale GUI changes

   - Added new Networks page to monitor the performance, configuration, and
      adapters of network configurations in the cluster. You can monitor the
      network performance with respect to the IP and RDMA interfaces used in the configuration.
    - Added new Monitoring > Thresholds page to create and monitor the threshold rules
      that are defined in the system.
    - Added Access > Remote Connections page to enable the GUI node of the
      local cluster to monitor the remote cluster by establishing a connection
      with the GUI node of the remote cluster.
    - Added Settings > Call Home page to configure call home. Configuring
      the call home feature helps the IBM? Support to monitor the system.
      Configuring call home also helps to reduce the response time of the
      IBM Support to resolve any issues.
      The diagnostic data that is downloaded through the Settings > Diagnostic Data
      can be uploaded to a problem management record (PMR) by using
      the call home feature in the backend. To upload the diagnostic data, right-click
      the relevant data set from the Previously Collected Diagnostic Data, and
      select Upload to PMR.
    - Added file system creation capabilities in GUI. Use the Files > File Systems > Create File System
      option to launch the Create File System wizard. In the Create File System wizard, you can specify
      the following details of the file system:
         - File system name
         - Storage pools
         - NSDs for the file systems
         - Failure groups
         - NSD order for data writes
         - Maximum number of Spectrum Scale clients
         - Maximum number of inodes of the root fileset
         - Whether to enable quota and scope for the quota definition
         - Whether to enable DMAPI
         - Mount point and automatic mount mode
    - Added the aggregation levels Access Point and Filesets and removed Account
      for the resource type Transparent Cloud Tiering in the Monitoring > Statisitics page.
    - The Files > Transparent Cloud Tiering page now displays the file systems
      and filesets that are mapped with the cloud service. It also shows the
      connection of such a container pair configuration to a cloud account and
      the corresponding CSAP that is configured for the cloud account.
    - Changes to capacity monitoring in the GUI
         - Moved the Capacity page from Monitoring to Files menu in the navigation
           and renamed the GUI page to User Capacity.
         - Only the file data user capacity can be viewed from the Files > User Capacity page.
           Removed the pools, filesets, file system capacity monitoring options from the Files > User Capacity page.
           You can monitor the capacity of these components from the respective GUI pages.
         - Replaced the GPFSPoolCap sensor with the GPFSPool sensor and separate data and metadata
           level capacity monitoring are introduced in the performance charts available in the

           Files > File Systems and Storage > Pools pages.
         - New GPFSPool-based data and metadata performance monitoring metrics are available
           for selection in the Files > Statistics > Edit > Capacity section.
           You need to select the aggregation level as Pool to view these metrics.
    - AFM monitoring changes in the GUI
         - Provides the number of AFM filesets and the corresponding export server maps.
           Each export map establishes a mapping between the gateway node and the NFS
           host name to allow parallel data transfers from cache to home.
         - By using the Request Access option available in the Files > Active File Management
           or Access > Remote Connection page in the GUI, you can now establish connection with remote clusters.
           After establishing the connection, you can monitor the following AFM and AFM DR
           configuration details across clusters:
              * On home and secondary, you can see the AFM relationships configuration,
                health status, and performance values of the Cache and Disaster Recovery grids.
              * On the Overview tab of the detailed view, the available home and secondary inodes are available.
              * On the Overview tab of the detailed view, the details such as NFS throughput,
                IOPs, and latency details are available, if the protocol is NFS.
    - New option to create AFM peer snapshots through GUI. Use the Create Peer Snapshot option
      in the Files > Snapshots page to create peer snapshots. You can view and delete these peer snapshots from
      the Snapshots page and also from the detailed view of the Files > Active File Management page.

Encryption: GSKit V8 improves cryptographic performance on IBM POWER8

   The IBM Global Security Kit (GSKit) Version 8 and later improves cryptographic performance
   on IBM POWER8 hardware. The version of GSKit that is shipped with IBM Spectrum Scale v5.0.0
   offers better performance on POWER8, compared with the versions shipped with earlier releases.

File compression: The lz4 library provides fast access to compressed data

   File compression supports the lz4 compression library. Lz4 is intended primarily for active data and
   favors read-access speed over maximized space saving.

File data: Block and subblock sizes improve I/O performance and reduce fragmentation

   The default block size is larger, 4 MiB instead of 256 KiB, and the sizes of subblocks relative to blocks
   s are smaller, for example, 8 KiB subblocks in a 4 MiB block. A larger block size improves the
   file system performance and a smaller subblock size reduces the amount of unused space. For many business
   applications, the default value of 4 MiB provides the best balance of improved performance
   and reduced fragmentation.

File encryption: AES-XTS encryption is faster on x86 in non-FIPS mode

    On x86 architecture in non-FIPS mode, file encryption with the AES algorithm in XTS mode
    is faster than it was.

File systems: File system rebalancing is faster

    Rebalancing is implemented by a lenient round-robin method that typically runs faster than the
    previously used method of strict round robin. The strict round robin method is available as an option.

installation toolkit changes

    - The installation toolkit has added support for the installation and the deployment
      of IBM Spectrum Scale in a cluster containing Elastic Storage Server (ESS).
    - The installation toolkit has added support for enabling and configuring call home.
    - The installation toolkit has added support for enabling and configuring file audit logging.
    - The installation toolkit has added support for the installation and the
      deployment of IBM Spectrum Scale on Ubuntu 16.04 LTS nodes.
    - The installation toolkit has added verification of passwordless SSH during
      prechecks before installation, deployment, or upgrade.
    - The installation toolkit has added support for cumulative object upgrade.

mmafmctl command

    The --outband parameter is deprecated.

mmcallhome command: Enhancements

    - Addition of -Y option
        * The -Y displays the command output in a parseable format with a colon (:) as a field delimiter.
    - Addition of --pmr option
        * The --pmr option allows you to upload data to existing PMRs using the
          mmcallhome run SendFile command.

mmchconfig command: Enhancements

    - Encrypted files can be copied into an LROC device
        * With the lrocEnableStoringClearText attribute, you can control whether file
          data from encrypted files, which is held in memory as cleartext, is
          copied into a local read-only cache (LROC) device.
    - InfiniBand addresses can be specified for RDMA transfers
        * In the verbsPorts attribute, you can specify InfiniBand addresses
          for RDMA transfers between an NSD client and server.

mmchnsd command: Change NSDs without unmounting the file system

    When you add or remove NSDs or do other operations with mmchnsd,
    you do not need to unmount the file system.

mmcrfs command: Enhancements

     - The default data block size is 4 MiB with an 8 KiB subblock size
         * If no block size is specified, a file system is created with a 4 MiB block size and an
           8 KiB subblock size. The minimum release level (minReleaseLevel) of the
           cluster must be 5.0.0 or greater when the file system is created.
     - The default log file size depends on block size and metadata size
         * If the block size is 512 KiB or larger and the metadata block size 256 KiB or larger,
           then the default log file size is 32 MiB. Otherwise, the default log file
           size is 4 MiB or the metadata block size, whichever is larger.
     - The default method for updating atime is relatime
         * If the minimum release level (minReleaseLevel) of the cluster is 5.0.0 or greater
           when the file system is created, the default method for updating atime is relatime.

mmdsh command: Several options are no longer supported

     The --ignoreSignal, -I, and -d options are no longer supported.
     Do not use these options unless instructed to by IBM support personnel.

mmfsck command: Display an interim status report at any time

     While a long-running instance of mmfsck is in progress, you can start another instance
     of mmfsck with the --status-report parameter to display current
     status information from all the nodes that are participating in the mmfsck run.

mmgetstate command: Display the unresponsive state

     The command returns the unresponsive state when the GPFS
     daemon is running but is not responding.

mmhealth command: Addition to measurement options

     Measurement options for filesystem, SMB node, and NFS node
     has been added to the mmhealth command.

mmkeyserv command: The simplified method supports certificate chains from a certificate authority.

     In the simplified method, with the --kmip-cert parameter, you can set up encryption with IBM?
     Security Key Lifecycle Manager (SKLM) as the key management server and with a certificate signed
     by a certificate authority (CA) on the KMIP port of the SKLM server.

mmnetverify command: Enhancements

     - Verify the network operation of nodes in a subnet
         * With the --subnets parameters, you can specify the subnet
           addresses of the nodes that you want to verify.
     - Verify that nodes can handle a new MTU size
         * With the ping-packet-size parameter, you can specify the size
           of the ICMP echo request packets that are sent between local node and the target
           node during the ping test.

mmtracectl command: Display the tracing status

     The --status parameter displays the tracing status of the specified nodes.

New feature for threshold monitoring

      Starting from IBM Spectrum Scale version 5.0.0, if multiple thresholds rules
      have overlapping entities for the same metrics, only one of the concurrent rules is
      made actively eligible.

NFS: Dynamic export changes

      You can dynamically change the export configuration without restarting the NFS service.

Object

      - Support for Ubuntu
      - Support for sudo wrapper for Object on Ubuntu
      - Support for cumulative upgrades from older versions
      - Object snap enhancement to contain keystore logs

Protocol support: Enhanced

      Protocol support is extended to Ubuntu 10.04

Setting up a system for storing crash files for Ubuntu

      A topic is added to describe how to set up a system for storing crash files for Ubuntu.

SMB: DFS redirects for SMB shares

      New option to configure DFS redirects for SMB shares.

SMB: SMB server upgrade changes

      Two events on CTDB version match/mismatch are added to the RAS events.

Sudo wrappers: Root-level processes can call administration commands directly

      Root-level background processes, such as cron and callback programs, can
      successfully call administration commands directly rather than through sudo
      when sudo wrappers are enabled.

Supported clients for NFS

      A topic is added listing the clients that are supported by NFS protocol.

Transparent cloud tiering

      - Support for multiple cloud storage accounts
      - Support for multiple file systems or filesets per node group
      - Enhanced support for large file systems provided by container spillover
      - Support for associating file sets with containers for enhanced granularity
      - Multiple URL and region support at the node level
      - Support for creating a cloud service separately for tiering and sharing operations.
      - Unique encryption key per cloud container
      - Support for remotely mounted clients.
      - Support for Amazon S3 regions requiring Sigv4 security support,
        including the US government cloud region.
      - Ability to enable or disable transparent recall for files for a given file
        system instantly, without having to rewrite a policy.
      - Support for backing up and restoring the Cloud services configuration in case of any disaster.
      - Support for backing up the Cloud services database to the cloud.
      - Support for restoring Transparent cloud tiering service on an identical backup cluster.
      - Support for checking the integrity of the Cloud services
        database after any system crash or outage.
      - Support for auditing events relating to each operation performed in Cloud services.

New commands

      mmaudit mmmsgqueue

Changed commands

      mmafmctl mmadddisk mmcallhome mmchattr mmchcluster mmchconfig mmchfs mmchnsd
      mmcloudgateway mmcrcluster mmcrfs mmdeldisk mmdsh mmfsck mmgetstate
      mmkeyserv mmnetverify mmnfs mmrestripefile mmrestripefs mmsmb mmtracectl
      mmuserauth

Deleted commands

      mmrest

New messages

      6027-1264, 6027-1757, 6027-2394, 6027-2395, 6027-2396, 6027-2397
      6027-2398, 6027-2399, 6027-2400, 6027-2401, 6027-3259, 6027-3408
      6027-3597, 6027-3598, 6027-3599, 6027-3600, 6027-3601, 6027-3602
      6027-3603, 6027-3604, 6027-3730, 6027-3921, 6027-3922, 6027-3923
      6027-3924, 6027-3925, 6027-3926, 6027-3927, 6027-3928
      6027-3929, 6027-3930, 6027-3931, 6027-4019

Changed messages

      6027-928


     Open this item<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab-234019da9b-2D248a-2D49ec-2D9503-2De612becd953e&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=Q6E0fg8sor2mORS3ugkurXkuV2ZUDqRd_C6_jpYXVVM&e=>


Posting Date:

Monday, December 18, 2017 at 2:26:38 PM EST



To unsubscribe or change settings, please go to your developerWorks community Settings.<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_news&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=5C4z0hhuaas4Ghs1g4YfFTCE0eyNYJVmLPYrGPCuE7Q&e=>
This is a notification sent from developerWorks community.

The information contained in this e-mail message is intended only for the use of the named recipients and is subject to the IBM Online Privacy Statement<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ibm.com_privacy_details_us_en_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=7gE9uoypP_yHb1jMMSjdCN_xyJ7UU2Ifrx1TchwMkQA&e=>. If you have received this e-mail in error, please contact the sender to report the error and then delete this message from your system.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/9c66dc47/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 289 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/9c66dc47/attachment-0002.png>

From Kevin.Buterbaugh at Vanderbilt.Edu  Mon Dec 18 19:51:56 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Mon, 18 Dec 2017 19:51:56 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
	<DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
Message-ID: <348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu>

Hi All,

Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up.

Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this.  Richards? suggestion below errors for me on the ?-g Administrator? part.  Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked.

No matter what I do, I cannot log in to the GUI.  Thanks for the suggestions, though?

Kevin

On Dec 7, 2017, at 6:10 AM, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:

Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error.

That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself.

[root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin
EFSSG0019I The user admin2 has been successfully created.
EFSSG1000I The command completed successfully.


Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 07 December 2017 11:57
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.
Get Outlook for Android<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fghei36&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=sPRr5u4lo%2BAwBPBQ2%2BdXw%2F2EWUAqy30Fk0UNssRFWHU%3D&reserved=0>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin


On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/8eab1462/attachment-0002.htm>

From Kevin.Buterbaugh at Vanderbilt.Edu  Mon Dec 18 20:06:30 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Mon, 18 Dec 2017 20:06:30 +0000
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
 Central
In-Reply-To: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
References: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
Message-ID: <662BADD6-27D5-4DE6-9EC8-5BE1E73C5873@vanderbilt.edu>

Hi All,

GPFS 5.0 was announced on Friday ? and today:

IBM Spectrum Scale
: IBM Spectrum Scale: NFS operations may fail with IO-Error<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ibm.com%2Fsupport%2Fdocview.wss%3Fuid%3Dssg1S1011791%26myns%3Ds033%26mynp%3DOCSTXKQY%26mynp%3DOCSWJ00%26mync%3DE%26cm_sp%3Ds033-_-OCSTXKQY-OCSWJ00-_-E&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C07b900d4a1aa4e92852e08d54634aa16%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492114479674351&sdata=i%2B8WSjOObZoaSpNLnEZLHq%2FqXwj%2BetYSJzojl9Rh%2FgQ%3D&reserved=0>

IBM has identified an issue with IBM Spectrum Scale 5.0.0.0 Protocol support for NFSv3/v4 in which IO-errors may be returned to the NFS client if the NFS server accumulates file-descriptor resources beyond the defined limit. Accumulation of file descriptor resources will occur when NFSv3 file create operations are sent against files that are already in use.
Bob?s suggestion in a previous e-mail to the list about installing this on a test cluster is almost certainly very, VERY good advice.  That?s certainly what we will do after the holiday break...

Kevin

On Dec 18, 2017, at 1:43 PM, Oesterlin, Robert <Robert.Oesterlin at nuance.com<mailto:Robert.Oesterlin at nuance.com>> wrote:

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Reply-To: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com<mailto:gpfs at us.ibm.com>' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum.

<image001.png> gpfs at us.ibm.com<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DYIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=4BVRC4gq9NXkxODASI9hJ5BV5dQQ94rUjzVc1fDndgs%3D&reserved=0> replied to the IBM Spectrum Scale V5.0 announcements<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3D05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=eCCEi0WsUIaKWOw1BHWueMPQ5O1LF6rYFZnHouwvZ8Y%3D&reserved=0> topic thread in the General Parallel File System - Announce (GPFS - Announce)<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0000-2D0000-2D000000001606%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DzTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=uxfVFydD9TI9l6%2BF%2F3rP7y%2F9Uo0NnBNlmr2tiMrEzbM%3D&reserved=0> forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www-2D933.ibm.com_support_fixcentral%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DiHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=cAwptvUoVH%2B57%2Fat%2FZgt3ft8Gv%2FLsNRWXQzv%2BiuAFuQ%3D&reserved=0>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
as updated, April 2017

Changes to this release of the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library include the following:

Added DMPs for TIP events

       A topic is added listing the directed maintenance procedures for TIP events.
       The DMPs help users resolve issues caused due to TIP events.

AFM and AFM DR

       - Compression and snapshot ILM policy supported.
       - A general recommendation added for the Gateway node.
       - Configuration parameters added - afmMaxParallelRecoveries, afmAsyncOpWaitTimeout,
           afmSyncOpWaitTimeout, and afmRevalOpWaitTimeout.
       - Configuration parameters modified - afmRPO and afmHashVersion.

Authentication: Authentication packages

       Updated the authentication page to include packages specific to Ubuntu.

Authentication: AD-based authentication

       New information is added on NFS with server-side group
       lookup and Active Directory authentication.

Authentication: Primary group selection configurable for AD + RFC2307 based authentication

       Ability to choose primary group as set in "UNIX attributes" of a user on
       Active Directory introduced with AD + RFC2307 based authentication scheme.
       Earlier, Windows primary group was by default selected as the primary group.

Big data and analytics
       - The GPFS Ambari integration package is now called the IBM Spectrum
         Scale Ambari management pack (in short, management pack or MPack).
       - IBM Spectrum Scale Ambari management pack version 2.4.2.1 with
         HDFS Transparency version 2.7.3.1 supports BI 4.2/BI 4.2.5 IOP migration
         to HDP 2.6.2.
       - Supports the remote mount configuration in Ambari.
       - Supports the multiple file systems configuration. In management
         pack version 2.4.2.1, the current limit is two file systems.
       - The Short circuit write is supported for better performance.
       - In HDFS Transparency, the Ranger performance is enhanced.

Changes to IBM Spectrum Scale management API

   Added the following new commands:

       GET /perfmon/data
       GET /filesystems/{filesystemName}/afm/state
       DELETE /nodes/{name}
       POST /nodes
       GET /nodeclasses
       POST /nodeclasses
       DELETE /nodeclasses/{nodeclassName}
       GET /nodeclasses/{nodeclassName}
       PUT /nodeclasses/{nodeclassName}
       DELETE /jobs/jobId
       POST /filesystems/{filesystemName}/filesets/{filesetName}/psnaps
       DELETE /filesystems/{filesystemName}/filesets/{filesetName}/psnaps/{snapshotName}
       GET /thresholds
       GET /thresholds/{name}
       POST /thresholds
       DELETE /thresholds/{name}

IBM Spectrum Scale GUI changes

   - Added new Networks page to monitor the performance, configuration, and
      adapters of network configurations in the cluster. You can monitor the
      network performance with respect to the IP and RDMA interfaces used in the configuration.
    - Added new Monitoring > Thresholds page to create and monitor the threshold rules
      that are defined in the system.
    - Added Access > Remote Connections page to enable the GUI node of the
      local cluster to monitor the remote cluster by establishing a connection
      with the GUI node of the remote cluster.
    - Added Settings > Call Home page to configure call home. Configuring
      the call home feature helps the IBM? Support to monitor the system.
      Configuring call home also helps to reduce the response time of the
      IBM Support to resolve any issues.
      The diagnostic data that is downloaded through the Settings > Diagnostic Data
      can be uploaded to a problem management record (PMR) by using
      the call home feature in the backend. To upload the diagnostic data, right-click
      the relevant data set from the Previously Collected Diagnostic Data, and
      select Upload to PMR.
    - Added file system creation capabilities in GUI. Use the Files > File Systems > Create File System
      option to launch the Create File System wizard. In the Create File System wizard, you can specify
      the following details of the file system:
         - File system name
         - Storage pools
         - NSDs for the file systems
         - Failure groups
         - NSD order for data writes
         - Maximum number of Spectrum Scale clients
         - Maximum number of inodes of the root fileset
         - Whether to enable quota and scope for the quota definition
         - Whether to enable DMAPI
         - Mount point and automatic mount mode
    - Added the aggregation levels Access Point and Filesets and removed Account
      for the resource type Transparent Cloud Tiering in the Monitoring > Statisitics page.
    - The Files > Transparent Cloud Tiering page now displays the file systems
      and filesets that are mapped with the cloud service. It also shows the
      connection of such a container pair configuration to a cloud account and
      the corresponding CSAP that is configured for the cloud account.
    - Changes to capacity monitoring in the GUI
         - Moved the Capacity page from Monitoring to Files menu in the navigation
           and renamed the GUI page to User Capacity.
         - Only the file data user capacity can be viewed from the Files > User Capacity page.
           Removed the pools, filesets, file system capacity monitoring options from the Files > User Capacity page.
           You can monitor the capacity of these components from the respective GUI pages.
         - Replaced the GPFSPoolCap sensor with the GPFSPool sensor and separate data and metadata
           level capacity monitoring are introduced in the performance charts available in the

           Files > File Systems and Storage > Pools pages.
         - New GPFSPool-based data and metadata performance monitoring metrics are available
           for selection in the Files > Statistics > Edit > Capacity section.
           You need to select the aggregation level as Pool to view these metrics.
    - AFM monitoring changes in the GUI
         - Provides the number of AFM filesets and the corresponding export server maps.
           Each export map establishes a mapping between the gateway node and the NFS
           host name to allow parallel data transfers from cache to home.
         - By using the Request Access option available in the Files > Active File Management
           or Access > Remote Connection page in the GUI, you can now establish connection with remote clusters.
           After establishing the connection, you can monitor the following AFM and AFM DR
           configuration details across clusters:
              * On home and secondary, you can see the AFM relationships configuration,
                health status, and performance values of the Cache and Disaster Recovery grids.
              * On the Overview tab of the detailed view, the available home and secondary inodes are available.
              * On the Overview tab of the detailed view, the details such as NFS throughput,
                IOPs, and latency details are available, if the protocol is NFS.
    - New option to create AFM peer snapshots through GUI. Use the Create Peer Snapshot option
      in the Files > Snapshots page to create peer snapshots. You can view and delete these peer snapshots from
      the Snapshots page and also from the detailed view of the Files > Active File Management page.

Encryption: GSKit V8 improves cryptographic performance on IBM POWER8

   The IBM Global Security Kit (GSKit) Version 8 and later improves cryptographic performance
   on IBM POWER8 hardware. The version of GSKit that is shipped with IBM Spectrum Scale v5.0.0
   offers better performance on POWER8, compared with the versions shipped with earlier releases.

File compression: The lz4 library provides fast access to compressed data

   File compression supports the lz4 compression library. Lz4 is intended primarily for active data and
   favors read-access speed over maximized space saving.

File data: Block and subblock sizes improve I/O performance and reduce fragmentation

   The default block size is larger, 4 MiB instead of 256 KiB, and the sizes of subblocks relative to blocks
   s are smaller, for example, 8 KiB subblocks in a 4 MiB block. A larger block size improves the
   file system performance and a smaller subblock size reduces the amount of unused space. For many business
   applications, the default value of 4 MiB provides the best balance of improved performance
   and reduced fragmentation.

File encryption: AES-XTS encryption is faster on x86 in non-FIPS mode

    On x86 architecture in non-FIPS mode, file encryption with the AES algorithm in XTS mode
    is faster than it was.

File systems: File system rebalancing is faster

    Rebalancing is implemented by a lenient round-robin method that typically runs faster than the
    previously used method of strict round robin. The strict round robin method is available as an option.

installation toolkit changes

    - The installation toolkit has added support for the installation and the deployment
      of IBM Spectrum Scale in a cluster containing Elastic Storage Server (ESS).
    - The installation toolkit has added support for enabling and configuring call home.
    - The installation toolkit has added support for enabling and configuring file audit logging.
    - The installation toolkit has added support for the installation and the
      deployment of IBM Spectrum Scale on Ubuntu 16.04 LTS nodes.
    - The installation toolkit has added verification of passwordless SSH during
      prechecks before installation, deployment, or upgrade.
    - The installation toolkit has added support for cumulative object upgrade.

mmafmctl command

    The --outband parameter is deprecated.

mmcallhome command: Enhancements

    - Addition of -Y option
        * The -Y displays the command output in a parseable format with a colon (:) as a field delimiter.
    - Addition of --pmr option
        * The --pmr option allows you to upload data to existing PMRs using the
          mmcallhome run SendFile command.

mmchconfig command: Enhancements

    - Encrypted files can be copied into an LROC device
        * With the lrocEnableStoringClearText attribute, you can control whether file
          data from encrypted files, which is held in memory as cleartext, is
          copied into a local read-only cache (LROC) device.
    - InfiniBand addresses can be specified for RDMA transfers
        * In the verbsPorts attribute, you can specify InfiniBand addresses
          for RDMA transfers between an NSD client and server.

mmchnsd command: Change NSDs without unmounting the file system

    When you add or remove NSDs or do other operations with mmchnsd,
    you do not need to unmount the file system.

mmcrfs command: Enhancements

     - The default data block size is 4 MiB with an 8 KiB subblock size
         * If no block size is specified, a file system is created with a 4 MiB block size and an
           8 KiB subblock size. The minimum release level (minReleaseLevel) of the
           cluster must be 5.0.0 or greater when the file system is created.
     - The default log file size depends on block size and metadata size
         * If the block size is 512 KiB or larger and the metadata block size 256 KiB or larger,
           then the default log file size is 32 MiB. Otherwise, the default log file
           size is 4 MiB or the metadata block size, whichever is larger.
     - The default method for updating atime is relatime
         * If the minimum release level (minReleaseLevel) of the cluster is 5.0.0 or greater
           when the file system is created, the default method for updating atime is relatime.

mmdsh command: Several options are no longer supported

     The --ignoreSignal, -I, and -d options are no longer supported.
     Do not use these options unless instructed to by IBM support personnel.

mmfsck command: Display an interim status report at any time

     While a long-running instance of mmfsck is in progress, you can start another instance
     of mmfsck with the --status-report parameter to display current
     status information from all the nodes that are participating in the mmfsck run.

mmgetstate command: Display the unresponsive state

     The command returns the unresponsive state when the GPFS
     daemon is running but is not responding.

mmhealth command: Addition to measurement options

     Measurement options for filesystem, SMB node, and NFS node
     has been added to the mmhealth command.

mmkeyserv command: The simplified method supports certificate chains from a certificate authority.

     In the simplified method, with the --kmip-cert parameter, you can set up encryption with IBM?
     Security Key Lifecycle Manager (SKLM) as the key management server and with a certificate signed
     by a certificate authority (CA) on the KMIP port of the SKLM server.

mmnetverify command: Enhancements

     - Verify the network operation of nodes in a subnet
         * With the --subnets parameters, you can specify the subnet
           addresses of the nodes that you want to verify.
     - Verify that nodes can handle a new MTU size
         * With the ping-packet-size parameter, you can specify the size
           of the ICMP echo request packets that are sent between local node and the target
           node during the ping test.

mmtracectl command: Display the tracing status

     The --status parameter displays the tracing status of the specified nodes.

New feature for threshold monitoring

      Starting from IBM Spectrum Scale version 5.0.0, if multiple thresholds rules
      have overlapping entities for the same metrics, only one of the concurrent rules is
      made actively eligible.

NFS: Dynamic export changes

      You can dynamically change the export configuration without restarting the NFS service.

Object

      - Support for Ubuntu
      - Support for sudo wrapper for Object on Ubuntu
      - Support for cumulative upgrades from older versions
      - Object snap enhancement to contain keystore logs

Protocol support: Enhanced

      Protocol support is extended to Ubuntu 10.04

Setting up a system for storing crash files for Ubuntu

      A topic is added to describe how to set up a system for storing crash files for Ubuntu.

SMB: DFS redirects for SMB shares

      New option to configure DFS redirects for SMB shares.

SMB: SMB server upgrade changes

      Two events on CTDB version match/mismatch are added to the RAS events.

Sudo wrappers: Root-level processes can call administration commands directly

      Root-level background processes, such as cron and callback programs, can
      successfully call administration commands directly rather than through sudo
      when sudo wrappers are enabled.

Supported clients for NFS

      A topic is added listing the clients that are supported by NFS protocol.

Transparent cloud tiering

      - Support for multiple cloud storage accounts
      - Support for multiple file systems or filesets per node group
      - Enhanced support for large file systems provided by container spillover
      - Support for associating file sets with containers for enhanced granularity
      - Multiple URL and region support at the node level
      - Support for creating a cloud service separately for tiering and sharing operations.
      - Unique encryption key per cloud container
      - Support for remotely mounted clients.
      - Support for Amazon S3 regions requiring Sigv4 security support,
        including the US government cloud region.
      - Ability to enable or disable transparent recall for files for a given file
        system instantly, without having to rewrite a policy.
      - Support for backing up and restoring the Cloud services configuration in case of any disaster.
      - Support for backing up the Cloud services database to the cloud.
      - Support for restoring Transparent cloud tiering service on an identical backup cluster.
      - Support for checking the integrity of the Cloud services
        database after any system crash or outage.
      - Support for auditing events relating to each operation performed in Cloud services.

New commands

      mmaudit mmmsgqueue

Changed commands

      mmafmctl mmadddisk mmcallhome mmchattr mmchcluster mmchconfig mmchfs mmchnsd
      mmcloudgateway mmcrcluster mmcrfs mmdeldisk mmdsh mmfsck mmgetstate
      mmkeyserv mmnetverify mmnfs mmrestripefile mmrestripefs mmsmb mmtracectl
      mmuserauth

Deleted commands

      mmrest

New messages

      6027-1264, 6027-1757, 6027-2394, 6027-2395, 6027-2396, 6027-2397
      6027-2398, 6027-2399, 6027-2400, 6027-2401, 6027-3259, 6027-3408
      6027-3597, 6027-3598, 6027-3599, 6027-3600, 6027-3601, 6027-3602
      6027-3603, 6027-3604, 6027-3730, 6027-3921, 6027-3922, 6027-3923
      6027-3924, 6027-3925, 6027-3926, 6027-3927, 6027-3928
      6027-3929, 6027-3930, 6027-3931, 6027-4019

Changed messages

      6027-928



     Open this item<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab-234019da9b-2D248a-2D49ec-2D9503-2De612becd953e%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3DQ6E0fg8sor2mORS3ugkurXkuV2ZUDqRd_C6_jpYXVVM%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=wQawdxgurt9CU5mLHBaUeqAshZxOd7tn7HrOeP4CZR8%3D&reserved=0>


Posting Date:

Monday, December 18, 2017 at 2:26:38 PM EST



To unsubscribe or change settings, please go to your developerWorks community Settings.<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__www.ibm.com_developerworks_community_news%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3D5C4z0hhuaas4Ghs1g4YfFTCE0eyNYJVmLPYrGPCuE7Q%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=KMU2UypqoZXXJlTopsMbsV3x0RUuNMJXEv1UyZEQoRY%3D&reserved=0>

This is a notification sent from developerWorks community.

The information contained in this e-mail message is intended only for the use of the named recipients and is subject to the IBM Online Privacy Statement<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__www.ibm.com_privacy_details_us_en_%26d%3DDwMFaQ%26c%3Ddjjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY%26r%3DLPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU%26m%3DNhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk%26s%3D7gE9uoypP_yHb1jMMSjdCN_xyJ7UU2Ifrx1TchwMkQA%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=NzaHT12VmX5MY%2FRIm3m4ovKIYNz%2FGMTnJj1%2FM3cOM7I%3D&reserved=0>. If you have received this e-mail in error, please contact the sender to report the error and then delete this message from your system.


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ce3b77f1a2da5444db93e08d5464f9fa0%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492230304916652&sdata=pHmKmOoSbQ5W4e2DiOfphjWgRdqT8D9NMBjtYZcH%2Bog%3D&reserved=0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/fd0d1cca/attachment-0002.htm>

From valdis.kletnieks at vt.edu  Mon Dec 18 20:08:48 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Mon, 18 Dec 2017 15:08:48 -0500
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID: <32207.1513627728@turing-police.cc.vt.edu>

Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux

that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4."

We don't use AFM or the install toolkit.  But we *do* make fairly heavy use
of mmces and nfs-ganesha - is that what they mean by "Integrated Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are doing
their upgrades - and going to 7.4 would be nice.  If there's a mine field there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)


From Kevin.Buterbaugh at Vanderbilt.Edu  Mon Dec 18 20:11:28 2017
From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L)
Date: Mon, 18 Dec 2017 20:11:28 +0000
Subject: [gpfsug-discuss] Password to GUI forgotten
In-Reply-To: <1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov>
References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu>
	<2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com>
	<26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu>
	<D4C17EBA-5252-4B13-9987-77547C795E06@gmail.com>
	<D9C70722-A688-4E62-AA1D-91776C10E1C9@vanderbilt.edu>
	<DB6PR0601MB210196EB748875471A6AFF7FDF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<DB6PR0601MB210134153E695BB890C57D50DF330@DB6PR0601MB2101.eurprd06.prod.outlook.com>
	<348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu>
	<1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov>
Message-ID: <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu>

Hi Jesse,

Thanks for the suggestion ? I find the following error very interesting:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/rmuser admin
EFSSP0010C CLI parser: The object "admin" specified for "userID" does not exist.
/root
root at testnsd1#

That says to me that I don?t have an admin user, which - if true - would explain why not a single password I can think of works.  ;-)

But as I mentioned in my original post I had this up and working earlier this fall.  While I can?t prove anything, I can?t imagine a scenario where I would deliberately choose a non-default username.  So if ?admin? has been the default login for the GPFS GUI all along then I am really mystified.

Thanks!

Kevin

On Dec 18, 2017, at 1:58 PM, Hanley, Jesse A. <hanleyja at ornl.gov<mailto:hanleyja at ornl.gov>> wrote:

Kevin,

I ran into this a couple times using 4.2.3.  This is what we used to get around it:

/usr/lpp/mmfs/gui/cli/rmuser admin
/usr/lpp/mmfs/gui/cli/mkuser admin -p <password> -g Administrator,SecurityAdmin

You may need to run the initgui command if those objects are present.  That typically gets run on first login to the GUI.

Thanks,
--
Jesse


From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Monday, December 18, 2017 at 2:52 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

Hi All,

Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up.

Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this.  Richards? suggestion below errors for me on the ?-g Administrator? part.  Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked.

No matter what I do, I cannot log in to the GUI.  Thanks for the suggestions, though?

Kevin


On Dec 7, 2017, at 6:10 AM, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:

Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error.

That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself.

[root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin
EFSSG0019I The user admin2 has been successfully created.
EFSSG1000I The command completed successfully.


Cheers
Richard

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 07 December 2017 11:57
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out.
Get Outlook for Android<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fghei36&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=sPRr5u4lo%2BAwBPBQ2%2BdXw%2F2EWUAqy30Fk0UNssRFWHU%3D&reserved=0>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Sent: Wednesday, December 6, 2017 10:41:12 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin
EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist.
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin
EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist.
/root
root at testnsd1#

I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case.

Thanks!

Kevin



On Dec 6, 2017, at 4:32 PM, Joshua Kwedar <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

Hmm.. odd.

Here?s what the lsuser output should look like.

# /usr/lpp/mmfs/gui/cli/lsuser
Name  Long name Password status Group names                 Failed login attempts
admin           active          Administrator,SecurityAdmin 0
EFSSG1000I The command completed successfully.

Can you try something like?

# /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin



From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, December 6, 2017 at 5:15 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Password to GUI forgotten

All,

Sorry - should?ve mentioned that:

/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231
EFSSG0001C Cannot validate option: login
/root
root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y
lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts:
/root
root at testnsd1#

Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November.  Thanks...

Kevin

On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) <jdkwedar at gmail.com<mailto:jdkwedar at gmail.com>> wrote:

The GUI password can be changed via command line using chuser.

/usr/lpp/mmfs/gui/cli/chuser

Usage is as follows (where userID = admin)


chuser userID {-p <newPassword> | -l <longName> | -a <name_or_id> | -d <name_or_id> | -g <name_or_id> | --expirePassword} [-o <currentPassword>]



Josh K

On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
Hi All,

So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall.  However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI.  :-(

Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch?  If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route.

Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-)

Thanks?

?
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=GyfwFULjOHTUHI2iRFDMdE1BA1A9zJebk8IxeTRGSiY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=aADPZlmzYyxhM0IhmZ4rUPOIJrR8DLodnSMhPrAJ5Dg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0

_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=VQ7O%2FNjy6wQL3MCn%2FeRJWcnvKKuzVMokqz2bj0QoxiA%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C3f1a258c20a240166fbe08d54651c592%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636492239506213881&sdata=qGxMvy3f6cfKQEmwH%2Bcc0SKVDAcaJ4Z66p%2BE9mWDwHg%3D&reserved=0>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/656e148e/attachment-0002.htm>

From aaron.s.knister at nasa.gov  Mon Dec 18 20:13:01 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Mon, 18 Dec 2017 20:13:01 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>,
	<CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
Message-ID: <6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>

Thanks Sven! That makes sense to me and is what I thought was the case which is why I was confused when I saw the reply to the thread that said the >32 subblocks code had no performance impact.

A couple more question for you? in your presentation there?s a benchmark that shows the file create performance without the zero padding. Since you mention this is done for security reasons was that feature ever promoted to a GA Scale release? I?m also wondering if you could explain the performance difference between the no zero padding code and the > 32 subblock code since given your the example of 32K files and 16MB block size I figure both cases ought to write the same amount to disk.

Thanks!

-Aaron





On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com> wrote:
i thought i answered that already, but maybe i just thought about answering it and then forgot about it :-D

so yes more than 32 subblocks per block significant increase the performance of filesystems with small files, for the sake of the argument let's say 32k in a large block filesystem again for sake of argument say 16MB.

you probably ask why ?

if you create a file and write 32k into it in a pre 5.0.0 Version 16 MB filesystem your client actually doesn't write 32k to the NSD Server, it writes 512k, because thats the subblock size and we need to write the full subblock (for security reasons). so first you waste significant memory on the client to cache that zero padding, you waste network bandwidth and you waste NSD Server cache because you store it there too. this means you overrun the cache more quickly, means you start doing read/modify writes earlier on all your nice large raid tracks... i guess you get the story by now.

in fact,  if you have a good raid code that can drive really a lot of bandwidth out of individual drives like a GNR system you get more performance for small file writes as larger your blocksize is, because we can 'pack' more files into larger i/os and therefore turn a small file create workload into a bandwidth workload, essentially exactly what we did and i demonstrated in the CORAL presentation .

hope that makes this crystal clear now .

sven



On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>> wrote:
Thanks, Alex. I'm all too familiar with the trade offs between large
blocks and small files and we do use pretty robust SSD storage for our
metadata. We support a wide range of workloads and we have some folks
with many small (<1M) files and other folks with many large (>256MB) files.

My point in this thread is that IBM has said over and over again in
presentations that there is a significant performance gain with the >32
subblocks code on filesystems with large block sizes (although to your
point I'm not clear on exactly what large means since I didn't define
large in this context). Therefore given that the >32 subblock code gives
a significant performance gain one could reasonably assume that having a
filesystem with >32 subblocks is required to see this gain (rather than
just running the >32 subblocks code on an fs w/o > 32 subblocks).

This lead me to ask about a migration tool because in my mind if there's
a performance gain from having >32 subblocks on the FS I'd like that
feature and having to manually copy 10's of PB to new hardware to get
this performance boost is unacceptable. However, IBM can't seem to make
up their mind about whether or not the >32 subblocks code *actually*
provides a performance increase. This seems like a pretty
straightforward question.

-Aaron

On 12/15/17 3:48 PM, Alex Chekholko wrote:
> Hey Aaron,
>
> Can you define your sizes for "large blocks" and "small files"?  If you
> dial one up and the other down, your performance will be worse.  And in
> any case it's a pathological corner case so it shouldn't matter much for
> your workflow, unless you've designed your system with the wrong values.
>
> For example, for bioinformatics workloads, I prefer to use 256KB
> filesystem block size, and I'd consider 4MB+ to be "large block size",
> which would make the filesystem obviously unsuitable for processing
> millions of 8KB files.
>
> You can make a histogram of file sizes in your existing filesystems and
> then make your subblock size (1/32 of block size) on the smaller end of
> that.   Also definitely use the "small file in inode" feature and put
> your metadata on SSD.
>
> Regards,
> Alex
>
> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov> <mailto:aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>>> wrote:
>
>     Thanks, Bill.
>
>     I still don't feel like I've got an clear answer from IBM and frankly
>     the core issue of a lack of migration tool was totally dodged.
>
>     Again in Sven's presentation from SSUG @ SC17
>     (http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
>     he mentions "It has a significant performance penalty for small files in
>     large block size filesystems" and the demonstrates that with several
>     mdtest runs (which show the effect with and without the >32
>     subblocks code):
>
>
>     4.2.1 base code - SUMMARY: (of 3 iterations)
>     File creation : Mean = 2237.644
>
>     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY: (of
>     3 iterations)
>     File creation : Mean = 12866.842
>
>     more sub blocks per block (4.2.2 + morethan32subblock code):
>     File creation : Mean = 40316.721
>
>     Can someone (ideally Sven) give me a straight answer as to whether or
>     not the > 32 subblock code actually makes a performance difference for
>     small files in large block filesystems? And if not, help me understand
>     why his slides and provided benchmark data have consistently indicated
>     it does?
>
>     -Aaron
>
>     On 12/1/17 11:44 AM, Bill Hartner wrote:
>     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
>     > results @ user group meeting.
>     >
>     > -Bill
>     >
>     > Bill Hartner
>     > IBM Systems
>     > Scalable I/O Development
>     > Austin, Texas
>     > bhartner at us.ibm.com<mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>>
>     > home office 512-784-0980<tel:(512)%20784-0980> <tel:512-784-0980<tel:(512)%20784-0980>>
>     >
>     >
>     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44
>     > AM---Bill, could you say something about what the metadataJan-Frode
>     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
>     > about what the metadata-storage here was? ESS/NL-SAS/3way replication?
>     >
>     > From: Jan-Frode Myklebust <janfrode at tanso.net<mailto:janfrode at tanso.net> <mailto:janfrode at tanso.net<mailto:janfrode at tanso.net>>>
>     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
>     <mailto:gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>>
>     > Date: 12/01/2017 06:53 AM
>     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     > Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>     >
>     >
>     ------------------------------------------------------------------------
>     >
>     >
>     >
>     > Bill, could you say something about what the metadata-storage here was?
>     > ESS/NL-SAS/3way replication?
>     >
>     > I just asked about this in the internal slack channel #scale-help today..
>     >
>     >
>     >
>     > -jf
>     >
>     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner at us.ibm.com_
>     > <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>>>>:
>     >
>     >     > "It has a significant performance penalty for small files in
>     large
>     >     > block size filesystems"
>     >
>     >     Aaron,
>     >
>     >     Below are mdtest results for a test we ran for CORAL - file
>     size was
>     >     32k.
>     >
>     >     We have not gone back and ran the test on a file system formatted
>     >     without > 32 subblocks. We'll do that at some point...
>     >
>     >     -Bill
>     >
>     >     -- started at 10/28/2017 17:51:38 --
>     >
>     >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
>     >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>     >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
>     -p 360
>     >     -u -y
>     >     Path: /ibm/fs2-16m-10
>     >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
>     >
>     >     228 tasks, 67239936 files
>     >
>     >     SUMMARY: (of 3 iterations)
>     >     Operation Max Min Mean Std Dev
>     >     --------- --- --- ---- -------
>     >     File creation : 51953.498 50558.517 51423.221 616.643
>     >     File stat : 0.000 0.000 0.000 0.000
>     >     File read : 0.000 0.000 0.000 0.000
>     >     File removal : 96746.376 92149.535 94658.774 1900.187
>     >     Tree creation : 1.588 0.070 0.599 0.700
>     >     Tree removal : 0.213 0.034 0.097 0.082
>     >
>     >     -- finished at 10/28/2017 19:51:54 --
>     >
>     >     Bill Hartner
>     >     IBM Systems
>     >     Scalable I/O Development
>     >     Austin, Texas_
>     >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>
>     <mailto:bhartner at us.ibm.com<mailto:bhartner at us.ibm.com>>>
>     >     home office 512-784-0980<tel:(512)%20784-0980> <tel:512-784-0980<tel:(512)%20784-0980>>
>     >
>     >     _
>     >     __gpfsug-discuss-bounces at spectrumscale.org_
>     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>> wrote on
>     >     11/29/2017 04:41:48 PM:
>     >
>     >     > From: Aaron Knister <_aaron.knister at gmail.com_
>     >     <mailto:aaron.knister at gmail.com<mailto:aaron.knister at gmail.com> <mailto:aaron.knister at gmail.com<mailto:aaron.knister at gmail.com>>>>
>     >
>     >
>     >     > To: gpfsug main discussion list
>     >     <_gpfsug-discuss at spectrumscale.org_
>     >     <mailto:gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
>     <mailto:gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>>>
>     >
>     >     > Date: 11/29/2017 04:42 PM
>     >
>     >
>     >     > Subject: Re: [gpfsug-discuss] Online data migration tool
>     >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>>
>     >
>     >     >
>     >
>     >     > Thanks, Nikhil. Most of that was consistent with my understnading,
>     >     > however I was under the impression that the >32 subblocks code is
>     >     > required to achieve the touted 50k file creates/second that Sven has
>     >     > talked about a bunch of times:
>     >     >
>     >     >
>     >
>      _http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>     <http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>     >     >
>     >
>      _http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
>     <http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>     >     > _Sven_Oehme_-_News_from_Research.pdf
>     >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>     >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>     >
>     >
>     >     > from those presentations regarding 32 subblocks:
>     >     >
>     >     > "It has a significant performance penalty for small files in large
>     >     > block size filesystems"
>     >
>     >     > although I'm not clear on the specific definition of "large". Many
>     >     > filesystems I encounter only have a 1M block size so it may not
>     >     > matter there, although that same presentation clearly shows the
>     >     > benefit of larger block sizes which is yet *another* thing for which
>     >     > a migration tool would be helpful.
>     >
>     >     > -Aaron
>     >     >
>     >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>     >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com<mailto:nikhilk at us.ibm.com>
>     <mailto:nikhilk at us.ibm.com<mailto:nikhilk at us.ibm.com>>>> wrote:
>     >
>     >     > Hi,
>     >     >
>     >     > I would like to clarify migration path to 5.0.0 from 4.X.X
>     clusters.
>     >     > For all Spectrum Scale clusters that are currently at 4.X.X,
>     it is
>     >     > possible to migrate to 5.0.0 with no offline data migration
>     and no
>     >     > need to move data. Once these clusters are at 5.0.0, they will
>     >     > benefit from the performance improvements, new features (such as
>     >     > file audit logging), and various enhancements that are
>     included in
>     >     5.0.0.
>     >     >
>     >     > That being said, there is one enhancement that will not be
>     applied
>     >     > to these clusters, and that is the increased number of
>     sub-blocks
>     >     > per block for small file allocation. This means that for file
>     >     > systems with a large block size and a lot of small files, the
>     >     > overall space utilization will be the same it currently is
>     in 4.X.X.
>     >     > Since file systems created at 4.X.X and earlier used a block
>     size
>     >     > that kept this allocation in mind, there should be very little
>     >     > impact on existing file systems.
>     >     >
>     >     > Outside of that one particular function, the remainder of the
>     >     > performance improvements, metadata improvements, updated
>     >     > compatibility, new functionality, and all of the other
>     enhancements
>     >     > will be immediately available to you once you complete the
>     upgrade
>     >     > to 5.0.0 -- with no need to reformat, move data, or take
>     your data
>     >     offline.
>     >     >
>     >     > I hope that clarifies things a little and makes the upgrade path
>     >     > more accessible.
>     >     >
>     >     > Please let me know if there are any other questions or concerns.
>     >     >
>     >     > Thank you,
>     >     > Nikhil Khandelwal
>     >     > Spectrum Scale Development
>     >     > Client Adoption
>     >     >
>     >     > _______________________________________________
>     >     > gpfsug-discuss mailing list
>     >     > gpfsug-discuss at _spectrumscale.org_
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>     >
>     >     > _______________________________________________
>     >     > gpfsug-discuss mailing list
>     >     > gpfsug-discuss at _spectrumscale.org_
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>     >
>     >     > _https://urldefense.proofpoint.com/v2/url?_
>     <https://urldefense.proofpoint.com/v2/url?_>
>     >     >
>     >     u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>     >     >
>     >     siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>     >
>     >     _______________________________________________
>     >     gpfsug-discuss mailing list
>     >     gpfsug-discuss at _spectrumscale.org_
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>     >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>     >
>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>     >     gpfsug-discuss mailing list
>     >     gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> <http://spectrumscale.org>
>     >
>      https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> <http://spectrumscale.org>
>     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>     >
>
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776<tel:(301)%20286-2776> <tel:%28301%29%20286-2776>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776<tel:(301)%20286-2776>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/62da1e91/attachment-0002.htm>

From taylorm at us.ibm.com  Mon Dec 18 20:27:42 2017
From: taylorm at us.ibm.com (Michael L Taylor)
Date: Mon, 18 Dec 2017 13:27:42 -0700
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
	Central
In-Reply-To: <mailman.3372.1513626221.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3372.1513626221.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <OFD183EA98.E6691217-ON002581FA.006FA7A4-072581FA.00706687@notes.na.collabserv.com>



Hi Bob,
Thanks for the note on 5.0.0
One correction however.... clusters can do a rolling upgrade to 5.0.0 from
any 4.2.x level (not just 4.2.3).




Today's Topics:

   1. FW: Spectrum Scale 5.0 now available on Fix Central
      (Oesterlin, Robert)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 19:43:35 +0000
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
		 Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53 at nuance.com>
Content-Type: text/plain; charset="utf-8"

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to
this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum
Scale V5.0 announcements' topic thread in the 'General Parallel File System
- Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=
> replied to the IBM Spectrum Scale V5.0 announcements<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=
> topic thread in the General Parallel File System - Announce (GPFS -
Announce)<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0


000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=>
 forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=
>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/157250d2/attachment-0002.htm>

From stockf at us.ibm.com  Mon Dec 18 21:10:55 2017
From: stockf at us.ibm.com (Frederick Stock)
Date: Mon, 18 Dec 2017 16:10:55 -0500
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
In-Reply-To: <32207.1513627728@turing-police.cc.vt.edu>
References: <32207.1513627728@turing-police.cc.vt.edu>
Message-ID: <OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF@notes.na.collabserv.com>

Yes the integrated protocols are the Samba and Ganesha that are bundled 
with Spectrum Scale.  These require the use of the CES component for 
monitoring the protocols.  If you do use them then you need to wait for a 
release of Spectrum Scale in which the integrated protocols are also 
supported on RHEL 7.4.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com



From:   valdis.kletnieks at vt.edu
To:     gpfsug-discuss at spectrumscale.org
Date:   12/18/2017 03:09 PM
Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux


that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on 
RHEL 7.4."

We don't use AFM or the install toolkit.  But we *do* make fairly heavy 
use
of mmces and nfs-ganesha - is that what they mean by "Integrated 
Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are 
doing
their upgrades - and going to 7.4 would be nice.  If there's a mine field 
there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e=






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/7c272339/attachment-0002.htm>

From erich at uw.edu  Mon Dec 18 21:46:02 2017
From: erich at uw.edu (Eric Horst)
Date: Mon, 18 Dec 2017 21:46:02 +0000
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
In-Reply-To: <OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF@notes.na.collabserv.com>
References: <32207.1513627728@turing-police.cc.vt.edu>
	<OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF@notes.na.collabserv.com>
Message-ID: <CAF-To-3w7FGHf_ks8cv4WZ=ZF42ZHVOZo4rbu2K8VTQtgAQUWg@mail.gmail.com>

Grr, this might explain why I experienced unhappiness when I tried to start
my long-delayed AFM based migration over the weekend. I had previously
tested AFM and found everything working, but 7.4 may have slipped in last
month. The AFM relationship seems to work but `mmafmctl premigrate`
commands fail. I would revert packages if I could figure out where the
issue lies.

-Eric

On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock <stockf at us.ibm.com> wrote:

> Yes the integrated protocols are the Samba and Ganesha that are bundled
> with Spectrum Scale.  These require the use of the CES component for
> monitoring the protocols.  If you do use them then you need to wait for a
> release of Spectrum Scale in which the integrated protocols are also
> supported on RHEL 7.4.
>
> Fred
> __________________________________________________
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821>
> stockf at us.ibm.com
>
>
>
> From:        valdis.kletnieks at vt.edu
> To:        gpfsug-discuss at spectrumscale.org
> Date:        12/18/2017 03:09 PM
> Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Currently, the IBM support matrix says:
>
> https://www.ibm.com/support/knowledgecenter/STXKQY/
> gpfsclustersfaq.html#linux
>
> that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:
>
> "AFM, Integrated Protocols, and Installation Toolkit are not supported on
> RHEL 7.4."
>
> We don't use AFM or the install toolkit.  But we *do* make fairly heavy use
> of mmces and nfs-ganesha - is that what they mean by "Integrated
> Protocols"?
>
> (We're looking at doing upgrades next month while our HPC clusters are
> doing
> their upgrades - and going to 7.4 would be nice.  If there's a mine field
> there, I need to
> make sure we stay at 7.3 - plus applicable non-7.4 updates)
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_
> iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=
> 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_
> I4NKvn8aFGXCqcJARCbu4AOE&e=
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171218/c01cc906/attachment-0002.htm>

From a.khiredine at meteo.dz  Tue Dec 19 08:08:03 2017
From: a.khiredine at meteo.dz (atmane khiredine)
Date: Tue, 19 Dec 2017 08:08:03 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35
In-Reply-To: <mailman.3382.1513633600.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3382.1513633600.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz>

IBM Spectrum Scale V5.0 not support RHEL 6.x

Only RHEL 7.1 or later

https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux


Atmane Khiredine
HPC System Administrator | Office National de la M?t?orologie
T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz

________________________________________
De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org]
Envoy? : lundi 18 d?cembre 2017 22:46
? : gpfsug-discuss at spectrumscale.org
Objet : gpfsug-discuss Digest, Vol 71, Issue 35

Send gpfsug-discuss mailing list submissions to
        gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
        gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
        gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Re: FW: Spectrum Scale 5.0 now available on Fix   Central
      (Michael L Taylor)
   2. Re: gpfs 4.2.3.5 and RHEL 7.4... (Frederick Stock)
   3. Re: gpfs 4.2.3.5 and RHEL 7.4... (Eric Horst)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 13:27:42 -0700
From: "Michael L Taylor" <taylorm at us.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on
        Fix     Central
Message-ID:
        <OFD183EA98.E6691217-ON002581FA.006FA7A4-072581FA.00706687 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"



Hi Bob,
Thanks for the note on 5.0.0
One correction however.... clusters can do a rolling upgrade to 5.0.0 from
any 4.2.x level (not just 4.2.3).




Today's Topics:

   1. FW: Spectrum Scale 5.0 now available on Fix Central
      (Oesterlin, Robert)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 19:43:35 +0000
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
                 Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53 at nuance.com>
Content-Type: text/plain; charset="utf-8"

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to
this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum
Scale V5.0 announcements' topic thread in the 'General Parallel File System
- Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=
> replied to the IBM Spectrum Scale V5.0 announcements<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=
> topic thread in the General Parallel File System - Announce (GPFS -
Announce)<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0


000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=>
 forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=
>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/157250d2/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 18 Dec 2017 16:10:55 -0500
From: "Frederick Stock" <stockf at us.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
        <OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

Yes the integrated protocols are the Samba and Ganesha that are bundled
with Spectrum Scale.  These require the use of the CES component for
monitoring the protocols.  If you do use them then you need to wait for a
release of Spectrum Scale in which the integrated protocols are also
supported on RHEL 7.4.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com



From:   valdis.kletnieks at vt.edu
To:     gpfsug-discuss at spectrumscale.org
Date:   12/18/2017 03:09 PM
Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux


that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on
RHEL 7.4."

We don't use AFM or the install toolkit.  But we *do* make fairly heavy
use
of mmces and nfs-ganesha - is that what they mean by "Integrated
Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are
doing
their upgrades - and going to 7.4 would be nice.  If there's a mine field
there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e=






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/7c272339/attachment-0001.html>

------------------------------

Message: 3
Date: Mon, 18 Dec 2017 21:46:02 +0000
From: Eric Horst <erich at uw.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
        <CAF-To-3w7FGHf_ks8cv4WZ=ZF42ZHVOZo4rbu2K8VTQtgAQUWg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Grr, this might explain why I experienced unhappiness when I tried to start
my long-delayed AFM based migration over the weekend. I had previously
tested AFM and found everything working, but 7.4 may have slipped in last
month. The AFM relationship seems to work but `mmafmctl premigrate`
commands fail. I would revert packages if I could figure out where the
issue lies.

-Eric

On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock <stockf at us.ibm.com> wrote:

> Yes the integrated protocols are the Samba and Ganesha that are bundled
> with Spectrum Scale.  These require the use of the CES component for
> monitoring the protocols.  If you do use them then you need to wait for a
> release of Spectrum Scale in which the integrated protocols are also
> supported on RHEL 7.4.
>
> Fred
> __________________________________________________
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821>
> stockf at us.ibm.com
>
>
>
> From:        valdis.kletnieks at vt.edu
> To:        gpfsug-discuss at spectrumscale.org
> Date:        12/18/2017 03:09 PM
> Subject:        [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Currently, the IBM support matrix says:
>
> https://www.ibm.com/support/knowledgecenter/STXKQY/
> gpfsclustersfaq.html#linux
>
> that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:
>
> "AFM, Integrated Protocols, and Installation Toolkit are not supported on
> RHEL 7.4."
>
> We don't use AFM or the install toolkit.  But we *do* make fairly heavy use
> of mmces and nfs-ganesha - is that what they mean by "Integrated
> Protocols"?
>
> (We're looking at doing upgrades next month while our HPC clusters are
> doing
> their upgrades - and going to 7.4 would be nice.  If there's a mine field
> there, I need to
> make sure we stay at 7.3 - plus applicable non-7.4 updates)
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_
> iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=
> 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_
> I4NKvn8aFGXCqcJARCbu4AOE&e=
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/c01cc906/attachment.html>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 71, Issue 35
**********************************************


From aaron.s.knister at nasa.gov  Tue Dec 19 08:31:37 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Tue, 19 Dec 2017 08:31:37 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35
In-Reply-To: <4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz>
References: <mailman.3382.1513633600.19082.gpfsug-discuss@spectrumscale.org>,
	<4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz>
Message-ID: <44E99F55-25CC-48DB-9AD6-E7D6794694DC@nasa.gov>

It?s not supported on SLES11 either.


IBM didn?t (that I saw) talk much about this publicly or give customers a chance to provide feedback about the decision. I know it was raised at the UG in NY and I recall a number of people saying it would be a significant issue for them (myself included) as is the fact they no longer support Debian with scale 5.0.

I?d raised the issue on the mailing list after the UG trying to start the discussion but IBM said they weren?t ready to talk about it publicly and I can only guess they had already set their sights and didn?t actually want feedback. This is actually pretty frustrating. I?m tempted to open an RFE but most of my RFEs either have been rejected or just sit idle so I?m not clear there?s a benefit.



On December 19, 2017 at 03:08:27 EST, atmane khiredine <a.khiredine at meteo.dz> wrote:
IBM Spectrum Scale V5.0 not support RHEL 6.x

Only RHEL 7.1 or later

https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux


Atmane Khiredine
HPC System Administrator | Office National de la M?t?orologie
T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz

________________________________________
De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org]
Envoy? : lundi 18 d?cembre 2017 22:46
? : gpfsug-discuss at spectrumscale.org
Objet : gpfsug-discuss Digest, Vol 71, Issue 35

Send gpfsug-discuss mailing list submissions to
gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

1. Re: FW: Spectrum Scale 5.0 now available on Fix Central
(Michael L Taylor)
2. Re: gpfs 4.2.3.5 and RHEL 7.4... (Frederick Stock)
3. Re: gpfs 4.2.3.5 and RHEL 7.4... (Eric Horst)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 13:27:42 -0700
From: "Michael L Taylor" <taylorm at us.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on
Fix Central
Message-ID:
<OFD183EA98.E6691217-ON002581FA.006FA7A4-072581FA.00706687 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"



Hi Bob,
Thanks for the note on 5.0.0
One correction however.... clusters can do a rolling upgrade to 5.0.0 from
any 4.2.x level (not just 4.2.3).




Today's Topics:

1. FW: Spectrum Scale 5.0 now available on Fix Central
(Oesterlin, Robert)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Dec 2017 19:43:35 +0000
From: "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
Central
Message-ID: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53 at nuance.com>
Content-Type: text/plain; charset="utf-8"

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to
this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Reply-To: "dW-notify at us.ibm.com" <dW-notify at us.ibm.com>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum
Scale V5.0 announcements' topic thread in the 'General Parallel File System
- Announce (GPFS - Announce)' forum.

[mid:4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75 at SDEB-EXC02.meteo.dz/forums.png] gpfs at us.ibm.com<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=
> replied to the IBM Spectrum Scale V5.0 announcements<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=
> topic thread in the General Parallel File System - Announce (GPFS -
Announce)<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0


000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=>
forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=
>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/157250d2/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 18 Dec 2017 16:10:55 -0500
From: "Frederick Stock" <stockf at us.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
<OF06A4AC60.B509ACDE-ON852581FA.00743B01-852581FA.00745AAF at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

Yes the integrated protocols are the Samba and Ganesha that are bundled
with Spectrum Scale. These require the use of the CES component for
monitoring the protocols. If you do use them then you need to wait for a
release of Spectrum Scale in which the integrated protocols are also
supported on RHEL 7.4.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com



From: valdis.kletnieks at vt.edu
To: gpfsug-discuss at spectrumscale.org
Date: 12/18/2017 03:09 PM
Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Sent by: gpfsug-discuss-bounces at spectrumscale.org



Currently, the IBM support matrix says:

https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux


that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:

"AFM, Integrated Protocols, and Installation Toolkit are not supported on
RHEL 7.4."

We don't use AFM or the install toolkit. But we *do* make fairly heavy
use
of mmces and nfs-ganesha - is that what they mean by "Integrated
Protocols"?

(We're looking at doing upgrades next month while our HPC clusters are
doing
their upgrades - and going to 7.4 would be nice. If there's a mine field
there, I need to
make sure we stay at 7.3 - plus applicable non-7.4 updates)
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e=






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/7c272339/attachment-0001.html>

------------------------------

Message: 3
Date: Mon, 18 Dec 2017 21:46:02 +0000
From: Eric Horst <erich at uw.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
Message-ID:
<CAF-To-3w7FGHf_ks8cv4WZ=ZF42ZHVOZo4rbu2K8VTQtgAQUWg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Grr, this might explain why I experienced unhappiness when I tried to start
my long-delayed AFM based migration over the weekend. I had previously
tested AFM and found everything working, but 7.4 may have slipped in last
month. The AFM relationship seems to work but `mmafmctl premigrate`
commands fail. I would revert packages if I could figure out where the
issue lies.

-Eric

On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock <stockf at us.ibm.com> wrote:

> Yes the integrated protocols are the Samba and Ganesha that are bundled
> with Spectrum Scale. These require the use of the CES component for
> monitoring the protocols. If you do use them then you need to wait for a
> release of Spectrum Scale in which the integrated protocols are also
> supported on RHEL 7.4.
>
> Fred
> __________________________________________________
> Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821>
> stockf at us.ibm.com
>
>
>
> From: valdis.kletnieks at vt.edu
> To: gpfsug-discuss at spectrumscale.org
> Date: 12/18/2017 03:09 PM
> Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4...
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Currently, the IBM support matrix says:
>
> https://www.ibm.com/support/knowledgecenter/STXKQY/
> gpfsclustersfaq.html#linux
>
> that 4.2.3.5 is supported on RHEL 7.4, but with a footnote:
>
> "AFM, Integrated Protocols, and Installation Toolkit are not supported on
> RHEL 7.4."
>
> We don't use AFM or the install toolkit. But we *do* make fairly heavy use
> of mmces and nfs-ganesha - is that what they mean by "Integrated
> Protocols"?
>
> (We're looking at doing upgrades next month while our HPC clusters are
> doing
> their upgrades - and going to 7.4 would be nice. If there's a mine field
> there, I need to
> make sure we stay at 7.3 - plus applicable non-7.4 updates)
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.
> org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_
> iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=
> 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_
> I4NKvn8aFGXCqcJARCbu4AOE&e=
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/c01cc906/attachment.html>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 71, Issue 35
**********************************************
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/78969e45/attachment-0002.htm>

From r.sobey at imperial.ac.uk  Tue Dec 19 09:06:08 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Tue, 19 Dec 2017 09:06:08 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
In-Reply-To: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
References: <E7371DDE-1AD6-4014-B2FD-610FEABE9E53@nuance.com>
Message-ID: <DB6PR0601MB2101BA78601B08F6F06469F8DF0F0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Hi Robert

Do you mean the minReleaseLevel from mmlsconfig or just making sure all the nodes are running 4.2.3?

Cheers!
Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert
Sent: 18 December 2017 19:44
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central

The Scale 5.0 fix level is now up on Fix Central.

You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Reply-To: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Date: Monday, December 18, 2017 at 1:27 PM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum.

[cid:forums.png] gpfs at us.ibm.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e=> replied to the IBM Spectrum Scale V5.0 announcements<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e=> topic thread in the General Parallel File System - Announce (GPFS - Announce)<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=> forum.

IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central:

http://www-933.ibm.com/support/fixcentral<https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e=>

This topic summarizes changes to the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library.

Summary of changes
for IBM Spectrum Scale version 5 release 0.0
as updated, April 2017

Changes to this release of the IBM Spectrum Scale licensed
program and the IBM Spectrum Scale library include the following:

Added DMPs for TIP events

       A topic is added listing the directed maintenance procedures for TIP events.
       The DMPs help users resolve issues caused due to TIP events.

AFM and AFM DR

       - Compression and snapshot ILM policy supported.
       - A general recommendation added for the Gateway node.
       - Configuration parameters added - afmMaxParallelRecoveries, afmAsyncOpWaitTimeout,
           afmSyncOpWaitTimeout, and afmRevalOpWaitTimeout.
       - Configuration parameters modified - afmRPO and afmHashVersion.

Authentication: Authentication packages

       Updated the authentication page to include packages specific to Ubuntu.

Authentication: AD-based authentication

       New information is added on NFS with server-side group
       lookup and Active Directory authentication.

Authentication: Primary group selection configurable for AD + RFC2307 based authentication

       Ability to choose primary group as set in "UNIX attributes" of a user on
       Active Directory introduced with AD + RFC2307 based authentication scheme.
       Earlier, Windows primary group was by default selected as the primary group.

Big data and analytics
       - The GPFS Ambari integration package is now called the IBM Spectrum
         Scale Ambari management pack (in short, management pack or MPack).
       - IBM Spectrum Scale Ambari management pack version 2.4.2.1 with
         HDFS Transparency version 2.7.3.1 supports BI 4.2/BI 4.2.5 IOP migration
         to HDP 2.6.2.
       - Supports the remote mount configuration in Ambari.
       - Supports the multiple file systems configuration. In management
         pack version 2.4.2.1, the current limit is two file systems.
       - The Short circuit write is supported for better performance.
       - In HDFS Transparency, the Ranger performance is enhanced.

Changes to IBM Spectrum Scale management API

   Added the following new commands:

       GET /perfmon/data
       GET /filesystems/{filesystemName}/afm/state
       DELETE /nodes/{name}
       POST /nodes
       GET /nodeclasses
       POST /nodeclasses
       DELETE /nodeclasses/{nodeclassName}
       GET /nodeclasses/{nodeclassName}
       PUT /nodeclasses/{nodeclassName}
       DELETE /jobs/jobId
       POST /filesystems/{filesystemName}/filesets/{filesetName}/psnaps
       DELETE /filesystems/{filesystemName}/filesets/{filesetName}/psnaps/{snapshotName}
       GET /thresholds
       GET /thresholds/{name}
       POST /thresholds
       DELETE /thresholds/{name}

IBM Spectrum Scale GUI changes

   - Added new Networks page to monitor the performance, configuration, and
      adapters of network configurations in the cluster. You can monitor the
      network performance with respect to the IP and RDMA interfaces used in the configuration.
    - Added new Monitoring > Thresholds page to create and monitor the threshold rules
      that are defined in the system.
    - Added Access > Remote Connections page to enable the GUI node of the
      local cluster to monitor the remote cluster by establishing a connection
      with the GUI node of the remote cluster.
    - Added Settings > Call Home page to configure call home. Configuring
      the call home feature helps the IBM? Support to monitor the system.
      Configuring call home also helps to reduce the response time of the
      IBM Support to resolve any issues.
      The diagnostic data that is downloaded through the Settings > Diagnostic Data
      can be uploaded to a problem management record (PMR) by using
      the call home feature in the backend. To upload the diagnostic data, right-click
      the relevant data set from the Previously Collected Diagnostic Data, and
      select Upload to PMR.
    - Added file system creation capabilities in GUI. Use the Files > File Systems > Create File System
      option to launch the Create File System wizard. In the Create File System wizard, you can specify
      the following details of the file system:
         - File system name
         - Storage pools
         - NSDs for the file systems
         - Failure groups
         - NSD order for data writes
         - Maximum number of Spectrum Scale clients
         - Maximum number of inodes of the root fileset
         - Whether to enable quota and scope for the quota definition
         - Whether to enable DMAPI
         - Mount point and automatic mount mode
    - Added the aggregation levels Access Point and Filesets and removed Account
      for the resource type Transparent Cloud Tiering in the Monitoring > Statisitics page.
    - The Files > Transparent Cloud Tiering page now displays the file systems
      and filesets that are mapped with the cloud service. It also shows the
      connection of such a container pair configuration to a cloud account and
      the corresponding CSAP that is configured for the cloud account.
    - Changes to capacity monitoring in the GUI
         - Moved the Capacity page from Monitoring to Files menu in the navigation
           and renamed the GUI page to User Capacity.
         - Only the file data user capacity can be viewed from the Files > User Capacity page.
           Removed the pools, filesets, file system capacity monitoring options from the Files > User Capacity page.
           You can monitor the capacity of these components from the respective GUI pages.
         - Replaced the GPFSPoolCap sensor with the GPFSPool sensor and separate data and metadata
           level capacity monitoring are introduced in the performance charts available in the

           Files > File Systems and Storage > Pools pages.
         - New GPFSPool-based data and metadata performance monitoring metrics are available
           for selection in the Files > Statistics > Edit > Capacity section.
           You need to select the aggregation level as Pool to view these metrics.
    - AFM monitoring changes in the GUI
         - Provides the number of AFM filesets and the corresponding export server maps.
           Each export map establishes a mapping between the gateway node and the NFS
           host name to allow parallel data transfers from cache to home.
         - By using the Request Access option available in the Files > Active File Management
           or Access > Remote Connection page in the GUI, you can now establish connection with remote clusters.
           After establishing the connection, you can monitor the following AFM and AFM DR
           configuration details across clusters:
              * On home and secondary, you can see the AFM relationships configuration,
                health status, and performance values of the Cache and Disaster Recovery grids.
              * On the Overview tab of the detailed view, the available home and secondary inodes are available.
              * On the Overview tab of the detailed view, the details such as NFS throughput,
                IOPs, and latency details are available, if the protocol is NFS.
    - New option to create AFM peer snapshots through GUI. Use the Create Peer Snapshot option
      in the Files > Snapshots page to create peer snapshots. You can view and delete these peer snapshots from
      the Snapshots page and also from the detailed view of the Files > Active File Management page.

Encryption: GSKit V8 improves cryptographic performance on IBM POWER8

   The IBM Global Security Kit (GSKit) Version 8 and later improves cryptographic performance
   on IBM POWER8 hardware. The version of GSKit that is shipped with IBM Spectrum Scale v5.0.0
   offers better performance on POWER8, compared with the versions shipped with earlier releases.

File compression: The lz4 library provides fast access to compressed data

   File compression supports the lz4 compression library. Lz4 is intended primarily for active data and
   favors read-access speed over maximized space saving.

File data: Block and subblock sizes improve I/O performance and reduce fragmentation

   The default block size is larger, 4 MiB instead of 256 KiB, and the sizes of subblocks relative to blocks
   s are smaller, for example, 8 KiB subblocks in a 4 MiB block. A larger block size improves the
   file system performance and a smaller subblock size reduces the amount of unused space. For many business
   applications, the default value of 4 MiB provides the best balance of improved performance
   and reduced fragmentation.

File encryption: AES-XTS encryption is faster on x86 in non-FIPS mode

    On x86 architecture in non-FIPS mode, file encryption with the AES algorithm in XTS mode
    is faster than it was.

File systems: File system rebalancing is faster

    Rebalancing is implemented by a lenient round-robin method that typically runs faster than the
    previously used method of strict round robin. The strict round robin method is available as an option.

installation toolkit changes

    - The installation toolkit has added support for the installation and the deployment
      of IBM Spectrum Scale in a cluster containing Elastic Storage Server (ESS).
    - The installation toolkit has added support for enabling and configuring call home.
    - The installation toolkit has added support for enabling and configuring file audit logging.
    - The installation toolkit has added support for the installation and the
      deployment of IBM Spectrum Scale on Ubuntu 16.04 LTS nodes.
    - The installation toolkit has added verification of passwordless SSH during
      prechecks before installation, deployment, or upgrade.
    - The installation toolkit has added support for cumulative object upgrade.

mmafmctl command

    The --outband parameter is deprecated.

mmcallhome command: Enhancements

    - Addition of -Y option
        * The -Y displays the command output in a parseable format with a colon (:) as a field delimiter.
    - Addition of --pmr option
        * The --pmr option allows you to upload data to existing PMRs using the
          mmcallhome run SendFile command.

mmchconfig command: Enhancements

    - Encrypted files can be copied into an LROC device
        * With the lrocEnableStoringClearText attribute, you can control whether file
          data from encrypted files, which is held in memory as cleartext, is
          copied into a local read-only cache (LROC) device.
    - InfiniBand addresses can be specified for RDMA transfers
        * In the verbsPorts attribute, you can specify InfiniBand addresses
          for RDMA transfers between an NSD client and server.

mmchnsd command: Change NSDs without unmounting the file system

    When you add or remove NSDs or do other operations with mmchnsd,
    you do not need to unmount the file system.

mmcrfs command: Enhancements

     - The default data block size is 4 MiB with an 8 KiB subblock size
         * If no block size is specified, a file system is created with a 4 MiB block size and an
           8 KiB subblock size. The minimum release level (minReleaseLevel) of the
           cluster must be 5.0.0 or greater when the file system is created.
     - The default log file size depends on block size and metadata size
         * If the block size is 512 KiB or larger and the metadata block size 256 KiB or larger,
           then the default log file size is 32 MiB. Otherwise, the default log file
           size is 4 MiB or the metadata block size, whichever is larger.
     - The default method for updating atime is relatime
         * If the minimum release level (minReleaseLevel) of the cluster is 5.0.0 or greater
           when the file system is created, the default method for updating atime is relatime.

mmdsh command: Several options are no longer supported

     The --ignoreSignal, -I, and -d options are no longer supported.
     Do not use these options unless instructed to by IBM support personnel.

mmfsck command: Display an interim status report at any time

     While a long-running instance of mmfsck is in progress, you can start another instance
     of mmfsck with the --status-report parameter to display current
     status information from all the nodes that are participating in the mmfsck run.

mmgetstate command: Display the unresponsive state

     The command returns the unresponsive state when the GPFS
     daemon is running but is not responding.

mmhealth command: Addition to measurement options

     Measurement options for filesystem, SMB node, and NFS node
     has been added to the mmhealth command.

mmkeyserv command: The simplified method supports certificate chains from a certificate authority.

     In the simplified method, with the --kmip-cert parameter, you can set up encryption with IBM?
     Security Key Lifecycle Manager (SKLM) as the key management server and with a certificate signed
     by a certificate authority (CA) on the KMIP port of the SKLM server.

mmnetverify command: Enhancements

     - Verify the network operation of nodes in a subnet
         * With the --subnets parameters, you can specify the subnet
           addresses of the nodes that you want to verify.
     - Verify that nodes can handle a new MTU size
         * With the ping-packet-size parameter, you can specify the size
           of the ICMP echo request packets that are sent between local node and the target
           node during the ping test.

mmtracectl command: Display the tracing status

     The --status parameter displays the tracing status of the specified nodes.

New feature for threshold monitoring

      Starting from IBM Spectrum Scale version 5.0.0, if multiple thresholds rules
      have overlapping entities for the same metrics, only one of the concurrent rules is
      made actively eligible.

NFS: Dynamic export changes

      You can dynamically change the export configuration without restarting the NFS service.

Object

      - Support for Ubuntu
      - Support for sudo wrapper for Object on Ubuntu
      - Support for cumulative upgrades from older versions
      - Object snap enhancement to contain keystore logs

Protocol support: Enhanced

      Protocol support is extended to Ubuntu 10.04

Setting up a system for storing crash files for Ubuntu

      A topic is added to describe how to set up a system for storing crash files for Ubuntu.

SMB: DFS redirects for SMB shares

      New option to configure DFS redirects for SMB shares.

SMB: SMB server upgrade changes

      Two events on CTDB version match/mismatch are added to the RAS events.

Sudo wrappers: Root-level processes can call administration commands directly

      Root-level background processes, such as cron and callback programs, can
      successfully call administration commands directly rather than through sudo
      when sudo wrappers are enabled.

Supported clients for NFS

      A topic is added listing the clients that are supported by NFS protocol.

Transparent cloud tiering

      - Support for multiple cloud storage accounts
      - Support for multiple file systems or filesets per node group
      - Enhanced support for large file systems provided by container spillover
      - Support for associating file sets with containers for enhanced granularity
      - Multiple URL and region support at the node level
      - Support for creating a cloud service separately for tiering and sharing operations.
      - Unique encryption key per cloud container
      - Support for remotely mounted clients.
      - Support for Amazon S3 regions requiring Sigv4 security support,
        including the US government cloud region.
      - Ability to enable or disable transparent recall for files for a given file
        system instantly, without having to rewrite a policy.
      - Support for backing up and restoring the Cloud services configuration in case of any disaster.
      - Support for backing up the Cloud services database to the cloud.
      - Support for restoring Transparent cloud tiering service on an identical backup cluster.
      - Support for checking the integrity of the Cloud services
        database after any system crash or outage.
      - Support for auditing events relating to each operation performed in Cloud services.

New commands

      mmaudit mmmsgqueue

Changed commands

      mmafmctl mmadddisk mmcallhome mmchattr mmchcluster mmchconfig mmchfs mmchnsd
      mmcloudgateway mmcrcluster mmcrfs mmdeldisk mmdsh mmfsck mmgetstate
      mmkeyserv mmnetverify mmnfs mmrestripefile mmrestripefs mmsmb mmtracectl
      mmuserauth

Deleted commands

      mmrest

New messages

      6027-1264, 6027-1757, 6027-2394, 6027-2395, 6027-2396, 6027-2397
      6027-2398, 6027-2399, 6027-2400, 6027-2401, 6027-3259, 6027-3408
      6027-3597, 6027-3598, 6027-3599, 6027-3600, 6027-3601, 6027-3602
      6027-3603, 6027-3604, 6027-3730, 6027-3921, 6027-3922, 6027-3923
      6027-3924, 6027-3925, 6027-3926, 6027-3927, 6027-3928
      6027-3929, 6027-3930, 6027-3931, 6027-4019

Changed messages

      6027-928


     Open this item<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab-234019da9b-2D248a-2D49ec-2D9503-2De612becd953e&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=Q6E0fg8sor2mORS3ugkurXkuV2ZUDqRd_C6_jpYXVVM&e=>


Posting Date:

Monday, December 18, 2017 at 2:26:38 PM EST



To unsubscribe or change settings, please go to your developerWorks community Settings.<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_news&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=5C4z0hhuaas4Ghs1g4YfFTCE0eyNYJVmLPYrGPCuE7Q&e=>
This is a notification sent from developerWorks community.

The information contained in this e-mail message is intended only for the use of the named recipients and is subject to the IBM Online Privacy Statement<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ibm.com_privacy_details_us_en_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=7gE9uoypP_yHb1jMMSjdCN_xyJ7UU2Ifrx1TchwMkQA&e=>. If you have received this e-mail in error, please contact the sender to report the error and then delete this message from your system.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/ced8c773/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 289 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/ced8c773/attachment-0002.png>

From Robert.Oesterlin at nuance.com  Tue Dec 19 12:38:12 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 19 Dec 2017 12:38:12 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
Message-ID: <73C34DA5-36DF-48C4-9E60-4DE75AA92405@nuance.com>

IBM corrected me a bit after I posted ? the cluster can be at any 4.2.X level (minreleaselevel)


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Sobey, Richard A" <r.sobey at imperial.ac.uk>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, December 19, 2017 at 3:07 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central

Hi Robert

Do you mean the minReleaseLevel from mmlsconfig or just making sure all the nodes are running 4.2.3?

Cheers!
Richard


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/c8a838b8/attachment-0002.htm>

From Robert.Oesterlin at nuance.com  Tue Dec 19 12:49:40 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 19 Dec 2017 12:49:40 +0000
Subject: [gpfsug-discuss] Scale 5.0 - Linux supported levels
Message-ID: <6844F00F-F07D-4FEF-B42D-CD32179A057E@nuance.com>

It?s a major problem for me that the dropped RH 6.X support with Scale 5.0. Now, I need to get my clients all to RH7 (which is problematic due to some older code needing the 6.X level) before I can take full advantage of Scale 5.0.


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]" <aaron.s.knister at nasa.gov>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, December 19, 2017 at 2:32 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35

It?s not supported on SLES11 either.


IBM didn?t (that I saw) talk much about this publicly or give customers a chance to provide feedback about the decision. I know it was raised at the UG in NY and I recall a number of people saying it would be a significant issue for them (myself included) as is the fact they no longer support Debian with scale 5.0.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/66dbccf7/attachment-0002.htm>

From duersch at us.ibm.com  Tue Dec 19 13:18:08 2017
From: duersch at us.ibm.com (Steve Duersch)
Date: Tue, 19 Dec 2017 08:18:08 -0500
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
In-Reply-To: <mailman.3390.1513674371.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3390.1513674371.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <OF252F90AC.16B92D2A-ON852581FB.00486631-852581FB.00491295@notes.na.collabserv.com>


As Mike Taylor pointed out in a previous post this was an incorrect
statement.
You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a
rolling upgrade.
The minReleaseLevel is not pertinent to a rolling upgrade.  The running
daemon is the important part.  So you can't have any 4.1.x nodes in your
cluster and do a rolling upgrade to 5.0.

Also, Aaron, as to the OS support.  This decision was not made without some
angst.  As I mentioned at the user group meeting in NYC...the key point is
that we would like to get to a more current compiler.  This will allow us
to take advantage of newer features and functions and hopefully make the
code better for customers.  SLES 12 has been around for over 2 years.

I hope this helps give some thinking behind the decision.


Steve Duersch
Spectrum Scale
845-433-7902
IBM Poughkeepsie, New York


> Today's Topics:
>
>    1. Re: Spectrum Scale 5.0 now available on Fix Central
>       (Sobey, Richard A)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 19 Dec 2017 09:06:08 +0000
> From: "Sobey, Richard A" <r.sobey at imperial.ac.uk>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix
>    Central
> Message-ID:
>
>
<DB6PR0601MB2101BA78601B08F6F06469F8DF0F0 at DB6PR0601MB2101.eurprd06.prod.outlook.com>

>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Robert
>
> Do you mean the minReleaseLevel from mmlsconfig or just making sure
> all the nodes are running 4.2.3?
>
> Cheers!
> Richard
>
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-
> discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert
> Sent: 18 December 2017 19:44
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix
Central
>
> The Scale 5.0 fix level is now up on Fix Central.
>
> You need to be at Scale 4.2.3 (cluster level) to do a rolling
> upgrade to this level.
>
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/07b56fa1/attachment-0002.htm>

From Robert.Oesterlin at nuance.com  Tue Dec 19 13:41:50 2017
From: Robert.Oesterlin at nuance.com (Oesterlin, Robert)
Date: Tue, 19 Dec 2017 13:41:50 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
Message-ID: <1A2E660D-A3F6-410E-9504-4B3BB5A34B09@nuance.com>

Thanks for clearing that up!


Bob Oesterlin
Sr Principal Storage Engineer, Nuance

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Steve Duersch <duersch at us.ibm.com>
Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, December 19, 2017 at 7:20 AM
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central

As Mike Taylor pointed out in a previous post this was an incorrect statement.
You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a rolling upgrade.
The minReleaseLevel is not pertinent to a rolling upgrade. The running daemon is the important part. So you can't have any 4.1.x nodes in your cluster and do a rolling upgrade to 5.0.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/ea1c3e28/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Tue Dec 19 13:55:34 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Tue, 19 Dec 2017 13:55:34 +0000
Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central
In-Reply-To: <OF252F90AC.16B92D2A-ON852581FB.00486631-852581FB.00491295@notes.na.collabserv.com>
References: <mailman.3390.1513674371.19082.gpfsug-discuss@spectrumscale.org>
	<OF252F90AC.16B92D2A-ON852581FB.00486631-852581FB.00491295@notes.na.collabserv.com>
Message-ID: <591BE98E-9C39-41D4-BAEA-DCFF53623E01@bham.ac.uk>

Maybe it would have been a good idea to make this clear in the ?What?s new in 5.0? slide decks used at SC. I don?t recall it being there. And the lack of forward public notification on this is not great, particularly for those not in NYC. Sure most of my clusters are on EL7 now, but I still have some nodes still running 6.x (notably some of our Spectrum Protect nodes which are not just systems we can reinstall).

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "duersch at us.ibm.com" <duersch at us.ibm.com>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Tuesday, 19 December 2017 at 13:19
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central


As Mike Taylor pointed out in a previous post this was an incorrect statement.
You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a rolling upgrade.
The minReleaseLevel is not pertinent to a rolling upgrade. The running daemon is the important part. So you can't have any 4.1.x nodes in your cluster and do a rolling upgrade to 5.0.

Also, Aaron, as to the OS support. This decision was not made without some angst. As I mentioned at the user group meeting in NYC...the key point is that we would like to get to a more current compiler. This will allow us to take advantage of newer features and functions and hopefully make the code better for customers. SLES 12 has been around for over 2 years.

I hope this helps give some thinking behind the decision.


Steve Duersch
Spectrum Scale
845-433-7902
IBM Poughkeepsie, New York


> Today's Topics:
>
>    1. Re: Spectrum Scale 5.0 now available on Fix Central
>       (Sobey, Richard A)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 19 Dec 2017 09:06:08 +0000
> From: "Sobey, Richard A" <r.sobey at imperial.ac.uk>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix
>    Central
> Message-ID:
>
> <DB6PR0601MB2101BA78601B08F6F06469F8DF0F0 at DB6PR0601MB2101.eurprd06.prod.outlook.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Robert
>
> Do you mean the minReleaseLevel from mmlsconfig or just making sure
> all the nodes are running 4.2.3?
>
> Cheers!
> Richard
>
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-
> discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert
> Sent: 18 December 2017 19:44
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central
>
> The Scale 5.0 fix level is now up on Fix Central.
>
> You need to be at Scale 4.2.3 (cluster level) to do a rolling
> upgrade to this level.
>
>
> Bob Oesterlin
> Sr Principal Storage Engineer, Nuance
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/3894c70b/attachment-0002.htm>

From Mark.Bush at siriuscom.com  Tue Dec 19 14:29:21 2017
From: Mark.Bush at siriuscom.com (Mark Bush)
Date: Tue, 19 Dec 2017 14:29:21 +0000
Subject: [gpfsug-discuss] pmcollector and NSD perf
Message-ID: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>

I've noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in the GUI on the monitoring screen with the default view the NSD Server Throughput graph shows "Performance Collector did not return any data".  I've seen that in other items (SMB before for example) but never for NSD.  Is there something that must be enabled in the zimon sensor or collector config file to grab this or is this a bug?


Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/d814d122/attachment-0002.htm>

From rohwedder at de.ibm.com  Tue Dec 19 15:23:55 2017
From: rohwedder at de.ibm.com (Markus Rohwedder)
Date: Tue, 19 Dec 2017 16:23:55 +0100
Subject: [gpfsug-discuss] pmcollector and NSD perf
In-Reply-To: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
References: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
Message-ID: <OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>

Hello Mark,

the NSD sensor is GPFSNSDDisk.

Some things to check:

1. Is the sensor activated?
In a GPFS managed sensor config you should be able to see something like
this when you call mmperfmon config show:

{
        name = "GPFSNSDDisk"
        period = 10
        restrict = "nsdNodes"
},

2. Perfmon designation
The NSD server nodes should have the perfmon designation.


[root at cache-41 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfsgui-cluster-4.localnet.com
  GPFS cluster id:           10583479681538672379
  GPFS UID domain:           localnet.com
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

 Node  Daemon node name       IP address   Admin node name
Designation
------------------------------------------------------------------------------
   1   cache-41.localnet.com  10.0.100.41  cache-41.localnet.com
quorum-perfmon
   2   cache-42.localnet.com  10.0.100.42  cache-42.localnet.com
quorum-gateway-perfmon
   3   cache-43.localnet.com  10.0.100.43  cache-43.localnet.com
gateway-perfmon

3. Direct Disk writes?
One reason why there may be no data on your system is if you are not using
the NSD protocol,
meaning the clients can directly write to disk as in a SAN environment.
In this case the sensor does not catch the transactions.

4. Cross cluster mount
Or maybe you are using a cross cluster mount.



Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

Spectrum Scale GUI Development
                                                                              
                                                                              
                                                                              
                                                                              
                                                                              
 Phone:            +49 7034 6430190      IBM Deutschland                      
                                                                              
 E-Mail:           rohwedder at de.ibm.com  Am Weiher 24                         
                                                                              
                                         65451 Kelsterbach                    
                                                                              
                                         Germany                              
                                                                              
                                                                              
                                                                              
                                                                              
                                                                              
 IBM Deutschland                                                              
 Research &                                                                   
 Development                                                                  
 GmbH /                                                                       
 Vorsitzender des                                                             
 Aufsichtsrats:                                                               
 Martina K?deritz                                                             
 Gesch?ftsf?hrung:                                                            
 Dirk Wittkopp                                                                
 Sitz der                                                                     
 Gesellschaft:                                                                
 B?blingen /                                                                  
 Registergericht:                                                             
 Amtsgericht                                                                  
 Stuttgart, HRB                                                               
 243294                                                                       
                                                                              





From:	Mark Bush <Mark.Bush at siriuscom.com>
To:	"gpfsug-discuss at spectrumscale.org"
            <gpfsug-discuss at spectrumscale.org>
Date:	12/19/2017 03:30 PM
Subject:	[gpfsug-discuss] pmcollector and NSD perf
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



I?ve noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in
the GUI on the monitoring screen with the default view the NSD Server
Throughput graph shows ?Performance Collector did not return any data?.
I?ve seen that in other items (SMB before for example) but never for NSD.
Is there something that must be enabled in the zimon sensor or collector
config file to grab this or is this a bug?


Mark_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=a6GCq72qeADy6hsfA-24PmWHU06W5z2xqx9tKIJ8qJ4&s=OQccy8ikWB-ByYgLsJFgI8szDs1ZrwnsaFrLCwTfTwI&e=



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment-0006.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1A013420.gif
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment-0007.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/5040daa7/attachment-0008.gif>

From Mark.Bush at siriuscom.com  Tue Dec 19 15:49:59 2017
From: Mark.Bush at siriuscom.com (Mark Bush)
Date: Tue, 19 Dec 2017 15:49:59 +0000
Subject: [gpfsug-discuss] pmcollector and NSD perf
In-Reply-To: <OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>
References: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
	<OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>
Message-ID: <DM5PR05MB3577119DC6FE73462E83BC72E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>

It appears number 3 on your list is the case.  My nodes are all SAN connected and until I get separate CES nodes no NSD is necessary (currently run CES on the NSD servers ? just for a test cluster).

Mark

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Markus Rohwedder
Sent: Tuesday, December 19, 2017 9:24 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] pmcollector and NSD perf


Hello Mark,

the NSD sensor is GPFSNSDDisk.

Some things to check:

1. Is the sensor activated?
In a GPFS managed sensor config you should be able to see something like this when you call mmperfmon config show:

{
name = "GPFSNSDDisk"
period = 10
restrict = "nsdNodes"
},

2. Perfmon designation
The NSD server nodes should have the perfmon designation.


[root at cache-41 ~]# mmlscluster

GPFS cluster information
========================
GPFS cluster name: gpfsgui-cluster-4.localnet.com
GPFS cluster id: 10583479681538672379
GPFS UID domain: localnet.com
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCR

Node Daemon node name IP address Admin node name Designation
------------------------------------------------------------------------------
1 cache-41.localnet.com 10.0.100.41 cache-41.localnet.com quorum-perfmon
2 cache-42.localnet.com 10.0.100.42 cache-42.localnet.com quorum-gateway-perfmon
3 cache-43.localnet.com 10.0.100.43 cache-43.localnet.com gateway-perfmon

3. Direct Disk writes?
One reason why there may be no data on your system is if you are not using the NSD protocol,
meaning the clients can directly write to disk as in a SAN environment.
In this case the sensor does not catch the transactions.

4. Cross cluster mount
Or maybe you are using a cross cluster mount.



Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

Spectrum Scale GUI Development

________________________________



Phone:

+49 7034 6430190

IBM Deutschland

[cid:image002.gif at 01D378AE.BAB5B6F0]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________



IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina K?deritz
Gesch?ftsf?hrung: Dirk Wittkopp
Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294



[Inactive hide details for Mark Bush ---12/19/2017 03:30:14 PM---I've noticed this in my test cluster both in 4.2.3.4 and 5.0.0.]Mark Bush ---12/19/2017 03:30:14 PM---I've noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in the GUI on the monitoring s

From: Mark Bush <Mark.Bush at siriuscom.com<mailto:Mark.Bush at siriuscom.com>>
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: 12/19/2017 03:30 PM
Subject: [gpfsug-discuss] pmcollector and NSD perf
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



I?ve noticed this in my test cluster both in 4.2.3.4 and 5.0.0.0 that in the GUI on the monitoring screen with the default view the NSD Server Throughput graph shows ?Performance Collector did not return any data?. I?ve seen that in other items (SMB before for example) but never for NSD. Is there something that must be enabled in the zimon sensor or collector config file to grab this or is this a bug?


Mark_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=a6GCq72qeADy6hsfA-24PmWHU06W5z2xqx9tKIJ8qJ4&s=OQccy8ikWB-ByYgLsJFgI8szDs1ZrwnsaFrLCwTfTwI&e=


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 166 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.gif
Type: image/gif
Size: 1851 bytes
Desc: image002.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment-0004.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.gif
Type: image/gif
Size: 105 bytes
Desc: image003.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/194aceb4/attachment-0005.gif>

From hoov at us.ibm.com  Tue Dec 19 18:40:56 2017
From: hoov at us.ibm.com (Theodore Hoover Jr)
Date: Tue, 19 Dec 2017 18:40:56 +0000
Subject: [gpfsug-discuss] Spectrum Scale on AWS - Join Sponsor User Program
Message-ID: <OFA4E895DE.FCD6420A-ON002581FB.00665ACE-002581FB.00669FEE@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/98449b9b/attachment-0002.htm>

From oehmes at gmail.com  Tue Dec 19 20:20:29 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Tue, 19 Dec 2017 20:20:29 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
	<CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
	<6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>
Message-ID: <CALssuR3wX4_6T668BEupSvdt75pxgZuBSj5rzbVVy=tWWoFhxQ@mail.gmail.com>

Hi,

the zero padding was never promoted into a GA stream, it was an experiment
to proof we are on the right track when we eliminate the overhead from
client to NSD Server, but also showed that alone is not good enough. the
work for the client is the same compared to the >32 subblocks, but the NSD
Server has more work as it can't pack as many subblocks and therefore files
into larger blocks, so you need to do more writes to store the same number
of files.
thats why there is the additional substantial improvement  when we then
went to >32 subblocks.

sven

On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:

> Thanks Sven! That makes sense to me and is what I thought was the case
> which is why I was confused when I saw the reply to the thread that said
> the >32 subblocks code had no performance impact.
>
> A couple more question for you? in your presentation there?s a benchmark
> that shows the file create performance without the zero padding. Since you
> mention this is done for security reasons was that feature ever promoted to
> a GA Scale release? I?m also wondering if you could explain the performance
> difference between the no zero padding code and the > 32 subblock code
> since given your the example of 32K files and 16MB block size I figure both
> cases ought to write the same amount to disk.
>
> Thanks!
>
> -Aaron
>
>
>
>
>
> On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com> wrote:
>
> i thought i answered that already, but maybe i just thought about
> answering it and then forgot about it :-D
>
> so yes more than 32 subblocks per block significant increase the
> performance of filesystems with small files, for the sake of the argument
> let's say 32k in a large block filesystem again for sake of argument say
> 16MB.
>
> you probably ask why ?
>
> if you create a file and write 32k into it in a pre 5.0.0 Version 16 MB
> filesystem your client actually doesn't write 32k to the NSD Server, it
> writes 512k, because thats the subblock size and we need to write the full
> subblock (for security reasons). so first you waste significant memory on
> the client to cache that zero padding, you waste network bandwidth and you
> waste NSD Server cache because you store it there too. this means you
> overrun the cache more quickly, means you start doing read/modify writes
> earlier on all your nice large raid tracks... i guess you get the story by
> now.
>
> in fact,  if you have a good raid code that can drive really a lot of
> bandwidth out of individual drives like a GNR system you get more
> performance for small file writes as larger your blocksize is, because we
> can 'pack' more files into larger i/os and therefore turn a small file
> create workload into a bandwidth workload, essentially exactly what we did
> and i demonstrated in the CORAL presentation .
>
> hope that makes this crystal clear now .
>
> sven
>
>
>
> On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister <aaron.s.knister at nasa.gov>
> wrote:
>
> Thanks, Alex. I'm all too familiar with the trade offs between large
> blocks and small files and we do use pretty robust SSD storage for our
> metadata. We support a wide range of workloads and we have some folks
> with many small (<1M) files and other folks with many large (>256MB) files.
>
> My point in this thread is that IBM has said over and over again in
> presentations that there is a significant performance gain with the >32
> subblocks code on filesystems with large block sizes (although to your
> point I'm not clear on exactly what large means since I didn't define
> large in this context). Therefore given that the >32 subblock code gives
> a significant performance gain one could reasonably assume that having a
> filesystem with >32 subblocks is required to see this gain (rather than
> just running the >32 subblocks code on an fs w/o > 32 subblocks).
>
> This lead me to ask about a migration tool because in my mind if there's
> a performance gain from having >32 subblocks on the FS I'd like that
> feature and having to manually copy 10's of PB to new hardware to get
> this performance boost is unacceptable. However, IBM can't seem to make
> up their mind about whether or not the >32 subblocks code *actually*
> provides a performance increase. This seems like a pretty
> straightforward question.
>
> -Aaron
>
> On 12/15/17 3:48 PM, Alex Chekholko wrote:
> > Hey Aaron,
> >
> > Can you define your sizes for "large blocks" and "small files"?  If you
> > dial one up and the other down, your performance will be worse.  And in
> > any case it's a pathological corner case so it shouldn't matter much for
> > your workflow, unless you've designed your system with the wrong values.
> >
> > For example, for bioinformatics workloads, I prefer to use 256KB
> > filesystem block size, and I'd consider 4MB+ to be "large block size",
> > which would make the filesystem obviously unsuitable for processing
> > millions of 8KB files.
> >
> > You can make a histogram of file sizes in your existing filesystems and
> > then make your subblock size (1/32 of block size) on the smaller end of
> > that.   Also definitely use the "small file in inode" feature and put
> > your metadata on SSD.
> >
> > Regards,
> > Alex
> >
> > On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
> > <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
> >
> >     Thanks, Bill.
> >
> >     I still don't feel like I've got an clear answer from IBM and frankly
> >     the core issue of a lack of migration tool was totally dodged.
> >
> >     Again in Sven's presentation from SSUG @ SC17
> >     (
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf <
> http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
> >     he mentions "It has a significant performance penalty for small
> files in
> >     large block size filesystems" and the demonstrates that with several
> >     mdtest runs (which show the effect with and without the >32
> >     subblocks code):
> >
> >
> >     4.2.1 base code - SUMMARY: (of 3 iterations)
> >     File creation : Mean = 2237.644
> >
> >     zero-end-of-file-padding (4.2.2 + ifdef for zero padding):  SUMMARY:
> (of
> >     3 iterations)
> >     File creation : Mean = 12866.842
> >
> >     more sub blocks per block (4.2.2 + morethan32subblock code):
> >     File creation : Mean = 40316.721
> >
> >     Can someone (ideally Sven) give me a straight answer as to whether or
> >     not the > 32 subblock code actually makes a performance difference
> for
> >     small files in large block filesystems? And if not, help me
> understand
> >     why his slides and provided benchmark data have consistently
> indicated
> >     it does?
> >
> >     -Aaron
> >
> >     On 12/1/17 11:44 AM, Bill Hartner wrote:
> >     > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the
> >     > results @ user group meeting.
> >     >
> >     > -Bill
> >     >
> >     > Bill Hartner
> >     > IBM Systems
> >     > Scalable I/O Development
> >     > Austin, Texas
> >     > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
> >     > home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >
> >     > Inactive hide details for Jan-Frode Myklebust ---12/01/2017
> 06:53:44
> >     > AM---Bill, could you say something about what the metadataJan-Frode
> >     > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something
> >     > about what the metadata-storage here was? ESS/NL-SAS/3way
> replication?
> >     >
> >     > From: Jan-Frode Myklebust <janfrode at tanso.net <mailto:
> janfrode at tanso.net>>
> >     > To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>
> >     > Date: 12/01/2017 06:53 AM
> >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     > Sent by: gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> >     >
> >     >
> >
>  ------------------------------------------------------------------------
> >     >
> >     >
> >     >
> >     > Bill, could you say something about what the metadata-storage here
> was?
> >     > ESS/NL-SAS/3way replication?
> >     >
> >     > I just asked about this in the internal slack channel #scale-help
> today..
> >     >
> >     >
> >     >
> >     > -jf
> >     >
> >     > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
> <_bhartner at us.ibm.com_
> >     > <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>:
> >     >
> >     >     > "It has a significant performance penalty for small files in
> >     large
> >     >     > block size filesystems"
> >     >
> >     >     Aaron,
> >     >
> >     >     Below are mdtest results for a test we ran for CORAL - file
> >     size was
> >     >     32k.
> >     >
> >     >     We have not gone back and ran the test on a file system
> formatted
> >     >     without > 32 subblocks. We'll do that at some point...
> >     >
> >     >     -Bill
> >     >
> >     >     -- started at 10/28/2017 17:51:38 --
> >     >
> >     >     mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)
> >     >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
> >     >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r
> >     -p 360
> >     >     -u -y
> >     >     Path: /ibm/fs2-16m-10
> >     >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%
> >     >
> >     >     228 tasks, 67239936 files
> >     >
> >     >     SUMMARY: (of 3 iterations)
> >     >     Operation Max Min Mean Std Dev
> >     >     --------- --- --- ---- -------
> >     >     File creation : 51953.498 50558.517 51423.221 616.643
> >     >     File stat : 0.000 0.000 0.000 0.000
> >     >     File read : 0.000 0.000 0.000 0.000
> >     >     File removal : 96746.376 92149.535 94658.774 1900.187
> >     >     Tree creation : 1.588 0.070 0.599 0.700
> >     >     Tree removal : 0.213 0.034 0.097 0.082
> >     >
> >     >     -- finished at 10/28/2017 19:51:54 --
> >     >
> >     >     Bill Hartner
> >     >     IBM Systems
> >     >     Scalable I/O Development
> >     >     Austin, Texas_
> >     >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
> >     <mailto:bhartner at us.ibm.com>>
> >     >     home office 512-784-0980 <(512)%20784-0980> <tel:512-784-0980
> <(512)%20784-0980>>
> >     >
> >     >     _
> >     >     __gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>> wrote on
> >     >     11/29/2017 04:41:48 PM:
> >     >
> >     >     > From: Aaron Knister <_aaron.knister at gmail.com_
> >     >     <mailto:aaron.knister at gmail.com <mailto:
> aaron.knister at gmail.com>>>
> >     >
> >     >
> >     >     > To: gpfsug main discussion list
> >     >     <_gpfsug-discuss at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss at spectrumscale.org
> >     <mailto:gpfsug-discuss at spectrumscale.org>>>
> >     >
> >     >     > Date: 11/29/2017 04:42 PM
> >     >
> >     >
> >     >     > Subject: Re: [gpfsug-discuss] Online data migration tool
> >     >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
> >     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> >     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
> >     >
> >     >     >
> >     >
> >     >     > Thanks, Nikhil. Most of that was consistent with my
> understnading,
> >     >     > however I was under the impression that the >32 subblocks
> code is
> >     >     > required to achieve the touted 50k file creates/second that
> Sven has
> >     >     > talked about a bunch of times:
> >     >     >
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >     <
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
> >
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >>
> >     >     >
> >     >
> >      _
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
> >     <
> http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >>
> >     >     > _Sven_Oehme_-_News_from_Research.pdf
> >     >     > _http://files.gpfsug.org/presentations/2016/SC16/12_-_
> >     <http://files.gpfsug.org/presentations/2016/SC16/12_-_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >>
> >     >     > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
> >     >
> >     >
> >     >     > from those presentations regarding 32 subblocks:
> >     >     >
> >     >     > "It has a significant performance penalty for small files in
> large
> >     >     > block size filesystems"
> >     >
> >     >     > although I'm not clear on the specific definition of
> "large". Many
> >     >     > filesystems I encounter only have a 1M block size so it may
> not
> >     >     > matter there, although that same presentation clearly shows
> the
> >     >     > benefit of larger block sizes which is yet *another* thing
> for which
> >     >     > a migration tool would be helpful.
> >     >
> >     >     > -Aaron
> >     >     >
> >     >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
> >     >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
> >     <mailto:nikhilk at us.ibm.com>>> wrote:
> >     >
> >     >     > Hi,
> >     >     >
> >     >     > I would like to clarify migration path to 5.0.0 from 4.X.X
> >     clusters.
> >     >     > For all Spectrum Scale clusters that are currently at 4.X.X,
> >     it is
> >     >     > possible to migrate to 5.0.0 with no offline data migration
> >     and no
> >     >     > need to move data. Once these clusters are at 5.0.0, they
> will
> >     >     > benefit from the performance improvements, new features
> (such as
> >     >     > file audit logging), and various enhancements that are
> >     included in
> >     >     5.0.0.
> >     >     >
> >     >     > That being said, there is one enhancement that will not be
> >     applied
> >     >     > to these clusters, and that is the increased number of
> >     sub-blocks
> >     >     > per block for small file allocation. This means that for file
> >     >     > systems with a large block size and a lot of small files, the
> >     >     > overall space utilization will be the same it currently is
> >     in 4.X.X.
> >     >     > Since file systems created at 4.X.X and earlier used a block
> >     size
> >     >     > that kept this allocation in mind, there should be very
> little
> >     >     > impact on existing file systems.
> >     >     >
> >     >     > Outside of that one particular function, the remainder of the
> >     >     > performance improvements, metadata improvements, updated
> >     >     > compatibility, new functionality, and all of the other
> >     enhancements
> >     >     > will be immediately available to you once you complete the
> >     upgrade
> >     >     > to 5.0.0 -- with no need to reformat, move data, or take
> >     your data
> >     >     offline.
> >     >     >
> >     >     > I hope that clarifies things a little and makes the upgrade
> path
> >     >     > more accessible.
> >     >     >
> >     >     > Please let me know if there are any other questions or
> concerns.
> >     >     >
> >     >     > Thank you,
> >     >     > Nikhil Khandelwal
> >     >     > Spectrum Scale Development
> >     >     > Client Adoption
> >     >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >     > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>
> >     >
> >     >     > _______________________________________________
> >     >     > gpfsug-discuss mailing list
> >     >     > gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
> >     >
> >     >     > _https://urldefense.proofpoint.com/v2/url?_
> >     <https://urldefense.proofpoint.com/v2/url?_>
> >     >     >
> >     >
>  u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
> >     >     >
> >     >
>  siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
> >     >
> >     >     _______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at _spectrumscale.org_
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>_
> >     >     __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
> >     >
> >      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>_______________________________________________
> >     >     gpfsug-discuss mailing list
> >     >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >     <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >
> >     >
> >     >
> >     >
> >     >
> >     > _______________________________________________
> >     > gpfsug-discuss mailing list
> >     > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >     >
> >
> >     --
> >     Aaron Knister
> >     NASA Center for Climate Simulation (Code 606.2)
> >     Goddard Space Flight Center
> >     (301) 286-2776 <tel:%28301%29%20286-2776>
> >     _______________________________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/92903846/attachment-0002.htm>

From aaron.s.knister at nasa.gov  Tue Dec 19 21:32:00 2017
From: aaron.s.knister at nasa.gov (Aaron Knister)
Date: Tue, 19 Dec 2017 16:32:00 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CALssuR3wX4_6T668BEupSvdt75pxgZuBSj5rzbVVy=tWWoFhxQ@mail.gmail.com>
References: <OFC51A0B73.179827C7-ON072581E7.0066EA0C-072581E7.00691EA7@notes.na.collabserv.com>
	<OF08C0FACC.77794356-ON002581E7.007CBA22@LocalDomain>
	<OFF0AD18CF.B3BED5B9-ON002581E9.004593B5-862581E9.00460146@notes.na.collabserv.com>
	<OF7F9F76D1.15BF9B7A-ON002581E9.0046D758@LocalDomain>
	<OF0A98233F.9F70598E-ON002581E9.005BD63E-862581E9.005BF449@notes.na.collabserv.com>
	<fbec7133-2f29-450d-dd6c-27211ae56b88@nasa.gov>
	<CANcy_PaWzGHW4ouBaGWJz+piOuEboM8k-726y2nKbiMKQxTWMw@mail.gmail.com>
	<03e3cf87-0e65-4c3f-7e27-9f98b445d9c3@nasa.gov>
	<CALssuR304f9_Gc1yCy6hcBRVL=ijOkkDsJR-XkMX-n2ewd-d2Q@mail.gmail.com>
	<6ADC0293-493D-497A-934E-530807ECCC4C@nasa.gov>
	<CALssuR3wX4_6T668BEupSvdt75pxgZuBSj5rzbVVy=tWWoFhxQ@mail.gmail.com>
Message-ID: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>

Thanks, Sven. Understood!

On 12/19/17 3:20 PM, Sven Oehme wrote:
> Hi,
> 
> the zero padding was never promoted into a GA stream, it was an
> experiment to proof we are on the right track when we eliminate the
> overhead from client to NSD Server, but also showed that alone is not
> good enough. the work for the client is the same compared to the >32
> subblocks, but the NSD Server has more work as it can't pack as many
> subblocks and therefore files into larger blocks, so you need to do more
> writes to store the same number of files.?
> thats why there is the additional substantial improvement? when we then
> went to >32 subblocks.?
> 
> sven
> 
> On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
> SCIENCE CORP] <aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>> wrote:
> 
>     Thanks Sven! That makes sense to me and is what I thought was the
>     case which is why I was confused when I saw the reply to the thread
>     that said the >32 subblocks code had no performance impact.?
> 
>     A couple more question for you? in your presentation there?s a
>     benchmark that shows the file create performance without the zero
>     padding. Since you mention this is done for security reasons was
>     that feature ever promoted to a GA Scale release? I?m also wondering
>     if you could explain the performance difference between the no zero
>     padding code and the > 32 subblock code since given your the example
>     of 32K files and 16MB block size I figure both cases ought to write
>     the same amount to disk.?
> 
>     Thanks!
> 
>     -Aaron
> 
> 
> 
> 
> 
>     On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com
>     <mailto:oehmes at gmail.com>> wrote:
>>     i thought i answered that already, but maybe i just thought about
>>     answering it and then forgot about it :-D
>>
>>     so yes more than 32 subblocks per block significant increase the
>>     performance of filesystems with small files, for the sake of the
>>     argument let's say 32k in a large block filesystem again for sake
>>     of argument say 16MB.?
>>
>>     you probably ask why ??
>>
>>     if you create a file and write 32k into it in a pre 5.0.0 Version
>>     16 MB filesystem your client actually doesn't write 32k to the NSD
>>     Server, it writes 512k, because thats the subblock size and we
>>     need to write the full subblock (for security reasons). so first
>>     you waste significant memory on the client to cache that zero
>>     padding, you waste network bandwidth and you waste NSD Server
>>     cache because you store it there too. this means you overrun the
>>     cache more quickly, means you start doing read/modify writes
>>     earlier on all your nice large raid tracks... i guess you get the
>>     story by now.?
>>
>>     in fact,? if you have a good raid code that can drive really a lot
>>     of bandwidth out of individual drives like a GNR system you get
>>     more performance for small file writes as larger your blocksize
>>     is, because we can 'pack' more files into larger i/os and
>>     therefore turn a small file create workload into a bandwidth
>>     workload, essentially exactly what we did and i demonstrated in
>>     the CORAL presentation .?
>>
>>     hope that makes this crystal clear now .?
>>
>>     sven
>>
>>
>>
>>     On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister
>>     <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
>>
>>         Thanks, Alex. I'm all too familiar with the trade offs between
>>         large
>>         blocks and small files and we do use pretty robust SSD storage
>>         for our
>>         metadata. We support a wide range of workloads and we have
>>         some folks
>>         with many small (<1M) files and other folks with many large
>>         (>256MB) files.
>>
>>         My point in this thread is that IBM has said over and over
>>         again in
>>         presentations that there is a significant performance gain
>>         with the >32
>>         subblocks code on filesystems with large block sizes (although
>>         to your
>>         point I'm not clear on exactly what large means since I didn't
>>         define
>>         large in this context). Therefore given that the >32 subblock
>>         code gives
>>         a significant performance gain one could reasonably assume
>>         that having a
>>         filesystem with >32 subblocks is required to see this gain
>>         (rather than
>>         just running the >32 subblocks code on an fs w/o > 32 subblocks).
>>
>>         This lead me to ask about a migration tool because in my mind
>>         if there's
>>         a performance gain from having >32 subblocks on the FS I'd
>>         like that
>>         feature and having to manually copy 10's of PB to new hardware
>>         to get
>>         this performance boost is unacceptable. However, IBM can't
>>         seem to make
>>         up their mind about whether or not the >32 subblocks code
>>         *actually*
>>         provides a performance increase. This seems like a pretty
>>         straightforward question.
>>
>>         -Aaron
>>
>>         On 12/15/17 3:48 PM, Alex Chekholko wrote:
>>         > Hey Aaron,
>>         >
>>         > Can you define your sizes for "large blocks" and "small
>>         files"?? If you
>>         > dial one up and the other down, your performance will be
>>         worse.? And in
>>         > any case it's a pathological corner case so it shouldn't
>>         matter much for
>>         > your workflow, unless you've designed your system with the
>>         wrong values.
>>         >
>>         > For example, for bioinformatics workloads, I prefer to use 256KB
>>         > filesystem block size, and I'd consider 4MB+ to be "large
>>         block size",
>>         > which would make the filesystem obviously unsuitable for
>>         processing
>>         > millions of 8KB files.
>>         >
>>         > You can make a histogram of file sizes in your existing
>>         filesystems and
>>         > then make your subblock size (1/32 of block size) on the
>>         smaller end of
>>         > that.? ?Also definitely use the "small file in inode"
>>         feature and put
>>         > your metadata on SSD.
>>         >
>>         > Regards,
>>         > Alex
>>         >
>>         > On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
>>         > <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>
>>         <mailto:aaron.s.knister at nasa.gov
>>         <mailto:aaron.s.knister at nasa.gov>>> wrote:
>>         >
>>         >? ? ?Thanks, Bill.
>>         >
>>         >? ? ?I still don't feel like I've got an clear answer from
>>         IBM and frankly
>>         >? ? ?the core issue of a lack of migration tool was totally
>>         dodged.
>>         >
>>         >? ? ?Again in Sven's presentation from SSUG @ SC17
>>         >? ?
>>         ?(http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf
>>         <http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf>)
>>         >? ? ?he mentions "It has a significant performance penalty
>>         for small files in
>>         >? ? ?large block size filesystems" and the demonstrates that
>>         with several
>>         >? ? ?mdtest runs (which show the effect with and without the >32
>>         >? ? ?subblocks code):
>>         >
>>         >
>>         >? ? ?4.2.1 base code - SUMMARY: (of 3 iterations)
>>         >? ? ?File creation : Mean = 2237.644
>>         >
>>         >? ? ?zero-end-of-file-padding (4.2.2 + ifdef for zero
>>         padding):? SUMMARY: (of
>>         >? ? ?3 iterations)
>>         >? ? ?File creation : Mean = 12866.842
>>         >
>>         >? ? ?more sub blocks per block (4.2.2 + morethan32subblock code):
>>         >? ? ?File creation : Mean = 40316.721
>>         >
>>         >? ? ?Can someone (ideally Sven) give me a straight answer as
>>         to whether or
>>         >? ? ?not the > 32 subblock code actually makes a performance
>>         difference for
>>         >? ? ?small files in large block filesystems? And if not, help
>>         me understand
>>         >? ? ?why his slides and provided benchmark data have
>>         consistently indicated
>>         >? ? ?it does?
>>         >
>>         >? ? ?-Aaron
>>         >
>>         >? ? ?On 12/1/17 11:44 AM, Bill Hartner wrote:
>>         >? ? ?> ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported
>>         some of the
>>         >? ? ?> results @ user group meeting.
>>         >? ? ?>
>>         >? ? ?> -Bill
>>         >? ? ?>
>>         >? ? ?> Bill Hartner
>>         >? ? ?> IBM Systems
>>         >? ? ?> Scalable I/O Development
>>         >? ? ?> Austin, Texas
>>         >? ? ?> bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
>>         <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>
>>         >? ? ?> home office 512-784-0980 <tel:(512)%20784-0980>
>>         <tel:512-784-0980 <tel:(512)%20784-0980>>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> Inactive hide details for Jan-Frode Myklebust
>>         ---12/01/2017 06:53:44
>>         >? ? ?> AM---Bill, could you say something about what the
>>         metadataJan-Frode
>>         >? ? ?> Myklebust ---12/01/2017 06:53:44 AM---Bill, could you
>>         say something
>>         >? ? ?> about what the metadata-storage here was?
>>         ESS/NL-SAS/3way replication?
>>         >? ? ?>
>>         >? ? ?> From: Jan-Frode Myklebust <janfrode at tanso.net
>>         <mailto:janfrode at tanso.net> <mailto:janfrode at tanso.net
>>         <mailto:janfrode at tanso.net>>>
>>         >? ? ?> To: gpfsug main discussion list
>>         <gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>>>
>>         >? ? ?> Date: 12/01/2017 06:53 AM
>>         >? ? ?> Subject: Re: [gpfsug-discuss] Online data migration tool
>>         >? ? ?> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ?
>>         ?------------------------------------------------------------------------
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> Bill, could you say something about what the
>>         metadata-storage here was?
>>         >? ? ?> ESS/NL-SAS/3way replication?
>>         >? ? ?>
>>         >? ? ?> I just asked about this in the internal slack channel
>>         #scale-help today..
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> -jf
>>         >? ? ?>
>>         >? ? ?> fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
>>         <_bhartner at us.ibm.com_
>>         >? ? ?> <mailto:bhartner at us.ibm.com
>>         <mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com
>>         <mailto:bhartner at us.ibm.com>>>>:
>>         >? ? ?>
>>         >? ? ?>? ? ?> "It has a significant performance penalty for
>>         small files in
>>         >? ? ?large
>>         >? ? ?>? ? ?> block size filesystems"
>>         >? ? ?>
>>         >? ? ?>? ? ?Aaron,
>>         >? ? ?>
>>         >? ? ?>? ? ?Below are mdtest results for a test we ran for
>>         CORAL - file
>>         >? ? ?size was
>>         >? ? ?>? ? ?32k.
>>         >? ? ?>
>>         >? ? ?>? ? ?We have not gone back and ran the test on a file
>>         system formatted
>>         >? ? ?>? ? ?without > 32 subblocks. We'll do that at some point...
>>         >? ? ?>
>>         >? ? ?>? ? ?-Bill
>>         >? ? ?>
>>         >? ? ?>? ? ?-- started at 10/28/2017 17:51:38 --
>>         >? ? ?>
>>         >? ? ?>? ? ?mdtest-1.9.3 was launched with 228 total task(s)
>>         on 12 node(s)
>>         >? ? ?>? ? ?Command line used: /tmp/mdtest-binary-dir/mdtest -d
>>         >? ? ?>? ? ?/ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w
>>         32768 -C -F -r
>>         >? ? ?-p 360
>>         >? ? ?>? ? ?-u -y
>>         >? ? ?>? ? ?Path: /ibm/fs2-16m-10
>>         >? ? ?>? ? ?FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used
>>         Inodes: 0.0%
>>         >? ? ?>
>>         >? ? ?>? ? ?228 tasks, 67239936 files
>>         >? ? ?>
>>         >? ? ?>? ? ?SUMMARY: (of 3 iterations)
>>         >? ? ?>? ? ?Operation Max Min Mean Std Dev
>>         >? ? ?>? ? ?--------- --- --- ---- -------
>>         >? ? ?>? ? ?File creation : 51953.498 50558.517 51423.221 616.643
>>         >? ? ?>? ? ?File stat : 0.000 0.000 0.000 0.000
>>         >? ? ?>? ? ?File read : 0.000 0.000 0.000 0.000
>>         >? ? ?>? ? ?File removal : 96746.376 92149.535 94658.774 1900.187
>>         >? ? ?>? ? ?Tree creation : 1.588 0.070 0.599 0.700
>>         >? ? ?>? ? ?Tree removal : 0.213 0.034 0.097 0.082
>>         >? ? ?>
>>         >? ? ?>? ? ?-- finished at 10/28/2017 19:51:54 --
>>         >? ? ?>
>>         >? ? ?>? ? ?Bill Hartner
>>         >? ? ?>? ? ?IBM Systems
>>         >? ? ?>? ? ?Scalable I/O Development
>>         >? ? ?>? ? ?Austin, Texas_
>>         >? ? ?>? ? ?__bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
>>         <mailto:bhartner at us.ibm.com>
>>         >? ? ?<mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>
>>         >? ? ?>? ? ?home office 512-784-0980 <tel:(512)%20784-0980>
>>         <tel:512-784-0980 <tel:(512)%20784-0980>>
>>         >? ? ?>
>>         >? ? ?>? ? ?_
>>         >? ? ?>? ? ?__gpfsug-discuss-bounces at spectrumscale.org_
>>         >? ? ?>? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>>>?wrote on
>>         >? ? ?>? ? ?11/29/2017 04:41:48 PM:
>>         >? ? ?>
>>         >? ? ?>? ? ?> From: Aaron Knister <_aaron.knister at gmail.com_
>>         >? ? ?>? ? ?<mailto:aaron.knister at gmail.com
>>         <mailto:aaron.knister at gmail.com>
>>         <mailto:aaron.knister at gmail.com
>>         <mailto:aaron.knister at gmail.com>>>>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> To: gpfsug main discussion list
>>         >? ? ?>? ? ?<_gpfsug-discuss at spectrumscale.org_
>>         >? ? ?>? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss at spectrumscale.org
>>         <mailto:gpfsug-discuss at spectrumscale.org>>>>
>>         >? ? ?>
>>         >? ? ?>? ? ?> Date: 11/29/2017 04:42 PM
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> Subject: Re: [gpfsug-discuss] Online data
>>         migration tool
>>         >? ? ?>? ? ?> Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>>         >? ? ?>? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>         >? ? ?<mailto:gpfsug-discuss-bounces at spectrumscale.org
>>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>>>
>>         >? ? ?>
>>         >? ? ?>? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> Thanks, Nikhil. Most of that was consistent with
>>         my understnading,
>>         >? ? ?>? ? ?> however I was under the impression that the >32
>>         subblocks code is
>>         >? ? ?>? ? ?> required to achieve the touted 50k file
>>         creates/second that Sven has
>>         >? ? ?>? ? ?> talked about a bunch of times:
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         >? ?
>>         ??_http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_
>>         >? ?
>>         ?<http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         >? ?
>>         ??_http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_
>>         >? ?
>>         ?<http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_>
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>>         >? ? ?>? ? ?> _Sven_Oehme_-_News_from_Research.pdf
>>         >? ? ?>? ? ?>
>>         _http://files.gpfsug.org/presentations/2016/SC16/12_-_
>>         >? ? ?<http://files.gpfsug.org/presentations/2016/SC16/12_-_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>>         >? ? ?>? ? ?>
>>         _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>? ? ?> from those presentations regarding 32 subblocks:
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> "It has a significant performance penalty for
>>         small files in large
>>         >? ? ?>? ? ?> block size filesystems"
>>         >? ? ?>
>>         >? ? ?>? ? ?> although I'm not clear on the specific
>>         definition of "large". Many
>>         >? ? ?>? ? ?> filesystems I encounter only have a 1M block
>>         size so it may not
>>         >? ? ?>? ? ?> matter there, although that same presentation
>>         clearly shows the
>>         >? ? ?>? ? ?> benefit of larger block sizes which is yet
>>         *another* thing for which
>>         >? ? ?>? ? ?> a migration tool would be helpful.
>>         >? ? ?>
>>         >? ? ?>? ? ?> -Aaron
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>>         >? ? ?>? ? ?<_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
>>         <mailto:nikhilk at us.ibm.com>
>>         >? ? ?<mailto:nikhilk at us.ibm.com
>>         <mailto:nikhilk at us.ibm.com>>>> wrote:
>>         >? ? ?>
>>         >? ? ?>? ? ?> Hi,
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> I would like to clarify migration path to 5.0.0
>>         from 4.X.X
>>         >? ? ?clusters.
>>         >? ? ?>? ? ?> For all Spectrum Scale clusters that are
>>         currently at 4.X.X,
>>         >? ? ?it is
>>         >? ? ?>? ? ?> possible to migrate to 5.0.0 with no offline
>>         data migration
>>         >? ? ?and no
>>         >? ? ?>? ? ?> need to move data. Once these clusters are at
>>         5.0.0, they will
>>         >? ? ?>? ? ?> benefit from the performance improvements, new
>>         features (such as
>>         >? ? ?>? ? ?> file audit logging), and various enhancements
>>         that are
>>         >? ? ?included in
>>         >? ? ?>? ? ?5.0.0.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> That being said, there is one enhancement that
>>         will not be
>>         >? ? ?applied
>>         >? ? ?>? ? ?> to these clusters, and that is the increased
>>         number of
>>         >? ? ?sub-blocks
>>         >? ? ?>? ? ?> per block for small file allocation. This means
>>         that for file
>>         >? ? ?>? ? ?> systems with a large block size and a lot of
>>         small files, the
>>         >? ? ?>? ? ?> overall space utilization will be the same it
>>         currently is
>>         >? ? ?in 4.X.X.
>>         >? ? ?>? ? ?> Since file systems created at 4.X.X and earlier
>>         used a block
>>         >? ? ?size
>>         >? ? ?>? ? ?> that kept this allocation in mind, there should
>>         be very little
>>         >? ? ?>? ? ?> impact on existing file systems.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> Outside of that one particular function, the
>>         remainder of the
>>         >? ? ?>? ? ?> performance improvements, metadata improvements,
>>         updated
>>         >? ? ?>? ? ?> compatibility, new functionality, and all of the
>>         other
>>         >? ? ?enhancements
>>         >? ? ?>? ? ?> will be immediately available to you once you
>>         complete the
>>         >? ? ?upgrade
>>         >? ? ?>? ? ?> to 5.0.0 -- with no need to reformat, move data,
>>         or take
>>         >? ? ?your data
>>         >? ? ?>? ? ?offline.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> I hope that clarifies things a little and makes
>>         the upgrade path
>>         >? ? ?>? ? ?> more accessible.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> Please let me know if there are any other
>>         questions or concerns.
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> Thank you,
>>         >? ? ?>? ? ?> Nikhil Khandelwal
>>         >? ? ?>? ? ?> Spectrum Scale Development
>>         >? ? ?>? ? ?> Client Adoption
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ? ?> _______________________________________________
>>         >? ? ?>? ? ?> gpfsug-discuss mailing list
>>         >? ? ?>? ? ?> gpfsug-discuss at _spectrumscale.org_
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>         >? ? ?>? ? ?> _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>>         >? ? ?>
>>         >? ? ?>? ? ?> _______________________________________________
>>         >? ? ?>? ? ?> gpfsug-discuss mailing list
>>         >? ? ?>? ? ?> gpfsug-discuss at _spectrumscale.org_
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>         >? ? ?>
>>         >? ? ?>? ? ?> _https://urldefense.proofpoint.com/v2/url?_
>>         >? ? ?<https://urldefense.proofpoint.com/v2/url?_>
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         ?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>>         >? ? ?>? ? ?>
>>         >? ? ?>? ?
>>         ?siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>>         >? ? ?>
>>         >? ? ?>? ? ?_______________________________________________
>>         >? ? ?>? ? ?gpfsug-discuss mailing list
>>         >? ? ?>? ? ?gpfsug-discuss at _spectrumscale.org_
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>>         >? ? ?>? ? ?__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_>
>>         >? ? ?>? ?
>>         >? ?
>>         ??<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>>         >? ? ?>? ? ?gpfsug-discuss mailing list
>>         >? ? ?>? ? ?gpfsug-discuss at spectrumscale.org
>>         <http://spectrumscale.org> <http://spectrumscale.org>
>>         >? ? ?>? ?
>>         >? ?
>>         ??https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>         >? ?
>>         ?<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?>
>>         >? ? ?> _______________________________________________
>>         >? ? ?> gpfsug-discuss mailing list
>>         >? ? ?> gpfsug-discuss at spectrumscale.org
>>         <http://spectrumscale.org> <http://spectrumscale.org>
>>         >? ? ?> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>>         >? ? ?>
>>         >
>>         >? ? ?--
>>         >? ? ?Aaron Knister
>>         >? ? ?NASA Center for Climate Simulation (Code 606.2)
>>         >? ? ?Goddard Space Flight Center
>>         >? ? ?(301) 286-2776 <tel:(301)%20286-2776>
>>         <tel:%28301%29%20286-2776>
>>         >? ? ?_______________________________________________
>>         >? ? ?gpfsug-discuss mailing list
>>         >? ? ?gpfsug-discuss at spectrumscale.org
>>         <http://spectrumscale.org> <http://spectrumscale.org>
>>         >? ? ?http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>         >? ? ?<http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>>         >
>>         >
>>         >
>>         >
>>         > _______________________________________________
>>         > gpfsug-discuss mailing list
>>         > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>>         > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>         >
>>
>>         --
>>         Aaron Knister
>>         NASA Center for Climate Simulation (Code 606.2)
>>         Goddard Space Flight Center
>>         (301) 286-2776 <tel:(301)%20286-2776>
>>         _______________________________________________
>>         gpfsug-discuss mailing list
>>         gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776


From bipcuds at gmail.com  Tue Dec 19 23:08:57 2017
From: bipcuds at gmail.com (Keith Ball)
Date: Tue, 19 Dec 2017 18:08:57 -0500
Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the
	Spectrum Scale FAQ
Message-ID: <CAAxuGpF6nw6R_Zmowx3Xv_pUbxikgvTXeR7g-p08TpMvh_ZgzQ@mail.gmail.com>

Hi All,

I was recently trying to determine the latest RHEL release that will work
with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code
uses Scale 4.1.0.8.

A specific X.Y GSS code build, from my experience, is intended to use a
specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken),
which no longer has EUS support from RedHat (only 6.7 still does).

GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be
used, which is a surprising statement given GSS/ESS code's sensitivity to
OS levels (any ESS I have ever seen has never been supported on more than
one version of RHEL).

According to the Scale FAQ (
https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux),
A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying
RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported
by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From
V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27!

Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install
GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is
supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option
here).

Many Thanks,
  Keith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171219/fa1226b4/attachment-0002.htm>

From olaf.weiser at de.ibm.com  Wed Dec 20 07:35:26 2017
From: olaf.weiser at de.ibm.com (Olaf Weiser)
Date: Wed, 20 Dec 2017 08:35:26 +0100
Subject: [gpfsug-discuss] pmcollector and NSD perf
In-Reply-To: <DM5PR05MB3577119DC6FE73462E83BC72E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
References: <DM5PR05MB357771F4628D9FE70A2F1877E40F0@DM5PR05MB3577.namprd05.prod.outlook.com><OFB3C3D3B8.626F1091-ON002581FB.0053F5A1-C12581FB.00549691@notes.na.collabserv.com>
	<DM5PR05MB3577119DC6FE73462E83BC72E40F0@DM5PR05MB3577.namprd05.prod.outlook.com>
Message-ID: <OFF3295D55.B6D7C020-ONC12581FC.0028AADD-C12581FC.0029B20E@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0004.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 166 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0017.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/1da0a669/attachment-0005.gif>

From john.hearns at asml.com  Wed Dec 20 08:41:07 2017
From: john.hearns at asml.com (John Hearns)
Date: Wed, 20 Dec 2017 08:41:07 +0000
Subject: [gpfsug-discuss] Expeliarmus
Message-ID: <HE1PR02MB1450E823234415275F6EADA3880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/a9561284/attachment-0002.htm>

From r.sobey at imperial.ac.uk  Wed Dec 20 08:58:20 2017
From: r.sobey at imperial.ac.uk (Sobey, Richard A)
Date: Wed, 20 Dec 2017 08:58:20 +0000
Subject: [gpfsug-discuss] Expeliarmus
In-Reply-To: <HE1PR02MB1450E823234415275F6EADA3880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
References: <HE1PR02MB1450E823234415275F6EADA3880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
Message-ID: <DB6PR0601MB21012910B19A735F4F5835DCDF0C0@DB6PR0601MB2101.eurprd06.prod.outlook.com>

Don't take it personally, I'm not entitled to even download 4.X. I can't be bothered to jump through the hoops necessary to get it going.

Richard

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of John Hearns
Sent: 20 December 2017 08:41
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Expeliarmus

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/b4f62495/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Wed Dec 20 09:31:25 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Wed, 20 Dec 2017 09:31:25 +0000
Subject: [gpfsug-discuss] Expeliarmus
Message-ID: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>

This is assuming you directly have ?IBM? licenses (as opposed to OEM licenses where the route is different ? or where the licenses are held by your VAR rather than you ?)

You need to have an IBM account which is attached to a (Passport Advantage) PA site that has current support for the product?

If you go to fix central, its also listed there now as well so again assuming your account is appropriately attached to a support contract, you can download it.

If you are struggling with these, then go speak to your business partner or IBM account manager.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "john.hearns at asml.com" <john.hearns at asml.com>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, 20 December 2017 at 08:41
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Expeliarmus

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/376c5117/attachment-0002.htm>

From john.hearns at asml.com  Wed Dec 20 13:02:54 2017
From: john.hearns at asml.com (John Hearns)
Date: Wed, 20 Dec 2017 13:02:54 +0000
Subject: [gpfsug-discuss] Expeliarmus
In-Reply-To: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>
References: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>
Message-ID: <HE1PR02MB1450FFCA4DEC69B3E2F70E4A880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>

Thnkyou both. I tapped on the door of Fix Central with my wand, and now have the eye of newt, wing of bat and a Spectrum Scale 5 installer.


From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support)
Sent: Wednesday, December 20, 2017 10:31 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Expeliarmus

This is assuming you directly have ?IBM? licenses (as opposed to OEM licenses where the route is different ? or where the licenses are held by your VAR rather than you ?)

You need to have an IBM account which is attached to a (Passport Advantage) PA site that has current support for the product?

If you go to fix central, its also listed there now as well so again assuming your account is appropriately attached to a support contract, you can download it.

If you are struggling with these, then go speak to your business partner or IBM account manager.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "john.hearns at asml.com<mailto:john.hearns at asml.com>" <john.hearns at asml.com<mailto:john.hearns at asml.com>>
Reply-To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date: Wednesday, 20 December 2017 at 08:41
To: "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] Expeliarmus

I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
For the life of me I cannot summon the spell needed to put Spectrum Scale 5 on my orders list.

Can some kindly witch give me the incantation please?
I would like to install on a test cluster, as the wisdom of the mages and mavens here has it.

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/55a82b6a/attachment-0002.htm>

From ewahl at osc.edu  Wed Dec 20 14:53:20 2017
From: ewahl at osc.edu (Edward Wahl)
Date: Wed, 20 Dec 2017 09:53:20 -0500
Subject: [gpfsug-discuss] Expeliarmus
In-Reply-To: <HE1PR02MB1450FFCA4DEC69B3E2F70E4A880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
References: <EA593CA3-4327-4800-BEBF-619D5816269A@bham.ac.uk>
	<HE1PR02MB1450FFCA4DEC69B3E2F70E4A880C0@HE1PR02MB1450.eurprd02.prod.outlook.com>
Message-ID: <20171220095320.1093db0d@osc.edu>


And it's not JUST a PA account. 

  You also have to have "someone" convert your old
GPFS part numbers into Spectrum Scale part numbers so that Passport Advantage
will show you the entitlements and downloads properly.  I went through this,
almost not painful at all, but rather slow.

Ed


On Wed, 20 Dec 2017 13:02:54 +0000
John Hearns <john.hearns at asml.com> wrote:

> Thnkyou both. I tapped on the door of Fix Central with my wand, and now have
> the eye of newt, wing of bat and a Spectrum Scale 5 installer.
> 
> 
> From: gpfsug-discuss-bounces at spectrumscale.org
> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson
> (IT Research Support) Sent: Wednesday, December 20, 2017 10:31 AM To: gpfsug
> main discussion list <gpfsug-discuss at spectrumscale.org> Subject: Re:
> [gpfsug-discuss] Expeliarmus
> 
> This is assuming you directly have ?IBM? licenses (as opposed to OEM licenses
> where the route is different ? or where the licenses are held by your VAR
> rather than you ?)
> 
> You need to have an IBM account which is attached to a (Passport Advantage)
> PA site that has current support for the product?
> 
> If you go to fix central, its also listed there now as well so again assuming
> your account is appropriately attached to a support contract, you can
> download it.
> 
> If you are struggling with these, then go speak to your business partner or
> IBM account manager.
> 
> Simon
> 
> From:
> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>>
> on behalf of "john.hearns at asml.com<mailto:john.hearns at asml.com>"
> <john.hearns at asml.com<mailto:john.hearns at asml.com>> Reply-To:
> "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>"
> <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
> Date: Wednesday, 20 December 2017 at 08:41 To:
> "gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>"
> <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
> Subject: [gpfsug-discuss] Expeliarmus
> 
> I have downloaded several versions of Spectrum Scale 4.X from the MyIBM site.
> For the life of me I cannot summon the spell needed to put Spectrum Scale 5
> on my orders list.
> 
> Can some kindly witch give me the incantation please?
> I would like to install on a test cluster, as the wisdom of the mages and
> mavens here has it.
> 
> -- The information contained in this communication and any attachments is
> confidential and may be privileged, and is for the sole use of the intended
> recipient(s). Any unauthorized review, use, disclosure or distribution is
> prohibited. Unless explicitly stated otherwise in the body of this
> communication or the attachment thereto (if any), the information is provided
> on an AS-IS basis without any express or implied warranties or liabilities.
> To the extent you are relying on this information, you are doing so at your
> own risk. If you are not the intended recipient, please notify the sender
> immediately by replying to this message and destroy all copies of this
> message and any attachments. Neither the sender nor the company/group of
> companies he or she represents shall be liable for the proper and complete
> transmission of the information contained in this communication, or for any
> delay in its receipt. -- The information contained in this communication and
> any attachments is confidential and may be privileged, and is for the sole
> use of the intended recipient(s). Any unauthorized review, use, disclosure or
> distribution is prohibited. Unless explicitly stated otherwise in the body of
> this communication or the attachment thereto (if any), the information is
> provided on an AS-IS basis without any express or implied warranties or
> liabilities. To the extent you are relying on this information, you are doing
> so at your own risk. If you are not the intended recipient, please notify the
> sender immediately by replying to this message and destroy all copies of this
> message and any attachments. Neither the sender nor the company/group of
> companies he or she represents shall be liable for the proper and complete
> transmission of the information contained in this communication, or for any
> delay in its receipt.



-- 

Ed Wahl
Ohio Supercomputer Center
614-292-9302


From eric.wonderley at vt.edu  Wed Dec 20 19:40:27 2017
From: eric.wonderley at vt.edu (J. Eric Wonderley)
Date: Wed, 20 Dec 2017 14:40:27 -0500
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same host
Message-ID: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>

Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all
mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core
switch.  I'm somewhat fearful as to how this is implemented in the juniper
switch we are about to test.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/3cf2f760/attachment-0002.htm>

From aaron.s.knister at nasa.gov  Wed Dec 20 20:25:04 2017
From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP])
Date: Wed, 20 Dec 2017 20:25:04 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
 host
In-Reply-To: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
References: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
Message-ID: <A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>



We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper.

Are you attempting to do RoCE or just plain TCP/IP?


On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu> wrote:
Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/a0eb8e44/attachment-0002.htm>

From eric.wonderley at vt.edu  Wed Dec 20 20:37:09 2017
From: eric.wonderley at vt.edu (J. Eric Wonderley)
Date: Wed, 20 Dec 2017 15:37:09 -0500
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
	host
In-Reply-To: <A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
References: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
Message-ID: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>

Just plain tcpip.

We have dual port connectx4s in our nsd servers.  Upon adding a second
connectx4 hba...no links go up or show "up".  I have one port on each hba
configured for eth and ibv_devinfo looks sane.

I cannot find anything indicating that this should not work.  I have a
ticket opened with mellanox.

On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER
SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:

>
>
> We?ve done a fair amount of VPI work but admittedly not with connectx4. Is
> it possible the cards are trying to talk IB rather than Eth? I figured
> you?re Ethernet based because of the mention of Juniper.
>
> Are you attempting to do RoCE or just plain TCP/IP?
>
>
> On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <
> eric.wonderley at vt.edu> wrote:
>
> Hello:
>
> Does anyone have this type of config?
>
> The host configuration looks sane but we seem to observe link-down on all
> mlx adapters no matter what we do.
>
> Big picture is that we are attempting to do mc(multichassis)-lags to a
> core switch.  I'm somewhat fearful as to how this is implemented in the
> juniper switch we are about to test.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/f26c0b63/attachment-0002.htm>

From S.J.Thompson at bham.ac.uk  Wed Dec 20 20:45:37 2017
From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support))
Date: Wed, 20 Dec 2017 20:45:37 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in
	same	host
In-Reply-To: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
References: <CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>,
	<CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
Message-ID: <CF45EE16DEF2FE4B9AA7FF2B6EE26545F5A42875@EX13.adf.bham.ac.uk>

I can't remember if this was on mlx4 or mlx5 driver cards, but we found we had to use LINKDELAY=20 when using bonding for Ethernet.

Simon
________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of J. Eric Wonderley [eric.wonderley at vt.edu]
Sent: 20 December 2017 20:37
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same      host

Just plain tcpip.

We have dual port connectx4s in our nsd servers.  Upon adding a second connectx4 hba...no links go up or show "up".  I have one port on each hba configured for eth and ibv_devinfo looks sane.

I cannot find anything indicating that this should not work.  I have a ticket opened with mellanox.

On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>> wrote:


We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper.

Are you attempting to do RoCE or just plain TCP/IP?


On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu<mailto:eric.wonderley at vt.edu>> wrote:
Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test.

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




From abeattie at au1.ibm.com  Wed Dec 20 21:46:53 2017
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Wed, 20 Dec 2017 21:46:53 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in
	samehost
In-Reply-To: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>,
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com><A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
Message-ID: <OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/4a535991/attachment-0002.htm>

From Paul.Sanchez at deshaw.com  Wed Dec 20 21:53:10 2017
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 20 Dec 2017 21:53:10 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter
	in	samehost
In-Reply-To: <OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>,
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com><A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
	<OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
Message-ID: <55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>

We have run multiple ConnectX-4 NICs in bonded MLAG (Arista) and VPC (Cisco) switch configurations on our NSD servers.  We used to see issues with firmware versions that didn?t support the optics we wanted to use (e.g. early CX3/CX4 and Cisco 40G-BiDi).  You may also want check mstflint to see whether the firmware levels match on the MLX cards, and if you upgrade firmware in some cases a power-cycle (not reboot) can be required to finish the process.

-Paul

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Andrew Beattie
Sent: Wednesday, December 20, 2017 4:47 PM
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in samehost

IBM ESS building blocks can have up to 3 dual port 10GBEth, 40GB Eth, 56GB IB, 100GB IB Mlx adapater cards, because we have 2 IO nodes this is up to a total of 12 ports per building block
so there should not be any reason for this to fail.

I regularly see a Mix of 10GB / 40GB or 10GB / IB configurations



Regards
Andrew Beattie
Software Defined Storage  - IT Specialist
Phone: 614-2133-7927
E-mail: abeattie at au1.ibm.com<mailto:abeattie at au1.ibm.com>


----- Original message -----
From: "J. Eric Wonderley" <eric.wonderley at vt.edu<mailto:eric.wonderley at vt.edu>>
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc:
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same host
Date: Thu, Dec 21, 2017 6:37 AM

Just plain tcpip.

We have dual port connectx4s in our nsd servers.  Upon adding a second connectx4 hba...no links go up or show "up".  I have one port on each hba configured for eth and ibv_devinfo looks sane.

I cannot find anything indicating that this should not work.  I have a ticket opened with mellanox.

On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov<mailto:aaron.s.knister at nasa.gov>> wrote:



We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper.

Are you attempting to do RoCE or just plain TCP/IP?

On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu<mailto:eric.wonderley at vt.edu>> wrote:
Hello:

Does anyone have this type of config?

The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.

Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test.

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=I5Dq2T7aYvC87Wp12fsz6CRLw4uo2-RVnrnpxRYfYuA&e=>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/0b3daff0/attachment-0002.htm>

From david_johnson at brown.edu  Wed Dec 20 22:00:49 2017
From: david_johnson at brown.edu (David D Johnson)
Date: Wed, 20 Dec 2017 17:00:49 -0500
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter
	in	samehost
In-Reply-To: <55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
	<OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
	<55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>
Message-ID: <D14EB35E-865A-400D-B81A-60FAC4002D22@brown.edu>

We're trying to get 40 gbe connection between Mellanox switches and Cisco switches down at the other end of the machine room  
The BiDi part seems to be the best given about 30m  run on multimode.  However Mellanox support says it's not supported.
Want to use this to get close to IB speeds for GPFS on nodes that aren't on the IB fabric.  
Does anyone have any luck getting 40 or 100 gig at 20-30m when the switches are different brands?

Thanks,
 -- ddj

> On Dec 20, 2017, at 4:53 PM, Sanchez, Paul <Paul.Sanchez at deshaw.com> wrote:
> 
> We have run multiple ConnectX-4 NICs in bonded MLAG (Arista) and VPC (Cisco) switch configurations on our NSD servers.  We used to see issues with firmware versions that didn?t support the optics we wanted to use (e.g. early CX3/CX4 and Cisco 40G-BiDi).  You may also want check mstflint to see whether the firmware levels match on the MLX cards, and if you upgrade firmware in some cases a power-cycle (not reboot) can be required to finish the process.
>  
> -Paul
> ? <>
> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org]On Behalf Of Andrew Beattie
> Sent: Wednesday, December 20, 2017 4:47 PM
> To: gpfsug-discuss at spectrumscale.org
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in samehost
>  
> IBM ESS building blocks can have up to 3 dual port 10GBEth, 40GB Eth, 56GB IB, 100GB IB Mlx adapater cards, because we have 2 IO nodes this is up to a total of 12 ports per building block
> so there should not be any reason for this to fail.
>  
> I regularly see a Mix of 10GB / 40GB or 10GB / IB configurations
>  
>  
>  
> Regards
> Andrew Beattie
> Software Defined Storage  - IT Specialist
> Phone: 614-2133-7927
> E-mail: abeattie at au1.ibm.com <mailto:abeattie at au1.ibm.com>
>  
>  
> ----- Original message -----
> From: "J. Eric Wonderley" <eric.wonderley at vt.edu <mailto:eric.wonderley at vt.edu>>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org <mailto:gpfsug-discuss-bounces at spectrumscale.org>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org <mailto:gpfsug-discuss at spectrumscale.org>>
> Cc:
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same host
> Date: Thu, Dec 21, 2017 6:37 AM
>   
> Just plain tcpip.
>  
> We have dual port connectx4s in our nsd servers.  Upon adding a second connectx4 hba...no links go up or show "up".  I have one port on each hba configured for eth and ibv_devinfo looks sane.
>  
> I cannot find anything indicating that this should not work.  I have a ticket opened with mellanox.
>  
> On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP] <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote: 
>  
> 
> 
> We?ve done a fair amount of VPI work but admittedly not with connectx4. Is it possible the cards are trying to talk IB rather than Eth? I figured you?re Ethernet based because of the mention of Juniper. 
>  
> Are you attempting to do RoCE or just plain TCP/IP?
>  
> On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <eric.wonderley at vt.edu <mailto:eric.wonderley at vt.edu>> wrote:
> Hello:
>  
> Does anyone have this type of config?
>  
> The host configuration looks sane but we seem to observe link-down on all mlx adapters no matter what we do.
>  
> Big picture is that we are attempting to do mc(multichassis)-lags to a core switch.  I'm somewhat fearful as to how this is implemented in the juniper switch we are about to test. 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=I5Dq2T7aYvC87Wp12fsz6CRLw4uo2-RVnrnpxRYfYuA&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>
>  
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171220/2b7ae14b/attachment-0002.htm>

From oehmes at gmail.com  Thu Dec 21 01:09:19 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Thu, 21 Dec 2017 01:09:19 +0000
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in
	samehost
In-Reply-To: <D14EB35E-865A-400D-B81A-60FAC4002D22@brown.edu>
References: <CABOSGQdJUa7khzJnW8U2A-=VTbyyVEwOB-E0vpSPhO9eAGY9AA@mail.gmail.com>
	<CABOSGQfFTVNpi5O_MhgFAH-zMbjTP50DHQHMdTBW4vrDWpTfpQ@mail.gmail.com>
	<A74A2A87-979C-4842-AFB6-0AA3E7594650@nasa.gov>
	<OFC3CFF68B.356510DF-ON002581FC.0077781F-002581FC.0077A65B@notes.na.collabserv.com>
	<55b310578aab448eaa818cb3e8fd7fdb@mbxtoa1.winmail.deshaw.com>
	<D14EB35E-865A-400D-B81A-60FAC4002D22@brown.edu>
Message-ID: <CALssuR3vrn_DO0o+mnquTNObOm=pALfxic1z4miMYnEwbC2vDw@mail.gmail.com>

i don't know if that works with cisco, but i use 50 an 100m cables for 40
as well as 100Gbit in my lab between 2 Mellanox switches :
http://www.mellanox.com/products/interconnect/ethernet-active-optical-cables.php
as paul pointed out one of the very first things one needs to do after
adding an adapter is to flash the firmware to a recent level. especially of
you have 2 adapters with different FW i have seen even once with higher
level not work properly, so before you do anything else get them to a
recent level and especially the same if its the same adapter types.

sven

On Wed, Dec 20, 2017 at 10:01 PM David D Johnson <david_johnson at brown.edu>
wrote:

> We're trying to get 40 gbe connection between Mellanox switches and Cisco
> switches down at the other end of the machine room
> The BiDi part seems to be the best given about 30m  run on multimode.
> However Mellanox support says it's not supported.
> Want to use this to get close to IB speeds for GPFS on nodes that aren't
> on the IB fabric.
> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
> switches are different brands?
>
> Thanks,
>  -- ddj
>
> On Dec 20, 2017, at 4:53 PM, Sanchez, Paul <Paul.Sanchez at deshaw.com>
> wrote:
>
> We have run multiple ConnectX-4 NICs in bonded MLAG (Arista) and VPC
> (Cisco) switch configurations on our NSD servers.  We used to see issues
> with firmware versions that didn?t support the optics we wanted to use
> (e.g. early CX3/CX4 and Cisco 40G-BiDi).  You may also want check mstflint
> to see whether the firmware levels match on the MLX cards, and if you
> upgrade firmware in some cases a power-cycle (not reboot) can be required
> to finish the process.
>
> -Paul
>
> *From:* gpfsug-discuss-bounces at spectrumscale.org [
> mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>]*On Behalf Of *Andrew Beattie
> *Sent:* Wednesday, December 20, 2017 4:47 PM
> *To:* gpfsug-discuss at spectrumscale.org
> *Subject:* Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in
> samehost
>
> IBM ESS building blocks can have up to 3 dual port 10GBEth, 40GB Eth, 56GB
> IB, 100GB IB Mlx adapater cards, because we have 2 IO nodes this is up to a
> total of 12 ports per building block
> so there should not be any reason for this to fail.
>
> I regularly see a Mix of 10GB / 40GB or 10GB / IB configurations
>
>
>
> Regards
> *Andrew Beattie*
> *Software Defined Storage  - IT Specialist*
> *Phone: *614-2133-7927
> *E-mail: *abeattie at au1.ibm.com
>
>
>
> ----- Original message -----
> From: "J. Eric Wonderley" <eric.wonderley at vt.edu>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
> host
> Date: Thu, Dec 21, 2017 6:37 AM
>
> Just plain tcpip.
>
> We have dual port connectx4s in our nsd servers.  Upon adding a second
> connectx4 hba...no links go up or show "up".  I have one port on each hba
> configured for eth and ibv_devinfo looks sane.
>
> I cannot find anything indicating that this should not work.  I have a
> ticket opened with mellanox.
>
> On Wed, Dec 20, 2017 at 3:25 PM, Knister, Aaron S. (GSFC-606.2)[COMPUTER
> SCIENCE CORP] <aaron.s.knister at nasa.gov> wrote:
>
>
>
>
> We?ve done a fair amount of VPI work but admittedly not with connectx4. Is
> it possible the cards are trying to talk IB rather than Eth? I figured
> you?re Ethernet based because of the mention of Juniper.
>
> Are you attempting to do RoCE or just plain TCP/IP?
>
> On December 20, 2017 at 14:40:48 EST, J. Eric Wonderley <
> eric.wonderley at vt.edu> wrote:
>
> Hello:
>
> Does anyone have this type of config?
>
> The host configuration looks sane but we seem to observe link-down on all
> mlx adapters no matter what we do.
>
> Big picture is that we are attempting to do mc(multichassis)-lags to a
> core switch.  I'm somewhat fearful as to how this is implemented in the
> juniper switch we are about to test.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=I5Dq2T7aYvC87Wp12fsz6CRLw4uo2-RVnrnpxRYfYuA&e=>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=Olk0lQk7rek9IplOjJ_2Vcd7P1LgVbnrSupC7O0hJHQ&s=hxNNpOkwGQ9zRmTnM3FEo5hgnPSUsPG0FNqZbK6eA6Q&e=
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/ef7b4e3c/attachment-0002.htm>

From kraemerf at de.ibm.com  Thu Dec 21 07:07:24 2017
From: kraemerf at de.ibm.com (Frank Kraemer)
Date: Thu, 21 Dec 2017 08:07:24 +0100
Subject: [gpfsug-discuss] more than one mlx connectx-4 adapter in same
	host
In-Reply-To: <mailman.3419.1513818573.19082.gpfsug-discuss@spectrumscale.org>
References: <mailman.3419.1513818573.19082.gpfsug-discuss@spectrumscale.org>
Message-ID: <OFE99AF1B2.2CCA57A0-ONC12581FD.00272158-C12581FD.00272158@notes.na.collabserv.com>

David,

> We're trying to get 40 gbe connection between Mellanox switches and Cisco
> switches down at the other end of the machine room
> The BiDi part seems to be the best given about 30m  run on multimode.
> However Mellanox support says it's not supported.
> Want to use this to get close to IB speeds for GPFS on nodes that aren't
> on the IB fabric.
> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
> switches are different brands?

maybe that's a good reason to get in contact with the team from Interoptic.
They claim a good expertise for these kind of problems - feedback is good.
http://packetpushers.net/podcast/podcasts/show-360-all-about-optics-interoptic-sponsored/
https://interoptic.com/

Frank Kraemer
IBM Consulting IT Specialist  / Client Technical Architect
Am Weiher 24, 65451 Kelsterbach
mailto:kraemerf at de.ibm.com
voice: +49-(0)171-3043699 / +4970342741078
IBM Germany
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/31aeae52/attachment-0002.htm>

From daniel.kidger at uk.ibm.com  Thu Dec 21 12:21:59 2017
From: daniel.kidger at uk.ibm.com (Daniel Kidger)
Date: Thu, 21 Dec 2017 12:21:59 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
Message-ID: <OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>

My suggestion is that it is better to not think of the performance coming from having more than 32 sub-blocks but instead that the performance comes from smaller sub-blocks. The fact that there are now more of them in say a 4MB blocksize filesytem is just a side effect.

Daniel


 

 
 	
Dr Daniel Kidger 
IBM Technical Sales Specialist
Software Defined Solution Sales

+ 44-(0)7818 522 266 
daniel.kidger at uk.ibm.com

> On 19 Dec 2017, at 21:32, Aaron Knister <aaron.s.knister at nasa.gov> wrote:
> 
> Thanks, Sven. Understood!
> 
>> On 12/19/17 3:20 PM, Sven Oehme wrote:
>> Hi,
>> 
>> the zero padding was never promoted into a GA stream, it was an
>> experiment to proof we are on the right track when we eliminate the
>> overhead from client to NSD Server, but also showed that alone is not
>> good enough. the work for the client is the same compared to the >32
>> subblocks, but the NSD Server has more work as it can't pack as many
>> subblocks and therefore files into larger blocks, so you need to do more
>> writes to store the same number of files. 
>> thats why there is the additional substantial improvement  when we then
>> went to >32 subblocks. 
>> 
>> sven
>> 
>> On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
>> SCIENCE CORP] <aaron.s.knister at nasa.gov
>> <mailto:aaron.s.knister at nasa.gov>> wrote:
>> 
>>    Thanks Sven! That makes sense to me and is what I thought was the
>>    case which is why I was confused when I saw the reply to the thread
>>    that said the >32 subblocks code had no performance impact. 
>> 
>>    A couple more question for you? in your presentation there?s a
>>    benchmark that shows the file create performance without the zero
>>    padding. Since you mention this is done for security reasons was
>>    that feature ever promoted to a GA Scale release? I?m also wondering
>>    if you could explain the performance difference between the no zero
>>    padding code and the > 32 subblock code since given your the example
>>    of 32K files and 16MB block size I figure both cases ought to write
>>    the same amount to disk. 
>> 
>>    Thanks!
>> 
>>    -Aaron
>> 
>> 
>> 
>> 
>> 
>>    On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com
>>    <mailto:oehmes at gmail.com>> wrote:
>>>    i thought i answered that already, but maybe i just thought about
>>>    answering it and then forgot about it :-D
>>> 
>>>    so yes more than 32 subblocks per block significant increase the
>>>    performance of filesystems with small files, for the sake of the
>>>    argument let's say 32k in a large block filesystem again for sake
>>>    of argument say 16MB. 
>>> 
>>>    you probably ask why ? 
>>> 
>>>    if you create a file and write 32k into it in a pre 5.0.0 Version
>>>    16 MB filesystem your client actually doesn't write 32k to the NSD
>>>    Server, it writes 512k, because thats the subblock size and we
>>>    need to write the full subblock (for security reasons). so first
>>>    you waste significant memory on the client to cache that zero
>>>    padding, you waste network bandwidth and you waste NSD Server
>>>    cache because you store it there too. this means you overrun the
>>>    cache more quickly, means you start doing read/modify writes
>>>    earlier on all your nice large raid tracks... i guess you get the
>>>    story by now. 
>>> 
>>>    in fact,  if you have a good raid code that can drive really a lot
>>>    of bandwidth out of individual drives like a GNR system you get
>>>    more performance for small file writes as larger your blocksize
>>>    is, because we can 'pack' more files into larger i/os and
>>>    therefore turn a small file create workload into a bandwidth
>>>    workload, essentially exactly what we did and i demonstrated in
>>>    the CORAL presentation . 
>>> 
>>>    hope that makes this crystal clear now . 
>>> 
>>>    sven
>>> 
>>> 
>>> 
>>>    On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister
>>>    <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
>>> 
>>>        Thanks, Alex. I'm all too familiar with the trade offs between
>>>        large
>>>        blocks and small files and we do use pretty robust SSD storage
>>>        for our
>>>        metadata. We support a wide range of workloads and we have
>>>        some folks
>>>        with many small (<1M) files and other folks with many large
>>>        (>256MB) files.
>>> 
>>>        My point in this thread is that IBM has said over and over
>>>        again in
>>>        presentations that there is a significant performance gain
>>>        with the >32
>>>        subblocks code on filesystems with large block sizes (although
>>>        to your
>>>        point I'm not clear on exactly what large means since I didn't
>>>        define
>>>        large in this context). Therefore given that the >32 subblock
>>>        code gives
>>>        a significant performance gain one could reasonably assume
>>>        that having a
>>>        filesystem with >32 subblocks is required to see this gain
>>>        (rather than
>>>        just running the >32 subblocks code on an fs w/o > 32 subblocks).
>>> 
>>>        This lead me to ask about a migration tool because in my mind
>>>        if there's
>>>        a performance gain from having >32 subblocks on the FS I'd
>>>        like that
>>>        feature and having to manually copy 10's of PB to new hardware
>>>        to get
>>>        this performance boost is unacceptable. However, IBM can't
>>>        seem to make
>>>        up their mind about whether or not the >32 subblocks code
>>>        *actually*
>>>        provides a performance increase. This seems like a pretty
>>>        straightforward question.
>>> 
>>>        -Aaron
>>> 
>>>>        On 12/15/17 3:48 PM, Alex Chekholko wrote:
>>>> Hey Aaron,
>>>> 
>>>> Can you define your sizes for "large blocks" and "small
>>>        files"?  If you
>>>> dial one up and the other down, your performance will be
>>>        worse.  And in
>>>> any case it's a pathological corner case so it shouldn't
>>>        matter much for
>>>> your workflow, unless you've designed your system with the
>>>        wrong values.
>>>> 
>>>> For example, for bioinformatics workloads, I prefer to use 256KB
>>>> filesystem block size, and I'd consider 4MB+ to be "large
>>>        block size",
>>>> which would make the filesystem obviously unsuitable for
>>>        processing
>>>> millions of 8KB files.
>>>> 
>>>> You can make a histogram of file sizes in your existing
>>>        filesystems and
>>>> then make your subblock size (1/32 of block size) on the
>>>        smaller end of
>>>> that.   Also definitely use the "small file in inode"
>>>        feature and put
>>>> your metadata on SSD.
>>>> 
>>>> Regards,
>>>> Alex
>>>> 
>>>> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
>>>> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>
>>>        <mailto:aaron.s.knister at nasa.gov
>>>        <mailto:aaron.s.knister at nasa.gov>>> wrote:
>>>> 
>>>>      Thanks, Bill.
>>>> 
>>>>      I still don't feel like I've got an clear answer from
>>>        IBM and frankly
>>>>      the core issue of a lack of migration tool was totally
>>>        dodged.
>>>> 
>>>>      Again in Sven's presentation from SSUG @ SC17
>>>>    
>>>         (https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=>)
>>>>      he mentions "It has a significant performance penalty
>>>        for small files in
>>>>      large block size filesystems" and the demonstrates that
>>>        with several
>>>>      mdtest runs (which show the effect with and without the >32
>>>>      subblocks code):
>>>> 
>>>> 
>>>>      4.2.1 base code - SUMMARY: (of 3 iterations)
>>>>      File creation : Mean = 2237.644
>>>> 
>>>>      zero-end-of-file-padding (4.2.2 + ifdef for zero
>>>        padding):  SUMMARY: (of
>>>>      3 iterations)
>>>>      File creation : Mean = 12866.842
>>>> 
>>>>      more sub blocks per block (4.2.2 + morethan32subblock code):
>>>>      File creation : Mean = 40316.721
>>>> 
>>>>      Can someone (ideally Sven) give me a straight answer as
>>>        to whether or
>>>>      not the > 32 subblock code actually makes a performance
>>>        difference for
>>>>      small files in large block filesystems? And if not, help
>>>        me understand
>>>>      why his slides and provided benchmark data have
>>>        consistently indicated
>>>>      it does?
>>>> 
>>>>      -Aaron
>>>> 
>>>>      On 12/1/17 11:44 AM, Bill Hartner wrote:
>>>>      > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported
>>>        some of the
>>>>      > results @ user group meeting.
>>>>      >
>>>>      > -Bill
>>>>      >
>>>>      > Bill Hartner
>>>>      > IBM Systems
>>>>      > Scalable I/O Development
>>>>      > Austin, Texas
>>>>      > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>
>>>        <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>
>>>>      > home office 512-784-0980 <tel:(512)%20784-0980>
>>>        <tel:512-784-0980 <tel:(512)%20784-0980>>
>>>>      >
>>>>      >
>>>>      > Inactive hide details for Jan-Frode Myklebust
>>>        ---12/01/2017 06:53:44
>>>>      > AM---Bill, could you say something about what the
>>>        metadataJan-Frode
>>>>      > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you
>>>        say something
>>>>      > about what the metadata-storage here was?
>>>        ESS/NL-SAS/3way replication?
>>>>      >
>>>>      > From: Jan-Frode Myklebust <janfrode at tanso.net
>>>        <mailto:janfrode at tanso.net> <mailto:janfrode at tanso.net
>>>        <mailto:janfrode at tanso.net>>>
>>>>      > To: gpfsug main discussion list
>>>        <gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>>>
>>>>      > Date: 12/01/2017 06:53 AM
>>>>      > Subject: Re: [gpfsug-discuss] Online data migration tool
>>>>      > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>>>>      >
>>>>      >
>>>>    
>>>         ------------------------------------------------------------------------
>>>>      >
>>>>      >
>>>>      >
>>>>      > Bill, could you say something about what the
>>>        metadata-storage here was?
>>>>      > ESS/NL-SAS/3way replication?
>>>>      >
>>>>      > I just asked about this in the internal slack channel
>>>        #scale-help today..
>>>>      >
>>>>      >
>>>>      >
>>>>      > -jf
>>>>      >
>>>>      > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
>>>        <_bhartner at us.ibm.com_
>>>>      > <mailto:bhartner at us.ibm.com
>>>        <mailto:bhartner at us.ibm.com> <mailto:bhartner at us.ibm.com
>>>        <mailto:bhartner at us.ibm.com>>>>:
>>>>      >
>>>>      >     > "It has a significant performance penalty for
>>>        small files in
>>>>      large
>>>>      >     > block size filesystems"
>>>>      >
>>>>      >     Aaron,
>>>>      >
>>>>      >     Below are mdtest results for a test we ran for
>>>        CORAL - file
>>>>      size was
>>>>      >     32k.
>>>>      >
>>>>      >     We have not gone back and ran the test on a file
>>>        system formatted
>>>>      >     without > 32 subblocks. We'll do that at some point...
>>>>      >
>>>>      >     -Bill
>>>>      >
>>>>      >     -- started at 10/28/2017 17:51:38 --
>>>>      >
>>>>      >     mdtest-1.9.3 was launched with 228 total task(s)
>>>        on 12 node(s)
>>>>      >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>>>>      >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w
>>>        32768 -C -F -r
>>>>      -p 360
>>>>      >     -u -y
>>>>      >     Path: /ibm/fs2-16m-10
>>>>      >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used
>>>        Inodes: 0.0%
>>>>      >
>>>>      >     228 tasks, 67239936 files
>>>>      >
>>>>      >     SUMMARY: (of 3 iterations)
>>>>      >     Operation Max Min Mean Std Dev
>>>>      >     --------- --- --- ---- -------
>>>>      >     File creation : 51953.498 50558.517 51423.221 616.643
>>>>      >     File stat : 0.000 0.000 0.000 0.000
>>>>      >     File read : 0.000 0.000 0.000 0.000
>>>>      >     File removal : 96746.376 92149.535 94658.774 1900.187
>>>>      >     Tree creation : 1.588 0.070 0.599 0.700
>>>>      >     Tree removal : 0.213 0.034 0.097 0.082
>>>>      >
>>>>      >     -- finished at 10/28/2017 19:51:54 --
>>>>      >
>>>>      >     Bill Hartner
>>>>      >     IBM Systems
>>>>      >     Scalable I/O Development
>>>>      >     Austin, Texas_
>>>>      >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
>>>        <mailto:bhartner at us.ibm.com>
>>>>      <mailto:bhartner at us.ibm.com <mailto:bhartner at us.ibm.com>>>
>>>>      >     home office 512-784-0980 <tel:(512)%20784-0980>
>>>        <tel:512-784-0980 <tel:(512)%20784-0980>>
>>>>      >
>>>>      >     _
>>>>      >     __gpfsug-discuss-bounces at spectrumscale.org_
>>>>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>>> wrote on
>>>>      >     11/29/2017 04:41:48 PM:
>>>>      >
>>>>      >     > From: Aaron Knister <_aaron.knister at gmail.com_
>>>>      >     <mailto:aaron.knister at gmail.com
>>>        <mailto:aaron.knister at gmail.com>
>>>        <mailto:aaron.knister at gmail.com
>>>        <mailto:aaron.knister at gmail.com>>>>
>>>>      >
>>>>      >
>>>>      >     > To: gpfsug main discussion list
>>>>      >     <_gpfsug-discuss at spectrumscale.org_
>>>>      >     <mailto:gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss at spectrumscale.org
>>>        <mailto:gpfsug-discuss at spectrumscale.org>>>>
>>>>      >
>>>>      >     > Date: 11/29/2017 04:42 PM
>>>>      >
>>>>      >
>>>>      >     > Subject: Re: [gpfsug-discuss] Online data
>>>        migration tool
>>>>      >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>>>>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>>>>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
>>>        <mailto:gpfsug-discuss-bounces at spectrumscale.org>>>
>>>>      >
>>>>      >     >
>>>>      >
>>>>      >     > Thanks, Nikhil. Most of that was consistent with
>>>        my understnading,
>>>>      >     > however I was under the impression that the >32
>>>        subblocks code is
>>>>      >     > required to achieve the touted 50k file
>>>        creates/second that Sven has
>>>>      >     > talked about a bunch of times:
>>>>      >     >
>>>>      >     >
>>>>      >   
>>>>    
>>>          _https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=>>
>>>>      >     >
>>>>      >   
>>>>    
>>>          _https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=>
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=>>
>>>>      >     > _Sven_Oehme_-_News_from_Research.pdf
>>>>      >     >
>>>        _https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=>>
>>>>      >     >
>>>        _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>>>>      >
>>>>      >
>>>>      >     > from those presentations regarding 32 subblocks:
>>>>      >     >
>>>>      >     > "It has a significant performance penalty for
>>>        small files in large
>>>>      >     > block size filesystems"
>>>>      >
>>>>      >     > although I'm not clear on the specific
>>>        definition of "large". Many
>>>>      >     > filesystems I encounter only have a 1M block
>>>        size so it may not
>>>>      >     > matter there, although that same presentation
>>>        clearly shows the
>>>>      >     > benefit of larger block sizes which is yet
>>>        *another* thing for which
>>>>      >     > a migration tool would be helpful.
>>>>      >
>>>>      >     > -Aaron
>>>>      >     >
>>>>      >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>>>>      >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
>>>        <mailto:nikhilk at us.ibm.com>
>>>>      <mailto:nikhilk at us.ibm.com
>>>        <mailto:nikhilk at us.ibm.com>>>> wrote:
>>>>      >
>>>>      >     > Hi,
>>>>      >     >
>>>>      >     > I would like to clarify migration path to 5.0.0
>>>        from 4.X.X
>>>>      clusters.
>>>>      >     > For all Spectrum Scale clusters that are
>>>        currently at 4.X.X,
>>>>      it is
>>>>      >     > possible to migrate to 5.0.0 with no offline
>>>        data migration
>>>>      and no
>>>>      >     > need to move data. Once these clusters are at
>>>        5.0.0, they will
>>>>      >     > benefit from the performance improvements, new
>>>        features (such as
>>>>      >     > file audit logging), and various enhancements
>>>        that are
>>>>      included in
>>>>      >     5.0.0.
>>>>      >     >
>>>>      >     > That being said, there is one enhancement that
>>>        will not be
>>>>      applied
>>>>      >     > to these clusters, and that is the increased
>>>        number of
>>>>      sub-blocks
>>>>      >     > per block for small file allocation. This means
>>>        that for file
>>>>      >     > systems with a large block size and a lot of
>>>        small files, the
>>>>      >     > overall space utilization will be the same it
>>>        currently is
>>>>      in 4.X.X.
>>>>      >     > Since file systems created at 4.X.X and earlier
>>>        used a block
>>>>      size
>>>>      >     > that kept this allocation in mind, there should
>>>        be very little
>>>>      >     > impact on existing file systems.
>>>>      >     >
>>>>      >     > Outside of that one particular function, the
>>>        remainder of the
>>>>      >     > performance improvements, metadata improvements,
>>>        updated
>>>>      >     > compatibility, new functionality, and all of the
>>>        other
>>>>      enhancements
>>>>      >     > will be immediately available to you once you
>>>        complete the
>>>>      upgrade
>>>>      >     > to 5.0.0 -- with no need to reformat, move data,
>>>        or take
>>>>      your data
>>>>      >     offline.
>>>>      >     >
>>>>      >     > I hope that clarifies things a little and makes
>>>        the upgrade path
>>>>      >     > more accessible.
>>>>      >     >
>>>>      >     > Please let me know if there are any other
>>>        questions or concerns.
>>>>      >     >
>>>>      >     > Thank you,
>>>>      >     > Nikhil Khandelwal
>>>>      >     > Spectrum Scale Development
>>>>      >     > Client Adoption
>>>>      >     >
>>>>      >     > _______________________________________________
>>>>      >     > gpfsug-discuss mailing list
>>>>      >     > gpfsug-discuss at _spectrumscale.org_
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>>>      >     > _https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>
>>>>      >
>>>>      >     > _______________________________________________
>>>>      >     > gpfsug-discuss mailing list
>>>>      >     > gpfsug-discuss at _spectrumscale.org_
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>
>>>>      >
>>>>      >     > _https://urldefense.proofpoint.com/v2/url?_
>>>>      <https://urldefense.proofpoint.com/v2/url?_>
>>>>      >     >
>>>>      >   
>>>         u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>>>>      >     >
>>>>      >   
>>>         siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=
>>>>      >
>>>>      >     _______________________________________________
>>>>      >     gpfsug-discuss mailing list
>>>>      >     gpfsug-discuss at _spectrumscale.org_
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=>>_
>>>>      >     __https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=>
>>>>      >   
>>>>    
>>>          <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>>_______________________________________________
>>>>      >     gpfsug-discuss mailing list
>>>>      >     gpfsug-discuss at spectrumscale.org
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>>      >   
>>>>    
>>>          https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>>>>    
>>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=>
>>>>      >
>>>>      >
>>>>      >
>>>>      >
>>>>      > _______________________________________________
>>>>      > gpfsug-discuss mailing list
>>>>      > gpfsug-discuss at spectrumscale.org
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>>      > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=>
>>>>      >
>>>> 
>>>>      --
>>>>      Aaron Knister
>>>>      NASA Center for Climate Simulation (Code 606.2)
>>>>      Goddard Space Flight Center
>>>>      (301) 286-2776 <tel:(301)%20286-2776>
>>>        <tel:%28301%29%20286-2776>
>>>>      _______________________________________________
>>>>      gpfsug-discuss mailing list
>>>>      gpfsug-discuss at spectrumscale.org
>>>        <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=> <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>>      https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>>>      <https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=>
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> gpfsug-discuss mailing list
>>>> gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>>> 
>>> 
>>>        --
>>>        Aaron Knister
>>>        NASA Center for Climate Simulation (Code 606.2)
>>>        Goddard Space Flight Center
>>>        (301) 286-2776 <tel:(301)%20286-2776>
>>>        _______________________________________________
>>>        gpfsug-discuss mailing list
>>>        gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>>        https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>>> 
>>    _______________________________________________
>>    gpfsug-discuss mailing list
>>    gpfsug-discuss at spectrumscale.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=E8EYx8GcVSBTAiWBsLOEIKqJVguZN9klgMUg-Ggeh28&e=>
>>    https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>> 
>> 
>> 
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
>> 
> 
> -- 
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=QVUNLL0-CptOraOHQIoZ4ApWaqAgO-JTb-rhyidzipI&e=
> 
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/facaf6ea/attachment-0002.htm>

From kh.atmane at gmail.com  Thu Dec 21 12:41:49 2017
From: kh.atmane at gmail.com (atmane)
Date: Thu, 21 Dec 2017 13:41:49 +0100
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 49
In-Reply-To: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
References: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
Message-ID: <op.zblj74uypgw25x@pc-atm>

Hi
This Is My Config

mmlsconfig | grep verbsRdma
verbsRdma enable
mmlsconfig | grep verbsPorts
verbsPorts mlx4_0/1 mlx4_0/2 mlx4_1/1 mlx4_1/2
ibstat | grep -E "CA|Port"

CA 'mlx4_0'
         CA type: MT4099
         Port 1:
                 Port GUID: 0x0002c903001a1571
         Port 2:
                 Port GUID: 0x0002c903001a1572
CA 'mlx4_1'
         CA type: MT4099
         Port 1:
                 Port GUID: 0x0002c903001a15c1
         Port 2:
                 Port GUID: 0x0002c903001a15c2
ethtool ib0 |grep "Link detected"
Link detected: yes
ethtool ib1 |grep "Link detected"
Link detected: yes
ethtool ib2 |grep "Link detected"
Link detected: yes
ethtool ib3 |grep "Link detected"
Link detected: yes
ifconfig ib0 |grep inet
inet 192.168.100.3 netmask 255.255.255.0  broadcast 192.168.100.255
ifconfig ib1 |grep inet
Infiniband hardware address can be incorrect! Please read BUGS section in  
ifconfig(8).
ifconfig ib2 |grep inet
Infiniband hardware address can be incorrect! Please read BUGS section in  
ifconfig(8).
ifconfig ib3 |grep inet
Infiniband hardware address can be incorrect! Please read BUGS section in  
ifconfig(8).
ip addr show ib1
8: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
1024
     link/infiniband  
a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:72 brd  
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
9: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
1024
     link/infiniband  
a0:00:02:20:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c1 brd  
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

10: ib3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP  
qlen 1024
     link/infiniband  
a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c2 brd  
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff






Le Thu, 21 Dec 2017 13:00:01 +0100,  
<gpfsug-discuss-request at spectrumscale.org> a ?crit:

> Send gpfsug-discuss mailing list submissions to
> 	gpfsug-discuss at spectrumscale.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
> 	gpfsug-discuss-request at spectrumscale.org
>
> You can reach the person managing the list at
> 	gpfsug-discuss-owner at spectrumscale.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: more than one mlx connectx-4 adapter in same	host
>       (Frank Kraemer)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 21 Dec 2017 08:07:24 +0100
> From: "Frank Kraemer" <kraemerf at de.ibm.com>
> To: gpfsug-discuss at spectrumscale.org
> Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in
> 	same	host
> Message-ID:
> 	<OFE99AF1B2.2CCA57A0-ONC12581FD.00272158-C12581FD.00272158 at notes.na.collabserv.com>
> 	
> Content-Type: text/plain; charset="us-ascii"
>
> David,
>
>> We're trying to get 40 gbe connection between Mellanox switches and  
>> Cisco
>> switches down at the other end of the machine room
>> The BiDi part seems to be the best given about 30m  run on multimode.
>> However Mellanox support says it's not supported.
>> Want to use this to get close to IB speeds for GPFS on nodes that aren't
>> on the IB fabric.
>> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
>> switches are different brands?
>
> maybe that's a good reason to get in contact with the team from  
> Interoptic.
> They claim a good expertise for these kind of problems - feedback is  
> good.
> http://packetpushers.net/podcast/podcasts/show-360-all-about-optics-interoptic-sponsored/
> https://interoptic.com/
>
> Frank Kraemer
> IBM Consulting IT Specialist  / Client Technical Architect
> Am Weiher 24, 65451 Kelsterbach
> mailto:kraemerf at de.ibm.com
> voice: +49-(0)171-3043699 / +4970342741078
> IBM Germany
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:  
> <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171221/31aeae52/attachment-0001.html>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 71, Issue 49
> **********************************************


-- 
Utilisant le logiciel de courrier d'Opera : http://www.opera.com/mail/


From a.khiredine at meteo.dz  Thu Dec 21 14:02:40 2017
From: a.khiredine at meteo.dz (atmane khiredine)
Date: Thu, 21 Dec 2017 14:02:40 +0000
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 49
In-Reply-To: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
References: <mailman.1.1513857601.59910.gpfsug-discuss@spectrumscale.org>
Message-ID: <4B32CB5C696F2849BDEF7DF9EACE884B72ACDD56@SDEB-EXC02.meteo.dz>



Atmane Khiredine
HPC System Administrator | Office National de la M?t?orologie
T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz

________________________________________
De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org]
Envoy? : jeudi 21 d?cembre 2017 13:00
? : gpfsug-discuss at spectrumscale.org
Objet : gpfsug-discuss Digest, Vol 71, Issue 49

Send gpfsug-discuss mailing list submissions to
        gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
        gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
        gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Re: more than one mlx connectx-4 adapter in same  host
      (Frank Kraemer)

  This Is My Config
    mmlsconfig | grep verbsRdma
    verbsRdma enable
    mmlsconfig | grep verbsPorts
    verbsPorts mlx4_0/1 mlx4_0/2 mlx4_1/1 mlx4_1/2
    ibstat | grep -E "CA|Port"
    CA 'mlx4_0'
             CA type: MT4099
             Port 1:
                     Port GUID: 0x0002c903001a1571
             Port 2:
                     Port GUID: 0x0002c903001a1572
    CA 'mlx4_1'
             CA type: MT4099
             Port 1:
                     Port GUID: 0x0002c903001a15c1
             Port 2:
                     Port GUID: 0x0002c903001a15c2
    ethtool ib0 |grep "Link detected"
    Link detected: yes
    ethtool ib1 |grep "Link detected"
    Link detected: yes
 

    ethtool ib2 |grep "Link detected"
    Link detected: yes
    ethtool ib3 |grep "Link detected"
    Link detected: yes
    ifconfig ib0 |grep inet
    inet 192.168.100.3 netmask 255.255.255.0  broadcast 192.168.100.255
    ifconfig ib1 |grep inet
    Infiniband hardware address can be incorrect! Please read BUGS section in  
    ifconfig(8).
    ifconfig ib2 |grep inet
    Infiniband hardware address can be incorrect! Please read BUGS section in  
    ifconfig(8).
    ifconfig ib3 |grep inet
    Infiniband hardware address can be incorrect! Please read BUGS section in  
    ifconfig(8).
    ip addr show ib1
    8: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
    1024
         link/infiniband  
    a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:72 brd  
    00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    9: ib2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP qlen  
    1024
         link/infiniband  
    a0:00:02:20:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c1 brd  
    00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
 
    10: ib3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc mq state UP  
    qlen 1024
         link/infiniband  
    a0:00:03:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:1a:15:c2 brd  
    00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

Message: 1
Date: Thu, 21 Dec 2017 08:07:24 +0100
From: "Frank Kraemer" <kraemerf at de.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] more than one mlx connectx-4 adapter in
        same    host
Message-ID:
        <OFE99AF1B2.2CCA57A0-ONC12581FD.00272158-C12581FD.00272158 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

David,

> We're trying to get 40 gbe connection between Mellanox switches and Cisco
> switches down at the other end of the machine room
> The BiDi part seems to be the best given about 30m  run on multimode.
> However Mellanox support says it's not supported.
> Want to use this to get close to IB speeds for GPFS on nodes that aren't
> on the IB fabric.
> Does anyone have any luck getting 40 or 100 gig at 20-30m when the
> switches are different brands?

maybe that's a good reason to get in contact with the team from Interoptic.
They claim a good expertise for these kind of problems - feedback is good.
http://packetpushers.net/podcast/podcasts/show-360-all-about-optics-interoptic-sponsored/
https://interoptic.com/

Frank Kraemer
IBM Consulting IT Specialist  / Client Technical Architect
Am Weiher 24, 65451 Kelsterbach
mailto:kraemerf at de.ibm.com
voice: +49-(0)171-3043699 / +4970342741078
IBM Germany
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171221/31aeae52/attachment-0001.html>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 71, Issue 49
**********************************************


From oehmes at gmail.com  Thu Dec 21 16:38:27 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Thu, 21 Dec 2017 16:38:27 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
References: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
	<OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
Message-ID: <CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>

Daniel,

while this might be easier to think about it, its not true :-)
lets just use an example.  a disk drive can do 100 io's per second with
128kb random writes and 80 iops with 256kb writes . now lets do the math
with a 8+2p setup for each of the 2 cases. this means you can do 100 times
1mb writes (8*128k) or 80 times 2 mb writes so 100 MB/sec or 160 MB/sec
with the exact same drives. given you can fit 2 times as many subblocks
into the 2mb block you would gain 60% of speed by just going to this larger
size. so if you now go to a 16MB blocksize and you have just 50 iops @ 2MB
each you can write ~800 MB/sec with the exact same setup and same size
small writes, that's a factor of 8 .
so i/o size AND number of subblocks matter.

Sven



On Thu, Dec 21, 2017 at 12:22 PM Daniel Kidger <daniel.kidger at uk.ibm.com>
wrote:

> My suggestion is that it is better to not think of the performance coming
> from having more than 32 sub-blocks but instead that the performance comes
> from smaller sub-blocks. The fact that there are now more of them in say a
> 4MB blocksize filesytem is just a side effect.
>
> Daniel
> [image: /spectrum_storage-banne]
>
>
> [image: Spectrum Scale Logo]
>
>
> *Dr Daniel Kidger*
> IBM Technical Sales Specialist
> Software Defined Solution Sales
>
> + <+%2044-7818%20522%20266> 44-(0)7818 522 266 <+%2044-7818%20522%20266>
> daniel.kidger at uk.ibm.com
>
> On 19 Dec 2017, at 21:32, Aaron Knister <aaron.s.knister at nasa.gov> wrote:
>
> Thanks, Sven. Understood!
>
> On 12/19/17 3:20 PM, Sven Oehme wrote:
>
> Hi,
>
>
> the zero padding was never promoted into a GA stream, it was an
>
> experiment to proof we are on the right track when we eliminate the
>
> overhead from client to NSD Server, but also showed that alone is not
>
> good enough. the work for the client is the same compared to the >32
>
> subblocks, but the NSD Server has more work as it can't pack as many
>
> subblocks and therefore files into larger blocks, so you need to do more
>
> writes to store the same number of files.
>
> thats why there is the additional substantial improvement  when we then
>
> went to >32 subblocks.
>
>
> sven
>
>
> On Mon, Dec 18, 2017 at 9:13 PM Knister, Aaron S. (GSFC-606.2)[COMPUTER
>
> SCIENCE CORP] <aaron.s.knister at nasa.gov
>
> <mailto:aaron.s.knister at nasa.gov <aaron.s.knister at nasa.gov>>> wrote:
>
>
>    Thanks Sven! That makes sense to me and is what I thought was the
>
>    case which is why I was confused when I saw the reply to the thread
>
>    that said the >32 subblocks code had no performance impact.
>
>
>    A couple more question for you? in your presentation there?s a
>
>    benchmark that shows the file create performance without the zero
>
>    padding. Since you mention this is done for security reasons was
>
>    that feature ever promoted to a GA Scale release? I?m also wondering
>
>    if you could explain the performance difference between the no zero
>
>    padding code and the > 32 subblock code since given your the example
>
>    of 32K files and 16MB block size I figure both cases ought to write
>
>    the same amount to disk.
>
>
>    Thanks!
>
>
>    -Aaron
>
>
>
>
>
>
>    On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes at gmail.com
>
>    <mailto:oehmes at gmail.com <oehmes at gmail.com>>> wrote:
>
>    i thought i answered that already, but maybe i just thought about
>
>    answering it and then forgot about it :-D
>
>
>    so yes more than 32 subblocks per block significant increase the
>
>    performance of filesystems with small files, for the sake of the
>
>    argument let's say 32k in a large block filesystem again for sake
>
>    of argument say 16MB.
>
>
>    you probably ask why ?
>
>
>    if you create a file and write 32k into it in a pre 5.0.0 Version
>
>    16 MB filesystem your client actually doesn't write 32k to the NSD
>
>    Server, it writes 512k, because thats the subblock size and we
>
>    need to write the full subblock (for security reasons). so first
>
>    you waste significant memory on the client to cache that zero
>
>    padding, you waste network bandwidth and you waste NSD Server
>
>    cache because you store it there too. this means you overrun the
>
>    cache more quickly, means you start doing read/modify writes
>
>    earlier on all your nice large raid tracks... i guess you get the
>
>    story by now.
>
>
>    in fact,  if you have a good raid code that can drive really a lot
>
>    of bandwidth out of individual drives like a GNR system you get
>
>    more performance for small file writes as larger your blocksize
>
>    is, because we can 'pack' more files into larger i/os and
>
>    therefore turn a small file create workload into a bandwidth
>
>    workload, essentially exactly what we did and i demonstrated in
>
>    the CORAL presentation .
>
>
>    hope that makes this crystal clear now .
>
>
>    sven
>
>
>
>
>    On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister
>
>    <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov
> <aaron.s.knister at nasa.gov>>> wrote:
>
>
>        Thanks, Alex. I'm all too familiar with the trade offs between
>
>        large
>
>        blocks and small files and we do use pretty robust SSD storage
>
>        for our
>
>        metadata. We support a wide range of workloads and we have
>
>        some folks
>
>        with many small (<1M) files and other folks with many large
>
>        (>256MB) files.
>
>
>        My point in this thread is that IBM has said over and over
>
>        again in
>
>        presentations that there is a significant performance gain
>
>        with the >32
>
>        subblocks code on filesystems with large block sizes (although
>
>        to your
>
>        point I'm not clear on exactly what large means since I didn't
>
>        define
>
>        large in this context). Therefore given that the >32 subblock
>
>        code gives
>
>        a significant performance gain one could reasonably assume
>
>        that having a
>
>        filesystem with >32 subblocks is required to see this gain
>
>        (rather than
>
>        just running the >32 subblocks code on an fs w/o > 32 subblocks).
>
>
>        This lead me to ask about a migration tool because in my mind
>
>        if there's
>
>        a performance gain from having >32 subblocks on the FS I'd
>
>        like that
>
>        feature and having to manually copy 10's of PB to new hardware
>
>        to get
>
>        this performance boost is unacceptable. However, IBM can't
>
>        seem to make
>
>        up their mind about whether or not the >32 subblocks code
>
>        *actually*
>
>        provides a performance increase. This seems like a pretty
>
>        straightforward question.
>
>
>        -Aaron
>
>
>        On 12/15/17 3:48 PM, Alex Chekholko wrote:
>
> Hey Aaron,
>
>
> Can you define your sizes for "large blocks" and "small
>
>        files"?  If you
>
> dial one up and the other down, your performance will be
>
>        worse.  And in
>
> any case it's a pathological corner case so it shouldn't
>
>        matter much for
>
> your workflow, unless you've designed your system with the
>
>        wrong values.
>
>
> For example, for bioinformatics workloads, I prefer to use 256KB
>
> filesystem block size, and I'd consider 4MB+ to be "large
>
>        block size",
>
> which would make the filesystem obviously unsuitable for
>
>        processing
>
> millions of 8KB files.
>
>
> You can make a histogram of file sizes in your existing
>
>        filesystems and
>
> then make your subblock size (1/32 of block size) on the
>
>        smaller end of
>
> that.   Also definitely use the "small file in inode"
>
>        feature and put
>
> your metadata on SSD.
>
>
> Regards,
>
> Alex
>
>
> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister
>
> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov
> <aaron.s.knister at nasa.gov>>
>
>        <mailto:aaron.s.knister at nasa.gov <aaron.s.knister at nasa.gov>
>
>        <mailto:aaron.s.knister at nasa.gov <aaron.s.knister at nasa.gov>>>>
> wrote:
>
>
>      Thanks, Bill.
>
>
>      I still don't feel like I've got an clear answer from
>
>        IBM and frankly
>
>      the core issue of a lack of migration tool was totally
>
>        dodged.
>
>
>      Again in Sven's presentation from SSUG @ SC17
>
>
>
>         (
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=
>
>        <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_SC17_SC17-2DUG-2DCORAL-5FV3.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=EdlC_gbmU-xxT7HcFq8IYttHSMts8BdrbqDSCqnt-_g&e=
> >)
>
>      he mentions "It has a significant performance penalty
>
>        for small files in
>
>      large block size filesystems" and the demonstrates that
>
>        with several
>
>      mdtest runs (which show the effect with and without the >32
>
>      subblocks code):
>
>
>
>      4.2.1 base code - SUMMARY: (of 3 iterations)
>
>      File creation : Mean = 2237.644
>
>
>      zero-end-of-file-padding (4.2.2 + ifdef for zero
>
>        padding):  SUMMARY: (of
>
>      3 iterations)
>
>      File creation : Mean = 12866.842
>
>
>      more sub blocks per block (4.2.2 + morethan32subblock code):
>
>      File creation : Mean = 40316.721
>
>
>      Can someone (ideally Sven) give me a straight answer as
>
>        to whether or
>
>      not the > 32 subblock code actually makes a performance
>
>        difference for
>
>      small files in large block filesystems? And if not, help
>
>        me understand
>
>      why his slides and provided benchmark data have
>
>        consistently indicated
>
>      it does?
>
>
>      -Aaron
>
>
>      On 12/1/17 11:44 AM, Bill Hartner wrote:
>
>      > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported
>
>        some of the
>
>      > results @ user group meeting.
>
>      >
>
>      > -Bill
>
>      >
>
>      > Bill Hartner
>
>      > IBM Systems
>
>      > Scalable I/O Development
>
>      > Austin, Texas
>
>      > bhartner at us.ibm.com <mailto:bhartner at us.ibm.com
> <bhartner at us.ibm.com>>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com> <
> mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>>
>
>      > home office 512-784-0980 <(512)%20784-0980> <tel:(512)%20784-0980>
>
>        <tel:512-784-0980 <512-784-0980> <tel:(512)%20784-0980>>
>
>      >
>
>      >
>
>      > Inactive hide details for Jan-Frode Myklebust
>
>        ---12/01/2017 06:53:44
>
>      > AM---Bill, could you say something about what the
>
>        metadataJan-Frode
>
>      > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you
>
>        say something
>
>      > about what the metadata-storage here was?
>
>        ESS/NL-SAS/3way replication?
>
>      >
>
>      > From: Jan-Frode Myklebust <janfrode at tanso.net
>
>        <mailto:janfrode at tanso.net <janfrode at tanso.net>> <
> mailto:janfrode at tanso.net <janfrode at tanso.net>
>
>        <mailto:janfrode at tanso.net <janfrode at tanso.net>>>>
>
>      > To: gpfsug main discussion list
>
>        <gpfsug-discuss at spectrumscale.org
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>>>
>
>      > Date: 12/01/2017 06:53 AM
>
>      > Subject: Re: [gpfsug-discuss] Online data migration tool
>
>      > Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>>
>
>      >
>
>      >
>
>
>
>
>         ------------------------------------------------------------------------
>
>      >
>
>      >
>
>      >
>
>      > Bill, could you say something about what the
>
>        metadata-storage here was?
>
>      > ESS/NL-SAS/3way replication?
>
>      >
>
>      > I just asked about this in the internal slack channel
>
>        #scale-help today..
>
>      >
>
>      >
>
>      >
>
>      > -jf
>
>      >
>
>      > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner
>
>        <_bhartner at us.ibm.com_
>
>      > <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>> <
> mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>>>>:
>
>      >
>
>      >     > "It has a significant performance penalty for
>
>        small files in
>
>      large
>
>      >     > block size filesystems"
>
>      >
>
>      >     Aaron,
>
>      >
>
>      >     Below are mdtest results for a test we ran for
>
>        CORAL - file
>
>      size was
>
>      >     32k.
>
>      >
>
>      >     We have not gone back and ran the test on a file
>
>        system formatted
>
>      >     without > 32 subblocks. We'll do that at some point...
>
>      >
>
>      >     -Bill
>
>      >
>
>      >     -- started at 10/28/2017 17:51:38 --
>
>      >
>
>      >     mdtest-1.9.3 was launched with 228 total task(s)
>
>        on 12 node(s)
>
>      >     Command line used: /tmp/mdtest-binary-dir/mdtest -d
>
>      >     /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w
>
>        32768 -C -F -r
>
>      -p 360
>
>      >     -u -y
>
>      >     Path: /ibm/fs2-16m-10
>
>      >     FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used
>
>        Inodes: 0.0%
>
>      >
>
>      >     228 tasks, 67239936 files
>
>      >
>
>      >     SUMMARY: (of 3 iterations)
>
>      >     Operation Max Min Mean Std Dev
>
>      >     --------- --- --- ---- -------
>
>      >     File creation : 51953.498 50558.517 51423.221 616.643
>
>      >     File stat : 0.000 0.000 0.000 0.000
>
>      >     File read : 0.000 0.000 0.000 0.000
>
>      >     File removal : 96746.376 92149.535 94658.774 1900.187
>
>      >     Tree creation : 1.588 0.070 0.599 0.700
>
>      >     Tree removal : 0.213 0.034 0.097 0.082
>
>      >
>
>      >     -- finished at 10/28/2017 19:51:54 --
>
>      >
>
>      >     Bill Hartner
>
>      >     IBM Systems
>
>      >     Scalable I/O Development
>
>      >     Austin, Texas_
>
>      >     __bhartner at us.ibm.com_ <mailto:bhartner at us.ibm.com
> <bhartner at us.ibm.com>
>
>        <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>
>
>      <mailto:bhartner at us.ibm.com <bhartner at us.ibm.com> <
> mailto:bhartner at us.ibm.com <bhartner at us.ibm.com>>>>
>
>      >     home office 512-784-0980 <(512)%20784-0980>
> <tel:(512)%20784-0980>
>
>        <tel:512-784-0980 <512-784-0980> <tel:(512)%20784-0980>>
>
>      >
>
>      >     _
>
>      >     __gpfsug-discuss-bounces at spectrumscale.org_
>
>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>>> wrote on
>
>      >     11/29/2017 04:41:48 PM:
>
>      >
>
>      >     > From: Aaron Knister <_aaron.knister at gmail.com_
>
>      >     <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>
>
>        <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>>
>
>        <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>
>
>        <mailto:aaron.knister at gmail.com <aaron.knister at gmail.com>>>>>
>
>      >
>
>      >
>
>      >     > To: gpfsug main discussion list
>
>      >     <_gpfsug-discuss at spectrumscale.org_
>
>      >     <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>
>
>        <mailto:gpfsug-discuss at spectrumscale.org
> <gpfsug-discuss at spectrumscale.org>>>>>
>
>      >
>
>      >     > Date: 11/29/2017 04:42 PM
>
>      >
>
>      >
>
>      >     > Subject: Re: [gpfsug-discuss] Online data
>
>        migration tool
>
>      >     > Sent by: _gpfsug-discuss-bounces at spectrumscale.org_
>
>      >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>
>
>      <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>
>
>        <mailto:gpfsug-discuss-bounces at spectrumscale.org
> <gpfsug-discuss-bounces at spectrumscale.org>>>>
>
>      >
>
>      >     >
>
>      >
>
>      >     > Thanks, Nikhil. Most of that was consistent with
>
>        my understnading,
>
>      >     > however I was under the impression that the >32
>
>        subblocks code is
>
>      >     > required to achieve the touted 50k file
>
>        creates/second that Sven has
>
>      >     > talked about a bunch of times:
>
>      >     >
>
>      >     >
>
>      >
>
>
>
>          _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=V_Pb-mxqz3Ji9fHRp9Ic9_ztzMsHk1bSzTmhbgGkRKU&e=
> >
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=
> >>
>
>      >     >
>
>      >
>
>
>
>          _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=61HBHh68SJXjnUv1Lyqjzmg_Vl24EG5cZ-0Z3WgLX3A&e=
> >
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=
> >>
>
>      >     > _Sven_Oehme_-_News_from_Research.pdf
>
>      >     >
>
>        _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=
>
>      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=fDAdLyWu9yx3_uj0z_N3IQ98yjXF7q5hDrg7ZYZYtRE&e=
> >
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=
> >>
>
>      >     >
>
>        _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf
>
>      >
>
>      >
>
>      >     > from those presentations regarding 32 subblocks:
>
>      >     >
>
>      >     > "It has a significant performance penalty for
>
>        small files in large
>
>      >     > block size filesystems"
>
>      >
>
>      >     > although I'm not clear on the specific
>
>        definition of "large". Many
>
>      >     > filesystems I encounter only have a 1M block
>
>        size so it may not
>
>      >     > matter there, although that same presentation
>
>        clearly shows the
>
>      >     > benefit of larger block sizes which is yet
>
>        *another* thing for which
>
>      >     > a migration tool would be helpful.
>
>      >
>
>      >     > -Aaron
>
>      >     >
>
>      >     > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal
>
>      >     <_nikhilk at us.ibm.com_ <mailto:nikhilk at us.ibm.com
> <nikhilk at us.ibm.com>
>
>        <mailto:nikhilk at us.ibm.com <nikhilk at us.ibm.com>>
>
>      <mailto:nikhilk at us.ibm.com <nikhilk at us.ibm.com>
>
>        <mailto:nikhilk at us.ibm.com <nikhilk at us.ibm.com>>>>> wrote:
>
>      >
>
>      >     > Hi,
>
>      >     >
>
>      >     > I would like to clarify migration path to 5.0.0
>
>        from 4.X.X
>
>      clusters.
>
>      >     > For all Spectrum Scale clusters that are
>
>        currently at 4.X.X,
>
>      it is
>
>      >     > possible to migrate to 5.0.0 with no offline
>
>        data migration
>
>      and no
>
>      >     > need to move data. Once these clusters are at
>
>        5.0.0, they will
>
>      >     > benefit from the performance improvements, new
>
>        features (such as
>
>      >     > file audit logging), and various enhancements
>
>        that are
>
>      included in
>
>      >     5.0.0.
>
>      >     >
>
>      >     > That being said, there is one enhancement that
>
>        will not be
>
>      applied
>
>      >     > to these clusters, and that is the increased
>
>        number of
>
>      sub-blocks
>
>      >     > per block for small file allocation. This means
>
>        that for file
>
>      >     > systems with a large block size and a lot of
>
>        small files, the
>
>      >     > overall space utilization will be the same it
>
>        currently is
>
>      in 4.X.X.
>
>      >     > Since file systems created at 4.X.X and earlier
>
>        used a block
>
>      size
>
>      >     > that kept this allocation in mind, there should
>
>        be very little
>
>      >     > impact on existing file systems.
>
>      >     >
>
>      >     > Outside of that one particular function, the
>
>        remainder of the
>
>      >     > performance improvements, metadata improvements,
>
>        updated
>
>      >     > compatibility, new functionality, and all of the
>
>        other
>
>      enhancements
>
>      >     > will be immediately available to you once you
>
>        complete the
>
>      upgrade
>
>      >     > to 5.0.0 -- with no need to reformat, move data,
>
>        or take
>
>      your data
>
>      >     offline.
>
>      >     >
>
>      >     > I hope that clarifies things a little and makes
>
>        the upgrade path
>
>      >     > more accessible.
>
>      >     >
>
>      >     > Please let me know if there are any other
>
>        questions or concerns.
>
>      >     >
>
>      >     > Thank you,
>
>      >     > Nikhil Khandelwal
>
>      >     > Spectrum Scale Development
>
>      >     > Client Adoption
>
>      >     >
>
>      >     > _______________________________________________
>
>      >     > gpfsug-discuss mailing list
>
>      >     > gpfsug-discuss at _spectrumscale.org_
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
> >>
>
>      >     > _
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
>
>      <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss-5F&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=ROiUtPAdbQ6DF9wWYS4MIUax_Xetm1p9qXbKzt6ZVf4&s=uD-N75Y8hXNsZ7FmnqLA4D6P8WsMrRGMIM9-Oy2vIgE&e=
> >
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
>
>
>
>         <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=
> >>
>
>      >
>
>      >     > _______________________________________________
>
>      >     > gpfsug-discuss mailing list
>
>      >     > gpfsug-discuss at _spectrumscale.org_
>
>      >
>
>
>
>          <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/6513c936/attachment-0002.htm>

From valdis.kletnieks at vt.edu  Thu Dec 21 20:35:26 2017
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Thu, 21 Dec 2017 15:35:26 -0500
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>
References: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
	<OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
	<CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>
Message-ID: <37655.1513888526@turing-police.cc.vt.edu>

On Thu, 21 Dec 2017 16:38:27 +0000, Sven Oehme said:

> size. so if you now go to a 16MB blocksize and you have just 50 iops @ 2MB
> each you can write ~800 MB/sec with the exact same setup and same size
> small writes, that's a factor of 8 .

That's assuming your metadata storage is able to handle open/read/write/close
on enough small files per second to push 800MB/sec.  If you're talking 128K subblocks,
you're going to need some 6,400 small files per second to fill that pipe...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171221/385aeb17/attachment-0002.sig>

From oehmes at gmail.com  Fri Dec 22 00:02:43 2017
From: oehmes at gmail.com (Sven Oehme)
Date: Fri, 22 Dec 2017 00:02:43 +0000
Subject: [gpfsug-discuss] Online data migration tool
In-Reply-To: <37655.1513888526@turing-police.cc.vt.edu>
References: <c6c74fa2-3115-5ade-6e70-1769905d72ba@nasa.gov>
	<OF9A5EF34E.442AFB80-ON002581FD.0043EE6E-1513858919875@notes.na.collabserv.com>
	<CALssuR11gCRPkNaVuD5W1aGVB-0UbQpHEk0qdarmdya-DDhWow@mail.gmail.com>
	<37655.1513888526@turing-police.cc.vt.edu>
Message-ID: <CALssuR0Pk=ck9rT-OTvp3SEUroDTDj2bcxFGDpHzRX1wztSjHQ@mail.gmail.com>

thats not how GPFS aehm Scale works :-)
each client has pre-allocated inodes in memory and creating files is a
matter of spooling records. yes, eventually you need to destage this to the
disk, but that happens only every few seconds and given this i/os are
usually very colocated so good storage cache technology can reduce i/os to
physical media significant.

to proof the point look at this numbers :
-- started at 10/17/2017 14:29:13 --

mdtest-1.9.3 was launched with 110 total task(s) on 11 node(s)
Command line used: /ghome/oehmes/mpi/bin/mdtest-pcmpi9131-existingdir -d
/ibm/fs2-16m-09/shared/mdtest-ec -i 1 -n 10000 -F -w 0 -Z -p 8 -N 11 -u
Path: /ibm/fs2-16m-09/shared
FS: 128.1 TiB   Used FS: 0.2%   Inodes: 476.8 Mi   Used Inodes: 0.0%

110 tasks, 1100000 files

SUMMARY: (of 1 iterations)
   Operation                      Max            Min           Mean
Std Dev
   ---------                      ---            ---           ----
-------
   File creation     :     444221.343     444221.343     444221.343
  0.000
   File stat         :    6704498.841    6704498.841    6704498.841
  0.000
   File read         :    3859105.596    3859105.596    3859105.596
  0.000
   File removal      :     409336.606     409336.606     409336.606
  0.000
   Tree creation     :          5.344          5.344          5.344
  0.000
   Tree removal      :          1.145          1.145          1.145
  0.000

-- finished at 10/17/2017 14:29:27 --

this is a run against a 16mb blocksize filesystem with only spinning disks
(just one GL6 ESS) , not a single SSD and as you can see , this system on
11 nodes produces 444k creates / second something far above and beyond of
what drives can do.

and yes i know this stuff is all very complicated and not easy to explain
:-)

sven


On Thu, Dec 21, 2017 at 8:35 PM <valdis.kletnieks at vt.edu> wrote:

> On Thu, 21 Dec 2017 16:38:27 +0000, Sven Oehme said:
>
> > size. so if you now go to a 16MB blocksize and you have just 50 iops @
> 2MB
> > each you can write ~800 MB/sec with the exact same setup and same size
> > small writes, that's a factor of 8 .
>
> That's assuming your metadata storage is able to handle
> open/read/write/close
> on enough small files per second to push 800MB/sec.  If you're talking
> 128K subblocks,
> you're going to need some 6,400 small files per second to fill that pipe...
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/a66aca84/attachment-0002.htm>

From damir.krstic at gmail.com  Fri Dec 22 17:44:50 2017
From: damir.krstic at gmail.com (Damir Krstic)
Date: Fri, 22 Dec 2017 17:44:50 +0000
Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group without
	takeover
Message-ID: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>

It's been a very frustrating couple of months with our 2 ESS systems. IBM
tells us we had blueflame bug and they came on site and updated our ESS to
the latest version back in middle of November. Wednesday night one of the
NSD servers in one of our ESS building blocks kernel panicked. No idea why
and none of the logs are insightful. We have a PMR open with IBM. I am not
very confident we will get to the bottom of what's causing kernel panics on
our IO servers. The system has gone down over 4 times now in 2 months.

When we tried brining it back up, it rejoined the recovery group and the IO
on the entire cluster locked up until we were able to find couple of
compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on
those nodes resolved the issue of the filesystem locking up.

So far we have never been successful in brining back an IO server and not
having a filesystem lock up until we find a node with pending state with
tscomm. Anyway, the system was stable for few minutes until the same IO
server that went down on Wednesday night went into an arbitrating mode. It
never recovered. We stopped gpfs on that server and IO recovered again. We
left gpfs down and cluster seems to be OK.

My question is, is there a way of brining back the IO server into the mix
without the recoverygroup takeover happening? Could I just start a gpfs and
have it back in the mix as a backup server for the recoverygroup and if so,
how do you do that. Right now that server is designated as primary server
for the recovery group. I would like to have both IO servers in the mix for
redundancy purposes.

This ESS situation is beyond frustrating and I don't see end in sight.

Any help is appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/4d155e13/attachment-0002.htm>

From janfrode at tanso.net  Fri Dec 22 17:55:39 2017
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 22 Dec 2017 17:55:39 +0000
Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group
	without takeover
In-Reply-To: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
References: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
Message-ID: <CAHwPatgUFx4GHitKsmiSWZJC=20FCVBoMgmaDnTK-8zUbj4J6g@mail.gmail.com>

Can?t you just reverse the mmchrecoverygroup --servers order, before
starting the io-server?


-jf
fre. 22. des. 2017 kl. 18:45 skrev Damir Krstic <damir.krstic at gmail.com>:

> It's been a very frustrating couple of months with our 2 ESS systems. IBM
> tells us we had blueflame bug and they came on site and updated our ESS to
> the latest version back in middle of November. Wednesday night one of the
> NSD servers in one of our ESS building blocks kernel panicked. No idea why
> and none of the logs are insightful. We have a PMR open with IBM. I am not
> very confident we will get to the bottom of what's causing kernel panics on
> our IO servers. The system has gone down over 4 times now in 2 months.
>
> When we tried brining it back up, it rejoined the recovery group and the
> IO on the entire cluster locked up until we were able to find couple of
> compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on
> those nodes resolved the issue of the filesystem locking up.
>
> So far we have never been successful in brining back an IO server and not
> having a filesystem lock up until we find a node with pending state with
> tscomm. Anyway, the system was stable for few minutes until the same IO
> server that went down on Wednesday night went into an arbitrating mode. It
> never recovered. We stopped gpfs on that server and IO recovered again. We
> left gpfs down and cluster seems to be OK.
>
> My question is, is there a way of brining back the IO server into the mix
> without the recoverygroup takeover happening? Could I just start a gpfs and
> have it back in the mix as a backup server for the recoverygroup and if so,
> how do you do that. Right now that server is designated as primary server
> for the recovery group. I would like to have both IO servers in the mix for
> redundancy purposes.
>
> This ESS situation is beyond frustrating and I don't see end in sight.
>
> Any help is appreciated.
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/d7a7cb53/attachment-0002.htm>

From alex at calicolabs.com  Fri Dec 22 18:01:58 2017
From: alex at calicolabs.com (Alex Chekholko)
Date: Fri, 22 Dec 2017 10:01:58 -0800
Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group
	without takeover
In-Reply-To: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
References: <CAKV+WqfwK0UTo-0LGXby=Lj6VyZ_HBOFK1HfE_1AnsgAJsSH9g@mail.gmail.com>
Message-ID: <CANcy_Pb8JnZG+nvhuDGFb_X+aKPerzwu74iM7VmVwvFckak9zQ@mail.gmail.com>

Hi Damir,

I'm not sure whether this applies to you, but this was my experience.

GPFS absolutely depends on a reliable network interconnect.  If anything
goes wrong on the network layer, GPFS may not be able to recover.

Do you have visibility and monitoring on all the low-level network counters
on all the relevant network interfaces?

e.g. if one of your clients is connected to a switch port that is flaky in
some way and some GPFS message goes unacked, you can get unusual client
state and then the whole cluster hangs...

In my case years ago we ended up replacing some flaky HP switches and all
our GPFS troubles went away!  And we were able to show those switches had
some issues when doing non-GPFS testing with iperf, etc.

Regards,
Alex

On Fri, Dec 22, 2017 at 9:44 AM, Damir Krstic <damir.krstic at gmail.com>
wrote:

> It's been a very frustrating couple of months with our 2 ESS systems. IBM
> tells us we had blueflame bug and they came on site and updated our ESS to
> the latest version back in middle of November. Wednesday night one of the
> NSD servers in one of our ESS building blocks kernel panicked. No idea why
> and none of the logs are insightful. We have a PMR open with IBM. I am not
> very confident we will get to the bottom of what's causing kernel panics on
> our IO servers. The system has gone down over 4 times now in 2 months.
>
> When we tried brining it back up, it rejoined the recovery group and the
> IO on the entire cluster locked up until we were able to find couple of
> compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on
> those nodes resolved the issue of the filesystem locking up.
>
> So far we have never been successful in brining back an IO server and not
> having a filesystem lock up until we find a node with pending state with
> tscomm. Anyway, the system was stable for few minutes until the same IO
> server that went down on Wednesday night went into an arbitrating mode. It
> never recovered. We stopped gpfs on that server and IO recovered again. We
> left gpfs down and cluster seems to be OK.
>
> My question is, is there a way of brining back the IO server into the mix
> without the recoverygroup takeover happening? Could I just start a gpfs and
> have it back in the mix as a backup server for the recoverygroup and if so,
> how do you do that. Right now that server is designated as primary server
> for the recovery group. I would like to have both IO servers in the mix for
> redundancy purposes.
>
> This ESS situation is beyond frustrating and I don't see end in sight.
>
> Any help is appreciated.
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171222/bded79d3/attachment-0002.htm>