From alandhae at gmx.de Thu Jun 1 10:35:37 2017 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Thu, 1 Jun 2017 11:35:37 +0200 (CEST) Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup Message-ID: Hello all out there, customer wants to receive periodical reports on the filesystem heat (relatively age) of files. We already switched on fileheat using mmchconfig. mmchconfig fileheatlosspercent=10,fileHeatPeriodMinutes=1440 for the reports, I think I need to know the file usage in a given time period. Are there any how-to for obtaining this information, examples for ILM policies to be used as a start? any help will be highly appreciated. Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From olaf.weiser at de.ibm.com Thu Jun 1 12:15:57 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 1 Jun 2017 13:15:57 +0200 Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Thu Jun 1 14:40:30 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 1 Jun 2017 09:40:30 -0400 Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup In-Reply-To: References: Message-ID: To generate a list of files and their file heat values... define([TS],esyscmd(date +%Y-%m-%d-%H-%M | tr -d '\n')) RULE 'x1' EXTERNAL LIST 'heat.TS' EXEC '' RULE 'x2' LIST 'heat.TS' SHOW(FILE_HEAT) WEIGHT(FILE_HEAT) /* use a WHERE clause to select or exclude files */ mmapplypolicy /gpfs-path-of-interest -I defer -P policy-rules-shown-above -f /path-for-result ... other good options are -N nodes -g /shared-temp To do it periodically use crontab. I defined the TS macro so each time you run it, you get a different filename. WEIGHT clause will cause "hottest" files to be listed firstmost. Notice that FILE_HEAT "shows" as a floating point number. --marc From: "Olaf Weiser" To: Andreas Landh?u?er , gpfsug main discussion list Date: 06/01/2017 07:17 AM Subject: Re: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Andreas, one could use the WEIGHT statement ... a simple policy for e.g. rule ?repack? MIGRATE FROM POOL ?xxxxxx? TO POOL ?xxxx? WEIGHT(FILE_HEAT) and then the -I prepare to see, what would be done by policy.. or you use the LIST function .. or .. and so on .. From: Andreas Landh?u?er To: gpfsug-discuss at spectrumscale.org Date: 06/01/2017 11:36 AM Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all out there, customer wants to receive periodical reports on the filesystem heat (relatively age) of files. We already switched on fileheat using mmchconfig. mmchconfig fileheatlosspercent=10,fileHeatPeriodMinutes=1440 for the reports, I think I need to know the file usage in a given time period. Are there any how-to for obtaining this information, examples for ILM policies to be used as a start? any help will be highly appreciated. Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Fri Jun 2 04:10:44 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Fri, 2 Jun 2017 03:10:44 +0000 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - Space Management (GPFS HSM) Message-ID: An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Fri Jun 2 10:28:36 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 02 Jun 2017 05:28:36 -0400 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - Space Management (GPFS HSM) In-Reply-To: References: Message-ID: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca> We have that situation. Users don't need to login to NSD's What you need is to add at least one gpfs client to the cluster (or multi-cluster), mount the DMAPI enabled file system, and use that node as a gateway for end-users. They can access the contents on the mount point with their own underprivileged accounts. Whether or not on a schedule, the moment an application or linux command (such as cp, cat, vi, etc) accesses a stub, the file will be staged. Jaime Quoting "Andrew Beattie" : > Quick question, Does anyone have a Scale / GPFS environment (HPC) > where users need the ability to recall data sets after they have been > stubbed, but only System Administrators are permitted to log onto the > NSD servers for security purposes. And if so how do you provide the > ability for the users to schedule their data set recalls? Regards, > Andrew Beattie Software Defined Storage - IT Specialist Phone: > 614-2133-7927 E-mail: abeattie at au1.ibm.com[1] > > > Links: > ------ > [1] mailto:abeattie at au1.ibm.com > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From abeattie at au1.ibm.com Fri Jun 2 10:48:11 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Fri, 2 Jun 2017 09:48:11 +0000 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) In-Reply-To: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca> References: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca>, Message-ID: An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Fri Jun 2 16:12:41 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 02 Jun 2017 11:12:41 -0400 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) In-Reply-To: References: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca>, Message-ID: <20170602111241.56882fx2qr2yz2ax@support.scinet.utoronto.ca> It has been a while since I used HSM with GPFS via TSM, but as far as I can remember, unprivileged users can run dsmmigrate and dsmrecall. Based on the instructions on the link, dsmrecall may now leverage the Recommended Access Order (RAO) available on enterprise drives, however root would have to be the one to invoke that feature. In that case we may have to develop a middleware/wrapper for dsmrecall that will run as root and act on behalf of the user when optimization is requested. Someone here more familiar with the latest version of TSM-HSM may be able to give us some hints on how people are doing this in practice. Jaime Quoting "Andrew Beattie" : > Thanks Jaime, How do you get around Optimised recalls? from what I > can see the optimised recall process needs a root level account to > retrieve a list of files > https://www.ibm.com/support/knowledgecenter/SSSR2R_7.1.1/com.ibm.itsm.hsmul.doc/c_recall_optimized_tape.html[1] > Regards, Andrew Beattie Software Defined Storage - IT Specialist > Phone: 614-2133-7927 E-mail: abeattie at au1.ibm.com[2] ----- > Original message ----- > From: "Jaime Pinto" > To: "gpfsug main discussion list" , > "Andrew Beattie" > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - > Space Management (GPFS HSM) > Date: Fri, Jun 2, 2017 7:28 PM > We have that situation. > Users don't need to login to NSD's > > What you need is to add at least one gpfs client to the cluster (or > multi-cluster), mount the DMAPI enabled file system, and use that > node > as a gateway for end-users. They can access the contents on the mount > > point with their own underprivileged accounts. > > Whether or not on a schedule, the moment an application or linux > command (such as cp, cat, vi, etc) accesses a stub, the file will be > > staged. > > Jaime > > Quoting "Andrew Beattie" : > >> Quick question, Does anyone have a Scale / GPFS environment (HPC) >> where users need the ability to recall data sets after they have > been >> stubbed, but only System Administrators are permitted to log onto > the >> NSD servers for security purposes. And if so how do you provide > the >> ability for the users to schedule their data set recalls? > Regards, >> Andrew Beattie Software Defined Storage - IT Specialist Phone: >> 614-2133-7927 E-mail: abeattie at au1.ibm.com[1] >> >> >> Links: >> ------ >> [1] mailto:abeattie at au1.ibm.com[3] >> > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials[4] > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From r.sobey at imperial.ac.uk Fri Jun 2 16:51:12 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 2 Jun 2017 15:51:12 +0000 Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS Message-ID: Hi all, Where should I start looking for a compatibility matrix between TSM and GPFS? Specifically, we are currently running TSM 7.1.6-2 and GPFS 4.2.1-2 with the intent to upgrade to GPFS 4.2.3-latest in early July. I've spent 30 minutes looking over various documents and the best I can find is this: http://www-01.ibm.com/support/docview.wss?uid=swg21248771 ..which talks about TSM in a Space Management context and would suggest that we need to upgrade to Spectrum Protect i.e. 8.1 and that GPFS 4.2.2.x is the maximum supported version... Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 2 17:40:11 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 2 Jun 2017 12:40:11 -0400 Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS In-Reply-To: References: Message-ID: Upgrading from GPFS 4.2.x to GPFS 4.2.y should not "break" TSM. If it does, someone goofed, that would be a bug. (My opinion) Think of it this way. TSM is an application that uses the OS and the FileSystem(s). TSM can't verify it will work with all future versions of OS and Filesystems, and the releases can't be in lock step. Having said that, 4.2.3 has been "out" for a while, so if there were a TSM incompatibility, someone would have likely hit it or will before July... Trust but verify... From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 06/02/2017 11:51 AM Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi all, Where should I start looking for a compatibility matrix between TSM and GPFS? Specifically, we are currently running TSM 7.1.6-2 and GPFS 4.2.1-2 with the intent to upgrade to GPFS 4.2.3-latest in early July. I?ve spent 30 minutes looking over various documents and the best I can find is this: http://www-01.ibm.com/support/docview.wss?uid=swg21248771 ..which talks about TSM in a Space Management context and would suggest that we need to upgrade to Spectrum Protect i.e. 8.1 and that GPFS 4.2.2.x is the maximum supported version? Cheers Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 08:51:10 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 08:51:10 +0100 Subject: [gpfsug-discuss] NSD access routes Message-ID: Morning all, Just a quick one about NSD access and read only disks. Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? Cheers, ----------------------------- Dave Goodbourn Head of Systems Milk Visual Effects Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Mon Jun 5 08:52:39 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Mon, 5 Jun 2017 07:52:39 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: Message-ID: Hi Have you look at LROC instead? Might fit in simpler way to what your are describing. -- Cheers > On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: > > Morning all, > > Just a quick one about NSD access and read only disks. > > Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. > > Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? > > Cheers, > ----------------------------- > Dave Goodbourn > > Head of Systems > Milk Visual Effects > Tel: +44 (0)20 3697 8448 > Mob: +44 (0)7917 411 069 Ellei edell?? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 09:02:16 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 09:02:16 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: Yeah, that was my back up plan but would be more costly in the cloud. Read only is a limitation of most cloud providers not something that I "want". Just trying to move a network bottleneck. Cheers, ----------------------------- Dave Goodbourn Head of Systems Milk Visual Effects Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 > On 5 Jun 2017, at 08:52, Luis Bolinches wrote: > > Hi > > Have you look at LROC instead? Might fit in simpler way to what your are describing. > > -- > Cheers > >> On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: >> >> Morning all, >> >> Just a quick one about NSD access and read only disks. >> >> Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. >> >> Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? >> >> Cheers, >> ----------------------------- >> Dave Goodbourn >> >> Head of Systems >> Milk Visual Effects >> Tel: +44 (0)20 3697 8448 >> Mob: +44 (0)7917 411 069 > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 13:19:47 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 13:19:47 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: OK scrap my first question, I can't do what I wanted to do anyway! I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. Cheers, ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 08:52, Luis Bolinches wrote: > Hi > > Have you look at LROC instead? Might fit in simpler way to what your are > describing. > > -- > Cheers > > On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: > > Morning all, > > Just a quick one about NSD access and read only disks. > > Can you have 2 NSD servers, one with read/write access to a disk and one > with just read only access to the same disk? I know you can write to a disk > over the network via another NSD server but can you mount the disk in read > only mode to increase the read performance? This is all virtual/cloud based. > > Is GPFS clever enough (or can it be configured) to know to read from the > locally attached read only disk but write back via another NSD server over > the GPFS network? > > Cheers, > ----------------------------- > Dave Goodbourn > > Head of Systems > Milk Visual Effects > Tel: +44 (0)20 3697 8448 <+44%2020%203697%208448> > Mob: +44 (0)7917 411 069 <+44%207917%20411069> > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Jun 5 13:24:27 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Mon, 5 Jun 2017 12:24:27 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: mmdiag --lroc ? From: > on behalf of "dave at milk-vfx.com" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 5 June 2017 at 13:19 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] NSD access routes OK scrap my first question, I can't do what I wanted to do anyway! I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. Cheers, ---------------------------------------------------- Dave Goodbourn Head of Systems MILK VISUAL EFFECTS [http://www.milk-vfx.com/src/milk_email_logo.jpg] 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 On 5 June 2017 at 08:52, Luis Bolinches > wrote: Hi Have you look at LROC instead? Might fit in simpler way to what your are describing. -- Cheers On 5 Jun 2017, at 10.51, Dave Goodbourn > wrote: Morning all, Just a quick one about NSD access and read only disks. Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? Cheers, ----------------------------- Dave Goodbourn Head of Systems Milk Visual Effects Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 5 13:48:48 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 5 Jun 2017 12:48:48 +0000 Subject: [gpfsug-discuss] NSD access routes Message-ID: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Hi Dave I?ve done a large-scale (600 node) LROC deployment here - feel free to reach out if you have questions. mmdiag --lroc is about all there is but it does give you a pretty good idea how the cache is performing but you can?t tell which files are cached. Also, watch out that the LROC cached will steal pagepool memory (1% of the LROC cache size) Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Dave Goodbourn Reply-To: gpfsug main discussion list Date: Monday, June 5, 2017 at 7:19 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] NSD access routes I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 14:10:21 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 14:10:21 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: Ah yep, thanks a lot. ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 13:24, Simon Thompson (IT Research Support) < S.J.Thompson at bham.ac.uk> wrote: > mmdiag --lroc > > ? > > > From: on behalf of " > dave at milk-vfx.com" > Reply-To: "gpfsug-discuss at spectrumscale.org" < > gpfsug-discuss at spectrumscale.org> > Date: Monday, 5 June 2017 at 13:19 > To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] NSD access routes > > OK scrap my first question, I can't do what I wanted to do anyway! > > I'm testing out the LROC idea. All seems to be working well, but, is there > anyway to monitor what's cached? How full it might be? The performance etc?? > > I can see some stats in mmfsadm dump lroc but that's about it. > > Cheers, > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 08:52, Luis Bolinches wrote: > >> Hi >> >> Have you look at LROC instead? Might fit in simpler way to what your are >> describing. >> >> -- >> Cheers >> >> On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: >> >> Morning all, >> >> Just a quick one about NSD access and read only disks. >> >> Can you have 2 NSD servers, one with read/write access to a disk and one >> with just read only access to the same disk? I know you can write to a disk >> over the network via another NSD server but can you mount the disk in read >> only mode to increase the read performance? This is all virtual/cloud based. >> >> Is GPFS clever enough (or can it be configured) to know to read from the >> locally attached read only disk but write back via another NSD server over >> the GPFS network? >> >> Cheers, >> ----------------------------- >> Dave Goodbourn >> >> Head of Systems >> Milk Visual Effects >> Tel: +44 (0)20 3697 8448 <+44%2020%203697%208448> >> Mob: +44 (0)7917 411 069 <+44%207917%20411069> >> >> >> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >> Oy IBM Finland Ab >> PL 265, 00101 Helsinki, Finland >> Business ID, Y-tunnus: 0195876-3 >> Registered in Finland >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 14:49:55 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 14:49:55 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: Thanks Bob, That pagepool comment has just answered my next question! But it doesn't seem to be working. Here's my mmdiag output: === mmdiag: lroc === LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' status Running Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 Max capacity: 1151997 MB, currently in use: 0 MB Statistics from: Mon Jun 5 13:40:50 2017 Total objects stored 0 (0 MB) recalled 0 (0 MB) objects failed to store 0 failed to recall 0 failed to inval 0 objects queried 0 (0 MB) not found 0 = 0.00 % objects invalidated 0 (0 MB) Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Inode objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Directory objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Data objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 agent inserts=0, reads=0 response times (usec): insert min/max/avg=0/0/0 read min/max/avg=0/0/0 ssd writeIOs=0, writePages=0 readIOs=0, readPages=0 response times (usec): write min/max/avg=0/0/0 read min/max/avg=0/0/0 I've restarted GPFS on that node just in case but that didn't seem to help. I have LROC on a node that DOESN'T have direct access to an NSD so will hopefully cache files that get requested over NFS. How often are these stats updated? The Statistics line doesn't seem to update when running the command again. Dave, ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 13:48, Oesterlin, Robert wrote: > Hi Dave > > > > I?ve done a large-scale (600 node) LROC deployment here - feel free to > reach out if you have questions. > > > > mmdiag --lroc is about all there is but it does give you a pretty good > idea how the cache is performing but you can?t tell which files are cached. > Also, watch out that the LROC cached will steal pagepool memory (1% of the > LROC cache size) > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > > > > > *From: * on behalf of Dave > Goodbourn > *Reply-To: *gpfsug main discussion list > *Date: *Monday, June 5, 2017 at 7:19 AM > *To: *gpfsug main discussion list > *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes > > > > I'm testing out the LROC idea. All seems to be working well, but, is there > anyway to monitor what's cached? How full it might be? The performance etc?? > > > > I can see some stats in mmfsadm dump lroc but that's about it. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 14:55:22 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 14:55:22 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: OK slightly ignore that last email. It's still not updating the output but I realise the Stats from line is when they started so probably won't update! :( Still nothing seems to being cached though. ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 14:49, Dave Goodbourn wrote: > Thanks Bob, > > That pagepool comment has just answered my next question! > > But it doesn't seem to be working. Here's my mmdiag output: > > === mmdiag: lroc === > LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' > status Running > Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 > Max capacity: 1151997 MB, currently in use: 0 MB > Statistics from: Mon Jun 5 13:40:50 2017 > > Total objects stored 0 (0 MB) recalled 0 (0 MB) > objects failed to store 0 failed to recall 0 failed to inval 0 > objects queried 0 (0 MB) not found 0 = 0.00 % > objects invalidated 0 (0 MB) > > Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Inode objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Directory objects failed to store 0 failed to recall 0 failed to > query 0 failed to inval 0 > > Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Data objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > agent inserts=0, reads=0 > response times (usec): > insert min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > ssd writeIOs=0, writePages=0 > readIOs=0, readPages=0 > response times (usec): > write min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > > I've restarted GPFS on that node just in case but that didn't seem to > help. I have LROC on a node that DOESN'T have direct access to an NSD so > will hopefully cache files that get requested over NFS. > > How often are these stats updated? The Statistics line doesn't seem to > update when running the command again. > > Dave, > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 13:48, Oesterlin, Robert > wrote: > >> Hi Dave >> >> >> >> I?ve done a large-scale (600 node) LROC deployment here - feel free to >> reach out if you have questions. >> >> >> >> mmdiag --lroc is about all there is but it does give you a pretty good >> idea how the cache is performing but you can?t tell which files are cached. >> Also, watch out that the LROC cached will steal pagepool memory (1% of the >> LROC cache size) >> >> >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> >> >> >> >> *From: * on behalf of Dave >> Goodbourn >> *Reply-To: *gpfsug main discussion list > > >> *Date: *Monday, June 5, 2017 at 7:19 AM >> *To: *gpfsug main discussion list >> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >> >> >> >> I'm testing out the LROC idea. All seems to be working well, but, is >> there anyway to monitor what's cached? How full it might be? The >> performance etc?? >> >> >> >> I can see some stats in mmfsadm dump lroc but that's about it. >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Jun 5 14:59:07 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Mon, 5 Jun 2017 13:59:07 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> , Message-ID: We've seen exactly this behaviour. Removing and readding the lroc nsd device worked for us. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of dave at milk-vfx.com [dave at milk-vfx.com] Sent: 05 June 2017 14:55 To: Oesterlin, Robert Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] NSD access routes OK slightly ignore that last email. It's still not updating the output but I realise the Stats from line is when they started so probably won't update! :( Still nothing seems to being cached though. ---------------------------------------------------- Dave Goodbourn Head of Systems MILK VISUAL EFFECTS [http://www.milk-vfx.com/src/milk_email_logo.jpg] 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 On 5 June 2017 at 14:49, Dave Goodbourn > wrote: Thanks Bob, That pagepool comment has just answered my next question! But it doesn't seem to be working. Here's my mmdiag output: === mmdiag: lroc === LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' status Running Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 Max capacity: 1151997 MB, currently in use: 0 MB Statistics from: Mon Jun 5 13:40:50 2017 Total objects stored 0 (0 MB) recalled 0 (0 MB) objects failed to store 0 failed to recall 0 failed to inval 0 objects queried 0 (0 MB) not found 0 = 0.00 % objects invalidated 0 (0 MB) Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Inode objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Directory objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Data objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 agent inserts=0, reads=0 response times (usec): insert min/max/avg=0/0/0 read min/max/avg=0/0/0 ssd writeIOs=0, writePages=0 readIOs=0, readPages=0 response times (usec): write min/max/avg=0/0/0 read min/max/avg=0/0/0 I've restarted GPFS on that node just in case but that didn't seem to help. I have LROC on a node that DOESN'T have direct access to an NSD so will hopefully cache files that get requested over NFS. How often are these stats updated? The Statistics line doesn't seem to update when running the command again. Dave, ---------------------------------------------------- Dave Goodbourn Head of Systems MILK VISUAL EFFECTS [http://www.milk-vfx.com/src/milk_email_logo.jpg] 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 On 5 June 2017 at 13:48, Oesterlin, Robert > wrote: Hi Dave I?ve done a large-scale (600 node) LROC deployment here - feel free to reach out if you have questions. mmdiag --lroc is about all there is but it does give you a pretty good idea how the cache is performing but you can?t tell which files are cached. Also, watch out that the LROC cached will steal pagepool memory (1% of the LROC cache size) Bob Oesterlin Sr Principal Storage Engineer, Nuance From: > on behalf of Dave Goodbourn > Reply-To: gpfsug main discussion list > Date: Monday, June 5, 2017 at 7:19 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] NSD access routes I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. From oehmes at gmail.com Mon Jun 5 14:59:44 2017 From: oehmes at gmail.com (Sven Oehme) Date: Mon, 05 Jun 2017 13:59:44 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: if you are using O_DIRECT calls they will be ignored by default for LROC, same for encrypted data. how exactly are you testing this? On Mon, Jun 5, 2017 at 6:50 AM Dave Goodbourn wrote: > Thanks Bob, > > That pagepool comment has just answered my next question! > > But it doesn't seem to be working. Here's my mmdiag output: > > === mmdiag: lroc === > LROC Device(s): > '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' > status Running > Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 > Max capacity: 1151997 MB, currently in use: 0 MB > Statistics from: Mon Jun 5 13:40:50 2017 > > Total objects stored 0 (0 MB) recalled 0 (0 MB) > objects failed to store 0 failed to recall 0 failed to inval 0 > objects queried 0 (0 MB) not found 0 = 0.00 % > objects invalidated 0 (0 MB) > > Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Inode objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Directory objects failed to store 0 failed to recall 0 failed to > query 0 failed to inval 0 > > Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Data objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > agent inserts=0, reads=0 > response times (usec): > insert min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > ssd writeIOs=0, writePages=0 > readIOs=0, readPages=0 > response times (usec): > write min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > > I've restarted GPFS on that node just in case but that didn't seem to > help. I have LROC on a node that DOESN'T have direct access to an NSD so > will hopefully cache files that get requested over NFS. > > How often are these stats updated? The Statistics line doesn't seem to > update when running the command again. > > Dave, > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 13:48, Oesterlin, Robert > wrote: > >> Hi Dave >> >> >> >> I?ve done a large-scale (600 node) LROC deployment here - feel free to >> reach out if you have questions. >> >> >> >> mmdiag --lroc is about all there is but it does give you a pretty good >> idea how the cache is performing but you can?t tell which files are cached. >> Also, watch out that the LROC cached will steal pagepool memory (1% of the >> LROC cache size) >> >> >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> >> >> >> >> *From: * on behalf of Dave >> Goodbourn >> *Reply-To: *gpfsug main discussion list > > >> *Date: *Monday, June 5, 2017 at 7:19 AM >> *To: *gpfsug main discussion list >> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >> >> >> >> I'm testing out the LROC idea. All seems to be working well, but, is >> there anyway to monitor what's cached? How full it might be? The >> performance etc?? >> >> >> >> I can see some stats in mmfsadm dump lroc but that's about it. >> >> >> >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 15:00:45 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 15:00:45 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: OK I'm going to hang my head in the corner...RTFM...I've not filled the memory buffer pool yet so I doubt it will have anything in it yet!! :( ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 14:55, Dave Goodbourn wrote: > OK slightly ignore that last email. It's still not updating the output but > I realise the Stats from line is when they started so probably won't > update! :( > > Still nothing seems to being cached though. > > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 14:49, Dave Goodbourn wrote: > >> Thanks Bob, >> >> That pagepool comment has just answered my next question! >> >> But it doesn't seem to be working. Here's my mmdiag output: >> >> === mmdiag: lroc === >> LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF >> 0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' status Running >> Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 >> Max capacity: 1151997 MB, currently in use: 0 MB >> Statistics from: Mon Jun 5 13:40:50 2017 >> >> Total objects stored 0 (0 MB) recalled 0 (0 MB) >> objects failed to store 0 failed to recall 0 failed to inval 0 >> objects queried 0 (0 MB) not found 0 = 0.00 % >> objects invalidated 0 (0 MB) >> >> Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >> Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >> Inode objects failed to store 0 failed to recall 0 failed to query >> 0 failed to inval 0 >> >> Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >> Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >> Directory objects failed to store 0 failed to recall 0 failed to >> query 0 failed to inval 0 >> >> Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >> Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >> Data objects failed to store 0 failed to recall 0 failed to query 0 >> failed to inval 0 >> >> agent inserts=0, reads=0 >> response times (usec): >> insert min/max/avg=0/0/0 >> read min/max/avg=0/0/0 >> >> ssd writeIOs=0, writePages=0 >> readIOs=0, readPages=0 >> response times (usec): >> write min/max/avg=0/0/0 >> read min/max/avg=0/0/0 >> >> >> I've restarted GPFS on that node just in case but that didn't seem to >> help. I have LROC on a node that DOESN'T have direct access to an NSD so >> will hopefully cache files that get requested over NFS. >> >> How often are these stats updated? The Statistics line doesn't seem to >> update when running the command again. >> >> Dave, >> ---------------------------------------------------- >> *Dave Goodbourn* >> Head of Systems >> *MILK VISUAL EFFECTS* >> >> 5th floor, Threeways House, >> 40-44 Clipstone Street London, W1W 5DW >> Tel: *+44 (0)20 3697 8448* >> Mob: *+44 (0)7917 411 069* >> >> On 5 June 2017 at 13:48, Oesterlin, Robert >> wrote: >> >>> Hi Dave >>> >>> >>> >>> I?ve done a large-scale (600 node) LROC deployment here - feel free to >>> reach out if you have questions. >>> >>> >>> >>> mmdiag --lroc is about all there is but it does give you a pretty good >>> idea how the cache is performing but you can?t tell which files are cached. >>> Also, watch out that the LROC cached will steal pagepool memory (1% of the >>> LROC cache size) >>> >>> >>> >>> Bob Oesterlin >>> Sr Principal Storage Engineer, Nuance >>> >>> >>> >>> >>> >>> >>> >>> *From: * on behalf of Dave >>> Goodbourn >>> *Reply-To: *gpfsug main discussion list >> org> >>> *Date: *Monday, June 5, 2017 at 7:19 AM >>> *To: *gpfsug main discussion list >>> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >>> >>> >>> >>> I'm testing out the LROC idea. All seems to be working well, but, is >>> there anyway to monitor what's cached? How full it might be? The >>> performance etc?? >>> >>> >>> >>> I can see some stats in mmfsadm dump lroc but that's about it. >>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Mon Jun 5 15:03:28 2017 From: oehmes at gmail.com (Sven Oehme) Date: Mon, 05 Jun 2017 14:03:28 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: yes as long as you haven't pushed anything to it (means pagepool got under enough pressure to free up space) you won't see anything in the stats :-) sven On Mon, Jun 5, 2017 at 7:00 AM Dave Goodbourn wrote: > OK I'm going to hang my head in the corner...RTFM...I've not filled the > memory buffer pool yet so I doubt it will have anything in it yet!! :( > > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 14:55, Dave Goodbourn wrote: > >> OK slightly ignore that last email. It's still not updating the output >> but I realise the Stats from line is when they started so probably won't >> update! :( >> >> Still nothing seems to being cached though. >> >> ---------------------------------------------------- >> *Dave Goodbourn* >> Head of Systems >> *MILK VISUAL EFFECTS* >> >> 5th floor, Threeways House, >> 40-44 Clipstone Street London, W1W 5DW >> Tel: *+44 (0)20 3697 8448* >> Mob: *+44 (0)7917 411 069* >> >> On 5 June 2017 at 14:49, Dave Goodbourn wrote: >> >>> Thanks Bob, >>> >>> That pagepool comment has just answered my next question! >>> >>> But it doesn't seem to be working. Here's my mmdiag output: >>> >>> === mmdiag: lroc === >>> LROC Device(s): >>> '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' >>> status Running >>> Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 >>> Max capacity: 1151997 MB, currently in use: 0 MB >>> Statistics from: Mon Jun 5 13:40:50 2017 >>> >>> Total objects stored 0 (0 MB) recalled 0 (0 MB) >>> objects failed to store 0 failed to recall 0 failed to inval 0 >>> objects queried 0 (0 MB) not found 0 = 0.00 % >>> objects invalidated 0 (0 MB) >>> >>> Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>> Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>> Inode objects failed to store 0 failed to recall 0 failed to query >>> 0 failed to inval 0 >>> >>> Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>> Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>> Directory objects failed to store 0 failed to recall 0 failed to >>> query 0 failed to inval 0 >>> >>> Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>> Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>> Data objects failed to store 0 failed to recall 0 failed to query >>> 0 failed to inval 0 >>> >>> agent inserts=0, reads=0 >>> response times (usec): >>> insert min/max/avg=0/0/0 >>> read min/max/avg=0/0/0 >>> >>> ssd writeIOs=0, writePages=0 >>> readIOs=0, readPages=0 >>> response times (usec): >>> write min/max/avg=0/0/0 >>> read min/max/avg=0/0/0 >>> >>> >>> I've restarted GPFS on that node just in case but that didn't seem to >>> help. I have LROC on a node that DOESN'T have direct access to an NSD so >>> will hopefully cache files that get requested over NFS. >>> >>> How often are these stats updated? The Statistics line doesn't seem to >>> update when running the command again. >>> >>> Dave, >>> ---------------------------------------------------- >>> *Dave Goodbourn* >>> Head of Systems >>> *MILK VISUAL EFFECTS* >>> >>> 5th floor, Threeways House, >>> 40-44 Clipstone Street London, W1W 5DW >>> Tel: *+44 (0)20 3697 8448* >>> Mob: *+44 (0)7917 411 069* >>> >>> On 5 June 2017 at 13:48, Oesterlin, Robert >>> wrote: >>> >>>> Hi Dave >>>> >>>> >>>> >>>> I?ve done a large-scale (600 node) LROC deployment here - feel free to >>>> reach out if you have questions. >>>> >>>> >>>> >>>> mmdiag --lroc is about all there is but it does give you a pretty good >>>> idea how the cache is performing but you can?t tell which files are cached. >>>> Also, watch out that the LROC cached will steal pagepool memory (1% of the >>>> LROC cache size) >>>> >>>> >>>> >>>> Bob Oesterlin >>>> Sr Principal Storage Engineer, Nuance >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *From: * on behalf of Dave >>>> Goodbourn >>>> *Reply-To: *gpfsug main discussion list < >>>> gpfsug-discuss at spectrumscale.org> >>>> *Date: *Monday, June 5, 2017 at 7:19 AM >>>> *To: *gpfsug main discussion list >>>> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >>>> >>>> >>>> >>>> I'm testing out the LROC idea. All seems to be working well, but, is >>>> there anyway to monitor what's cached? How full it might be? The >>>> performance etc?? >>>> >>>> >>>> >>>> I can see some stats in mmfsadm dump lroc but that's about it. >>>> >>>> >>>> >>>> >>> >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 15:15:00 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 15:15:00 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: Ha! A quick shrink of the pagepool and we're in action! Thanks all. Dave. ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 15:03, Sven Oehme wrote: > yes as long as you haven't pushed anything to it (means pagepool got under > enough pressure to free up space) you won't see anything in the stats :-) > > sven > > > On Mon, Jun 5, 2017 at 7:00 AM Dave Goodbourn wrote: > >> OK I'm going to hang my head in the corner...RTFM...I've not filled the >> memory buffer pool yet so I doubt it will have anything in it yet!! :( >> >> ---------------------------------------------------- >> *Dave Goodbourn* >> Head of Systems >> *MILK VISUAL EFFECTS* >> >> 5th floor, Threeways House, >> 40-44 Clipstone Street London, W1W 5DW >> Tel: *+44 (0)20 3697 8448* >> Mob: *+44 (0)7917 411 069* >> >> On 5 June 2017 at 14:55, Dave Goodbourn wrote: >> >>> OK slightly ignore that last email. It's still not updating the output >>> but I realise the Stats from line is when they started so probably won't >>> update! :( >>> >>> Still nothing seems to being cached though. >>> >>> ---------------------------------------------------- >>> *Dave Goodbourn* >>> Head of Systems >>> *MILK VISUAL EFFECTS* >>> >>> 5th floor, Threeways House, >>> 40-44 Clipstone Street London, W1W 5DW >>> Tel: *+44 (0)20 3697 8448* >>> Mob: *+44 (0)7917 411 069* >>> >>> On 5 June 2017 at 14:49, Dave Goodbourn wrote: >>> >>>> Thanks Bob, >>>> >>>> That pagepool comment has just answered my next question! >>>> >>>> But it doesn't seem to be working. Here's my mmdiag output: >>>> >>>> === mmdiag: lroc === >>>> LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' >>>> status Running >>>> Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 >>>> Max capacity: 1151997 MB, currently in use: 0 MB >>>> Statistics from: Mon Jun 5 13:40:50 2017 >>>> >>>> Total objects stored 0 (0 MB) recalled 0 (0 MB) >>>> objects failed to store 0 failed to recall 0 failed to inval 0 >>>> objects queried 0 (0 MB) not found 0 = 0.00 % >>>> objects invalidated 0 (0 MB) >>>> >>>> Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>>> Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>>> Inode objects failed to store 0 failed to recall 0 failed to >>>> query 0 failed to inval 0 >>>> >>>> Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>>> Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>>> Directory objects failed to store 0 failed to recall 0 failed to >>>> query 0 failed to inval 0 >>>> >>>> Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>>> Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>>> Data objects failed to store 0 failed to recall 0 failed to query >>>> 0 failed to inval 0 >>>> >>>> agent inserts=0, reads=0 >>>> response times (usec): >>>> insert min/max/avg=0/0/0 >>>> read min/max/avg=0/0/0 >>>> >>>> ssd writeIOs=0, writePages=0 >>>> readIOs=0, readPages=0 >>>> response times (usec): >>>> write min/max/avg=0/0/0 >>>> read min/max/avg=0/0/0 >>>> >>>> >>>> I've restarted GPFS on that node just in case but that didn't seem to >>>> help. I have LROC on a node that DOESN'T have direct access to an NSD so >>>> will hopefully cache files that get requested over NFS. >>>> >>>> How often are these stats updated? The Statistics line doesn't seem to >>>> update when running the command again. >>>> >>>> Dave, >>>> ---------------------------------------------------- >>>> *Dave Goodbourn* >>>> Head of Systems >>>> *MILK VISUAL EFFECTS* >>>> >>>> 5th floor, Threeways House, >>>> 40-44 Clipstone Street London, W1W 5DW >>>> Tel: *+44 (0)20 3697 8448* >>>> Mob: *+44 (0)7917 411 069* >>>> >>>> On 5 June 2017 at 13:48, Oesterlin, Robert >>> > wrote: >>>> >>>>> Hi Dave >>>>> >>>>> >>>>> >>>>> I?ve done a large-scale (600 node) LROC deployment here - feel free to >>>>> reach out if you have questions. >>>>> >>>>> >>>>> >>>>> mmdiag --lroc is about all there is but it does give you a pretty good >>>>> idea how the cache is performing but you can?t tell which files are cached. >>>>> Also, watch out that the LROC cached will steal pagepool memory (1% of the >>>>> LROC cache size) >>>>> >>>>> >>>>> >>>>> Bob Oesterlin >>>>> Sr Principal Storage Engineer, Nuance >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *From: * on behalf of Dave >>>>> Goodbourn >>>>> *Reply-To: *gpfsug main discussion list >>>> org> >>>>> *Date: *Monday, June 5, 2017 at 7:19 AM >>>>> *To: *gpfsug main discussion list >>>>> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >>>>> >>>>> >>>>> >>>>> I'm testing out the LROC idea. All seems to be working well, but, is >>>>> there anyway to monitor what's cached? How full it might be? The >>>>> performance etc?? >>>>> >>>>> >>>>> >>>>> I can see some stats in mmfsadm dump lroc but that's about it. >>>>> >>>>> >>>>> >>>>> >>>> >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 5 16:54:09 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 5 Jun 2017 15:54:09 +0000 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add Message-ID: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> Our node build process re-adds a node to the cluster and then does a ?service gpfs start?, but GPFS doesn?t start. From the build log: + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' + rc=0 + chkconfig gpfs on + service gpfs start The ?service gpfs start? command hangs and never seems to return. If I look at the process tree: [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12292 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e s/\.num$// 21639 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 This is GPFS 4.2.2-1 This seems to occur only on the initial startup after build - if I try to start GPFS again, it works just fine - any ideas on what it?s sitting here waiting? Nothing in mmfslog (does not exist) Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Mon Jun 5 20:54:31 2017 From: ewahl at osc.edu (Edward Wahl) Date: Mon, 5 Jun 2017 15:54:31 -0400 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add In-Reply-To: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> References: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> Message-ID: <20170605155431.75b42322@osc.edu> Just a thought, as we noticed the EXACT opposite of this, and what I think is new behavior in either mmmount or mmfsfuncs.. Does the file system exist in your /etc/fstab (or AIX equiv) yet? Ed On Mon, 5 Jun 2017 15:54:09 +0000 "Oesterlin, Robert" wrote: > Our node build process re-adds a node to the cluster and then does a ?service > gpfs start?, but GPFS doesn?t start. From the build log: > > + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com > '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' > + rc=0 > + chkconfig gpfs on > + service gpfs start > > The ?service gpfs start? command hangs and never seems to return. > > If I look at the process tree: > > [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" > 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post > 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 > 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? > S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S > 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > 12292 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num > 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e > s/\.num$// 21639 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 > > This is GPFS 4.2.2-1 > > This seems to occur only on the initial startup after build - if I try to > start GPFS again, it works just fine - any ideas on what it?s sitting here > waiting? Nothing in mmfslog (does not exist) > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From ewahl at osc.edu Mon Jun 5 20:56:55 2017 From: ewahl at osc.edu (Edward Wahl) Date: Mon, 5 Jun 2017 15:56:55 -0400 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add In-Reply-To: <20170605155431.75b42322@osc.edu> References: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> <20170605155431.75b42322@osc.edu> Message-ID: <20170605155655.3ce54084@osc.edu> On Mon, 5 Jun 2017 15:54:31 -0400 Edward Wahl wrote: > Just a thought, as we noticed the EXACT opposite of this, and what I think is > new behavior in either mmmount or .. Does the file system exist in > your /etc/fstab (or AIX equiv) yet? Apologies, I meant mmsdrfsdef, not mmfsfuncs. Ed > > Ed > > On Mon, 5 Jun 2017 15:54:09 +0000 > "Oesterlin, Robert" wrote: > > > Our node build process re-adds a node to the cluster and then does a > > ?service gpfs start?, but GPFS doesn?t start. >From the build log: > > > > + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com > > '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' > > + rc=0 > > + chkconfig gpfs on > > + service gpfs start > > > > The ?service gpfs start? command hangs and never seems to return. > > > > If I look at the process tree: > > > > [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" > > 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post > > 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 > > 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? > > S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S > > 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > > 12292 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > > 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num > > 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e > > s/\.num$// 21639 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 > > > > This is GPFS 4.2.2-1 > > > > This seems to occur only on the initial startup after build - if I try to > > start GPFS again, it works just fine - any ideas on what it?s sitting here > > waiting? Nothing in mmfslog (does not exist) > > > > Bob Oesterlin > > Sr Principal Storage Engineer, Nuance > > 507-269-0413 > > > > > > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From scale at us.ibm.com Mon Jun 5 22:49:23 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 5 Jun 2017 17:49:23 -0400 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add In-Reply-To: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> References: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> Message-ID: Looks like a bug in the code. The command hung in grep command. It has missing argument. Please open a PMR to have this fix. Instead of "service gpfs start", can you use mmstartup? You can also try to run mm list command before service gpfs start as a workaround. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/05/2017 11:54 AM Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add Sent by: gpfsug-discuss-bounces at spectrumscale.org Our node build process re-adds a node to the cluster and then does a ?service gpfs start?, but GPFS doesn?t start. From the build log: + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' + rc=0 + chkconfig gpfs on + service gpfs start The ?service gpfs start? command hangs and never seems to return. If I look at the process tree: [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12292 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e s/\.num$// 21639 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 This is GPFS 4.2.2-1 This seems to occur only on the initial startup after build - if I try to start GPFS again, it works just fine - any ideas on what it?s sitting here waiting? Nothing in mmfslog (does not exist) Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From stijn.deweirdt at ugent.be Tue Jun 6 08:05:06 2017 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Tue, 6 Jun 2017 09:05:06 +0200 Subject: [gpfsug-discuss] gpfs waiters debugging Message-ID: hi all, we have recently been hit by quite a few cases that triggered long waiters. we are aware of the excellent slides http://files.gpfsug.org/presentations/2017/NERSC/GPFS-Troubleshooting-Apr-2017.pdf but we are wondering if and how we can cause those waiters ourself, so we can train ourself in debugging and resolving them (either on test system or in controlled environment on the production clusters). all hints welcome. stijn From Robert.Oesterlin at nuance.com Tue Jun 6 12:44:31 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 6 Jun 2017 11:44:31 +0000 Subject: [gpfsug-discuss] gpfs waiters debugging Message-ID: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> Hi Stijn You need to provide some more details on the type and duration of the waiters before the group can offer some advice. Bob Oesterlin Sr Principal Storage Engineer, Nuance On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Stijn De Weirdt" wrote: but we are wondering if and how we can cause those waiters ourself, so we can train ourself in debugging and resolving them (either on test system or in controlled environment on the production clusters). all hints welcome. stijn _______________________________________________ From stijn.deweirdt at ugent.be Tue Jun 6 13:29:43 2017 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Tue, 6 Jun 2017 14:29:43 +0200 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> Message-ID: <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> hi bob, waiters from RPC replies and/or threads waiting on mutex are most "popular". but my question is not how to resolve them, the question is how to create such a waiter so we can train ourself in grep and mmfsadm etc etc we want to recreate the waiters a few times, try out some things and either script or at least put instructions on our internal wiki what to do. the instructions in the slides are clear enough, but there are a lot of slides, and typically when this occurs offshift, you don't want to start with rereading the slides and wondering what to do next; let alone debug scripts ;) thanks, stijn On 06/06/2017 01:44 PM, Oesterlin, Robert wrote: > Hi Stijn > > You need to provide some more details on the type and duration of the waiters before the group can offer some advice. > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Stijn De Weirdt" wrote: > > > but we are wondering if and how we can cause those waiters ourself, so > we can train ourself in debugging and resolving them (either on test > system or in controlled environment on the production clusters). > > all hints welcome. > > stijn > _______________________________________________ > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From stockf at us.ibm.com Tue Jun 6 13:57:00 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 6 Jun 2017 08:57:00 -0400 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> Message-ID: Realize that generally any waiter under 1 second should be ignored. In an active GPFS system there are always waiters and the greater the use of the system likely the more waiters you will see. The point is waiters themselves are not an indication your system is having problems. As for creating them any steady level of activity against the file system should cause waiters to appear, though most should be of a short duration. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: Stijn De Weirdt To: gpfsug-discuss at spectrumscale.org Date: 06/06/2017 08:31 AM Subject: Re: [gpfsug-discuss] gpfs waiters debugging Sent by: gpfsug-discuss-bounces at spectrumscale.org hi bob, waiters from RPC replies and/or threads waiting on mutex are most "popular". but my question is not how to resolve them, the question is how to create such a waiter so we can train ourself in grep and mmfsadm etc etc we want to recreate the waiters a few times, try out some things and either script or at least put instructions on our internal wiki what to do. the instructions in the slides are clear enough, but there are a lot of slides, and typically when this occurs offshift, you don't want to start with rereading the slides and wondering what to do next; let alone debug scripts ;) thanks, stijn On 06/06/2017 01:44 PM, Oesterlin, Robert wrote: > Hi Stijn > > You need to provide some more details on the type and duration of the waiters before the group can offer some advice. > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Stijn De Weirdt" wrote: > > > but we are wondering if and how we can cause those waiters ourself, so > we can train ourself in debugging and resolving them (either on test > system or in controlled environment on the production clusters). > > all hints welcome. > > stijn > _______________________________________________ > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Tue Jun 6 14:06:57 2017 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Tue, 6 Jun 2017 15:06:57 +0200 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> Message-ID: oh sure, i meant waiters that last > 300 seconds or so (something that could trigger deadlock). obviously we're not interested in debugging the short ones, it's not that gpfs doesn't work or anything ;) stijn On 06/06/2017 02:57 PM, Frederick Stock wrote: > Realize that generally any waiter under 1 second should be ignored. In an > active GPFS system there are always waiters and the greater the use of the > system likely the more waiters you will see. The point is waiters > themselves are not an indication your system is having problems. > > As for creating them any steady level of activity against the file system > should cause waiters to appear, though most should be of a short duration. > > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > stockf at us.ibm.com > > > > From: Stijn De Weirdt > To: gpfsug-discuss at spectrumscale.org > Date: 06/06/2017 08:31 AM > Subject: Re: [gpfsug-discuss] gpfs waiters debugging > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > hi bob, > > waiters from RPC replies and/or threads waiting on mutex are most > "popular". > > but my question is not how to resolve them, the question is how to > create such a waiter so we can train ourself in grep and mmfsadm etc etc > > we want to recreate the waiters a few times, try out some things and > either script or at least put instructions on our internal wiki what to > do. > > the instructions in the slides are clear enough, but there are a lot of > slides, and typically when this occurs offshift, you don't want to start > with rereading the slides and wondering what to do next; let alone debug > scripts ;) > > thanks, > > stijn > > On 06/06/2017 01:44 PM, Oesterlin, Robert wrote: >> Hi Stijn >> >> You need to provide some more details on the type and duration of the > waiters before the group can offer some advice. >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of Stijn De Weirdt" stijn.deweirdt at ugent.be> wrote: >> >> >> but we are wondering if and how we can cause those waiters ourself, > so >> we can train ourself in debugging and resolving them (either on test >> system or in controlled environment on the production clusters). >> >> all hints welcome. >> >> stijn >> _______________________________________________ >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From valdis.kletnieks at vt.edu Tue Jun 6 17:45:51 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 06 Jun 2017 12:45:51 -0400 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> Message-ID: <6873.1496767551@turing-police.cc.vt.edu> On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt said: > oh sure, i meant waiters that last > 300 seconds or so (something that > could trigger deadlock). obviously we're not interested in debugging the > short ones, it's not that gpfs doesn't work or anything ;) At least at one time, a lot of the mm(whatever) administrative commands would leave one dangling waiter for the duration of the command - which could be a while if the command was mmdeldisk or mmrestripefs. I admit not having specifically checked for gpfs 4.2, but it was true for 3.2 through 4.1.... And my addition to the collective debugging knowledge: A bash one-liner to dump all the waiters across a cluster, sorted by wait time. Note that our clusters tend to be 5-8 servers, this may be painful for those of you who have 400+ node clusters. :) ##!/bin/bash for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh $i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done | sort -n -r -k 3 -t' ' We've found it useful - if you have 1 waiter on one node that's 1278 seconds old, and 3 other nodes have waiters that are 1275 seconds old, it's a good chance the other 3 nodes waiters are waiting on the first node's waiter to resolve itself.... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From stockf at us.ibm.com Tue Jun 6 17:54:06 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 6 Jun 2017 12:54:06 -0400 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: <6873.1496767551@turing-police.cc.vt.edu> References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com><3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> <6873.1496767551@turing-police.cc.vt.edu> Message-ID: On recent releases you can accomplish the same with the command, "mmlsnode -N waiters -L". Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list Date: 06/06/2017 12:46 PM Subject: Re: [gpfsug-discuss] gpfs waiters debugging Sent by: gpfsug-discuss-bounces at spectrumscale.org On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt said: > oh sure, i meant waiters that last > 300 seconds or so (something that > could trigger deadlock). obviously we're not interested in debugging the > short ones, it's not that gpfs doesn't work or anything ;) At least at one time, a lot of the mm(whatever) administrative commands would leave one dangling waiter for the duration of the command - which could be a while if the command was mmdeldisk or mmrestripefs. I admit not having specifically checked for gpfs 4.2, but it was true for 3.2 through 4.1.... And my addition to the collective debugging knowledge: A bash one-liner to dump all the waiters across a cluster, sorted by wait time. Note that our clusters tend to be 5-8 servers, this may be painful for those of you who have 400+ node clusters. :) ##!/bin/bash for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh $i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done | sort -n -r -k 3 -t' ' We've found it useful - if you have 1 waiter on one node that's 1278 seconds old, and 3 other nodes have waiters that are 1275 seconds old, it's a good chance the other 3 nodes waiters are waiting on the first node's waiter to resolve itself.... [attachment "attltepl.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Tue Jun 6 19:05:15 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 6 Jun 2017 14:05:15 -0400 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) In-Reply-To: <20170602111241.56882fx2qr2yz2ax@support.scinet.utoronto.ca> References: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca>, <20170602111241.56882fx2qr2yz2ax@support.scinet.utoronto.ca> Message-ID: Hi, Just as Jaime has explained, any GPFS node in the cluster, can induce a recall (as he called "staged") by access to file data. It is not optimized by tape order, and a dynamic file access of any pattern, such as "find" or "cat *" will surely result in an inefficient processing of the data recall if all data lives in physical tape. But if migrated data lives on spinning disk on the TSM server, there is no harm in such a recall pattern because recalls from a disk pool incur no significant overhead or delay for tape loading and positioning. Unprivileged users may not run "dsmcrecall" because creating a DMAPI session as the dsmrecall program must do, requires admin user privilege on that node. You may be able to wrap dsmrecall in a set-uid wrapper if you want to permit users to run that, but of course that comes with the danger that a recall storm could monopolize resources on your cluster. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Jaime Pinto" To: "Andrew Beattie" Cc: gpfsug-discuss at spectrumscale.org Date: 06/02/2017 11:13 AM Subject: Re: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) Sent by: gpfsug-discuss-bounces at spectrumscale.org It has been a while since I used HSM with GPFS via TSM, but as far as I can remember, unprivileged users can run dsmmigrate and dsmrecall. Based on the instructions on the link, dsmrecall may now leverage the Recommended Access Order (RAO) available on enterprise drives, however root would have to be the one to invoke that feature. In that case we may have to develop a middleware/wrapper for dsmrecall that will run as root and act on behalf of the user when optimization is requested. Someone here more familiar with the latest version of TSM-HSM may be able to give us some hints on how people are doing this in practice. Jaime Quoting "Andrew Beattie" : > Thanks Jaime, How do you get around Optimised recalls? from what I > can see the optimised recall process needs a root level account to > retrieve a list of files > https://www.ibm.com/support/knowledgecenter/SSSR2R_7.1.1/com.ibm.itsm.hsmul.doc/c_recall_optimized_tape.html [1] > Regards, Andrew Beattie Software Defined Storage - IT Specialist > Phone: 614-2133-7927 E-mail: abeattie at au1.ibm.com[2] ----- > Original message ----- > From: "Jaime Pinto" > To: "gpfsug main discussion list" , > "Andrew Beattie" > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - > Space Management (GPFS HSM) > Date: Fri, Jun 2, 2017 7:28 PM > We have that situation. > Users don't need to login to NSD's > > What you need is to add at least one gpfs client to the cluster (or > multi-cluster), mount the DMAPI enabled file system, and use that > node > as a gateway for end-users. They can access the contents on the mount > > point with their own underprivileged accounts. > > Whether or not on a schedule, the moment an application or linux > command (such as cp, cat, vi, etc) accesses a stub, the file will be > > staged. > > Jaime > > Quoting "Andrew Beattie" : > >> Quick question, Does anyone have a Scale / GPFS environment (HPC) >> where users need the ability to recall data sets after they have > been >> stubbed, but only System Administrators are permitted to log onto > the >> NSD servers for security purposes. And if so how do you provide > the >> ability for the users to schedule their data set recalls? > Regards, >> Andrew Beattie Software Defined Storage - IT Specialist Phone: >> 614-2133-7927 E-mail: abeattie at au1.ibm.com[1] >> >> >> Links: >> ------ >> [1] mailto:abeattie at au1.ibm.com[3] >> > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials[4] > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jun 6 19:15:22 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 6 Jun 2017 18:15:22 +0000 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> <6873.1496767551@turing-police.cc.vt.edu> Message-ID: All, mmlsnode -N waiters is great ? I also appreciate the ?-s? option to it. Very helpful when you know the problem started say, slightly more than half an hour ago and you therefore don?t care about sub-1800 second waiters? Kevin On Jun 6, 2017, at 11:54 AM, Frederick Stock > wrote: On recent releases you can accomplish the same with the command, "mmlsnode -N waiters -L". Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list > Date: 06/06/2017 12:46 PM Subject: Re: [gpfsug-discuss] gpfs waiters debugging Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt said: > oh sure, i meant waiters that last > 300 seconds or so (something that > could trigger deadlock). obviously we're not interested in debugging the > short ones, it's not that gpfs doesn't work or anything ;) At least at one time, a lot of the mm(whatever) administrative commands would leave one dangling waiter for the duration of the command - which could be a while if the command was mmdeldisk or mmrestripefs. I admit not having specifically checked for gpfs 4.2, but it was true for 3.2 through 4.1.... And my addition to the collective debugging knowledge: A bash one-liner to dump all the waiters across a cluster, sorted by wait time. Note that our clusters tend to be 5-8 servers, this may be painful for those of you who have 400+ node clusters. :) ##!/bin/bash for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh $i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done | sort -n -r -k 3 -t' ' We've found it useful - if you have 1 waiter on one node that's 1278 seconds old, and 3 other nodes have waiters that are 1275 seconds old, it's a good chance the other 3 nodes waiters are waiting on the first node's waiter to resolve itself.... [attachment "attltepl.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Tue Jun 6 21:31:01 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 06 Jun 2017 16:31:01 -0400 Subject: [gpfsug-discuss] mmapplypolicy and ltfsee - identifying progress... Message-ID: <25944.1496781061@turing-police.cc.vt.edu> So I'm trying to get a handle on where exactly an mmapplypolicy that's doing a premigrate is in its progress. I've already determined that 'ltfsee info jobs' will only report where in the current batch it is, but that still leaves me unable to tell the difference between [I] 2017-06-05 at 17:31:47.995 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 10000 files dispatched. and [I] 2017-06-06 at 02:44:48.236 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 225000 files dispatched. Is there any better way to figure out where it is than writing the cron job to launch it as mmapplypolicy | tee /tmp/something and then go scraping the messages? (And yes, I know not all chunks of 1,000 files are created equal. Sometimes it's 1,000 C source files that total to less than a megabyte, other times it's 1,000 streaming video files that total to over a terabye - but even knowing it's 194,000 into 243,348 files is better than what I have now...) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jun 6 22:20:31 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 6 Jun 2017 21:20:31 +0000 Subject: [gpfsug-discuss] mmapplypolicy and ltfsee - identifying progress... In-Reply-To: <25944.1496781061@turing-police.cc.vt.edu> References: <25944.1496781061@turing-police.cc.vt.edu> Message-ID: <5987990A-39F6-47A5-981D-A34A3054E4D8@vanderbilt.edu> Hi Valdis, I?m not sure this is ?better?, but what I typically do is have mmapplypolicy running from a shell script launched by a cron job and redirecting output to a file in /tmp. Once the mmapplypolicy finishes the SysAdmin?s get the tmp file e-mailed to them and then it gets deleted. Of course, while the mmapplypolicy is running you can ?tail -f /tmp/mmapplypolicy.log? or grep it or whatever. HTHAL? Kevin On Jun 6, 2017, at 3:31 PM, valdis.kletnieks at vt.edu wrote: So I'm trying to get a handle on where exactly an mmapplypolicy that's doing a premigrate is in its progress. I've already determined that 'ltfsee info jobs' will only report where in the current batch it is, but that still leaves me unable to tell the difference between [I] 2017-06-05 at 17:31:47.995 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 10000 files dispatched. and [I] 2017-06-06 at 02:44:48.236 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 225000 files dispatched. Is there any better way to figure out where it is than writing the cron job to launch it as mmapplypolicy | tee /tmp/something and then go scraping the messages? (And yes, I know not all chunks of 1,000 files are created equal. Sometimes it's 1,000 C source files that total to less than a megabyte, other times it's 1,000 streaming video files that total to over a terabye - but even knowing it's 194,000 into 243,348 files is better than what I have now...) ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From TROPPENS at de.ibm.com Wed Jun 7 09:30:17 2017 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Wed, 7 Jun 2017 08:30:17 +0000 Subject: [gpfsug-discuss] ISC 2017 - Agenda User Meeting Message-ID: An HTML attachment was scrubbed... URL: From jan.sundermann at kit.edu Wed Jun 7 15:04:28 2017 From: jan.sundermann at kit.edu (Sundermann, Jan Erik (SCC)) Date: Wed, 7 Jun 2017 14:04:28 +0000 Subject: [gpfsug-discuss] Upgrade with architecture change Message-ID: Hi, we are operating a small Spectrum Scale cluster with about 100 clients and 6 NSD servers. The cluster is FPO-enabled. For historical reasons the NSD servers are running on ppc64 while the clients are a mixture of ppc64le and x86_64 machines. Most machines are running Red Hat Enterprise Linux 7 but we also have few machines running AIX. At the moment we have installed Spectrum Scale version 4.1.1 but would like to do an upgrade to 4.2.3. In the course of the upgrade we would like to change the architecture of all NSD servers and reinstall them with ppc64le instead of ppc64. From what I?ve learned so far it should be possible to upgrade directly from 4.1.1 to 4.2.3. Before doing the upgrade we would like to ask for some advice on the best strategy. For the NSD servers, one by one, we are thinking about doing the following: 1) Disable auto recovery 2) Unmount GPFS file system 3) Suspend disks 4) Shutdown gpfs 5) Reboot and reinstall with changed architecture ppc64le 6) Install gpfs 4.2.3 7) Recover cluster config using mmsdrrestore 8) Resume and start disks 9) Reenable auto recovery Can GPFS handle the change of the NSD server?s architecture and would it be fine to operate a mixture of different architectures for the NSD servers? Thanks, Jan Erik From tarak.patel at canada.ca Wed Jun 7 16:42:45 2017 From: tarak.patel at canada.ca (Patel, Tarak (SSC/SPC)) Date: Wed, 7 Jun 2017 15:42:45 +0000 Subject: [gpfsug-discuss] Remote cluster gpfs communication on IP different then one for Daemon or Admin node name. Message-ID: <50fd0dc6cf47485c8728fc09b7ae0263@PEVDACDEXC009.birch.int.bell.ca> Hi all, We've been experiencing issues with remote cluster node expelling CES nodes causing remote filesystems to unmount. The issue is related gpfs communication using Ethernet IP rather than IP defined on IB which is used for Daemon node name and Admin node name. So remote cluster is aware of IPs that are not defined in GPFS configuration as Admin/Daemon node name. The CES nodes are configure to have IB as well as Ethernet (for client interactive and NFS access). We've double checked /etc/hosts and DNS and all looks to be in order since the CES IPoIB IP is present in /etc/hosts of remote cluster. I'm unsure where cluster manager for remote cluster is getting the Ethernet IP if there is no mention of it in GPFS configuration. The CES nodes were added later therefore they are not listed as Contact Nodes in 'mmremotecluster show' output. The CES nodes use IP defined on IB for GPFS configuration and we also have Ethernet which has the default route defined. In order to ensure that all IB communication passes via IPoIB, we've even defined a static route so that all GPFS communication will use IPoIB (since we are dealing with a different fabric). 'mmfsadm dump tscomm' reports multiple IPs for CES nodes which includes the Ethernet and also the IPoIB. I'm unsure if there is a way to drop some connections on GPFS (cluster wide) after stopping a specific CES node and ensure that only IB is listed. I realize that one option would be to define subnet parameter for remote cluster which will require a downtime (solution to be explored at later date). Hope that someone can explain how or why remote cluster is picking IPs not used in GPFS config for remote nodes and how to ensure those IPs are not used in future. Thank you, Tarak -- Tarak Patel Chef d'?quipe, Integration HPC, Solution de calcul E-Science Service partag? Canada / Gouvernment du Canada tarak.patel at canada.ca 1-514-421-7299 Team Lead, HPC Integration, E-Science Computing Solution Shared Services Canada, Government of Canada tarak.patel at canada.ca 1-514-421-7299 -------------- next part -------------- An HTML attachment was scrubbed... URL: From chekh at stanford.edu Wed Jun 7 23:12:56 2017 From: chekh at stanford.edu (Alex Chekholko) Date: Wed, 7 Jun 2017 15:12:56 -0700 Subject: [gpfsug-discuss] Upgrade with architecture change In-Reply-To: References: Message-ID: Hi Jan, I don't have hands-on experience with FPO or ppc64 but your procedure sounds OK to me. How do you currently handle just shutting down an NSD node for maintenance? I guess you'd have the same process except skip 5,6,7 How do you currently handle OS rebuild on NSD node? Maybe try that first without the architecture change. But I don't see why it would matter so long as you don't touch the GPFS disks. Regards, Alex On 06/07/2017 07:04 AM, Sundermann, Jan Erik (SCC) wrote: > Hi, > > we are operating a small Spectrum Scale cluster with about 100 clients and 6 NSD servers. The cluster is FPO-enabled. For historical reasons the NSD servers are running on ppc64 while the clients are a mixture of ppc64le and x86_64 machines. Most machines are running Red Hat Enterprise Linux 7 but we also have few machines running AIX. > > At the moment we have installed Spectrum Scale version 4.1.1 but would like to do an upgrade to 4.2.3. In the course of the upgrade we would like to change the architecture of all NSD servers and reinstall them with ppc64le instead of ppc64. > > From what I?ve learned so far it should be possible to upgrade directly from 4.1.1 to 4.2.3. Before doing the upgrade we would like to ask for some advice on the best strategy. > > For the NSD servers, one by one, we are thinking about doing the following: > > 1) Disable auto recovery > 2) Unmount GPFS file system > 3) Suspend disks > 4) Shutdown gpfs > 5) Reboot and reinstall with changed architecture ppc64le > 6) Install gpfs 4.2.3 > 7) Recover cluster config using mmsdrrestore > 8) Resume and start disks > 9) Reenable auto recovery > > Can GPFS handle the change of the NSD server?s architecture and would it be fine to operate a mixture of different architectures for the NSD servers? > > > Thanks, > Jan Erik From Philipp.Rehs at uni-duesseldorf.de Thu Jun 8 10:35:57 2017 From: Philipp.Rehs at uni-duesseldorf.de (Philipp Helo Rehs) Date: Thu, 8 Jun 2017 11:35:57 +0200 Subject: [gpfsug-discuss] GPFS for aarch64? Message-ID: <5848d2c0-d526-3d81-a469-6b7a10b9bf3a@uni-duesseldorf.de> Hello, we got a Cavium ThunderX-based Server and would like to use GPFS on it. Are the any package for gpfs on aarch64? Kind regards Philipp Rehs --------------------------- Zentrum f?r Informations- und Medientechnologie Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern Heinrich-Heine-Universit?t D?sseldorf Universit?tsstr. 1 Raum 25.41.00.51 40225 D?sseldorf / Germany Tel: +49-211-81-15557 From abeattie at au1.ibm.com Thu Jun 8 10:45:38 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Thu, 8 Jun 2017 09:45:38 +0000 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: <5848d2c0-d526-3d81-a469-6b7a10b9bf3a@uni-duesseldorf.de> References: <5848d2c0-d526-3d81-a469-6b7a10b9bf3a@uni-duesseldorf.de> Message-ID: An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Thu Jun 8 10:54:15 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 8 Jun 2017 09:54:15 +0000 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: Message-ID: And Linux on Z/VM If interested feel free to open a RFE -- Cheers > On 8 Jun 2017, at 12.46, Andrew Beattie wrote: > > Philipp, > > Not to my knowledge, > > AIX > Linux on x86 ( RHEL / SUSE / Ubuntu) > Linux on Power (RHEL / SUSE) > WIndows > > are the current supported platforms > Andrew Beattie > Software Defined Storage - IT Specialist > Phone: 614-2133-7927 > E-mail: abeattie at au1.ibm.com > > > ----- Original message ----- > From: Philipp Helo Rehs > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [gpfsug-discuss] GPFS for aarch64? > Date: Thu, Jun 8, 2017 7:36 PM > > Hello, > > we got a Cavium ThunderX-based Server and would like to use GPFS on it. > > Are the any package for gpfs on aarch64? > > > Kind regards > > Philipp Rehs > > --------------------------- > > Zentrum f?r Informations- und Medientechnologie > Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern > > Heinrich-Heine-Universit?t D?sseldorf > Universit?tsstr. 1 > Raum 25.41.00.51 > 40225 D?sseldorf / Germany > Tel: +49-211-81-15557 > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From Philipp.Rehs at uni-duesseldorf.de Thu Jun 8 11:40:23 2017 From: Philipp.Rehs at uni-duesseldorf.de (Philipp Helo Rehs) Date: Thu, 8 Jun 2017 12:40:23 +0200 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: References: Message-ID: <9f47c897-74ff-9473-2ab3-343e4ce69d15@uni-duesseldorf.de> Thanks for the Information. I created an RFE: https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=106218 Kind regards, Philipp Rehs > Message: 6 > Date: Thu, 8 Jun 2017 09:54:15 +0000 > From: "Luis Bolinches" > To: "gpfsug main discussion list" > Subject: Re: [gpfsug-discuss] GPFS for aarch64? > Message-ID: > > > Content-Type: text/plain; charset="utf-8" > > And Linux on Z/VM > > If interested feel free to open a RFE > > -- > Cheers > >> On 8 Jun 2017, at 12.46, Andrew Beattie wrote: >> >> Philipp, >> >> Not to my knowledge, >> >> AIX >> Linux on x86 ( RHEL / SUSE / Ubuntu) >> Linux on Power (RHEL / SUSE) >> WIndows >> >> are the current supported platforms >> Andrew Beattie >> Software Defined Storage - IT Specialist >> Phone: 614-2133-7927 >> E-mail: abeattie at au1.ibm.com >> >> >> ----- Original message ----- >> From: Philipp Helo Rehs >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: >> Subject: [gpfsug-discuss] GPFS for aarch64? >> Date: Thu, Jun 8, 2017 7:36 PM >> >> Hello, >> >> we got a Cavium ThunderX-based Server and would like to use GPFS on it. >> >> Are the any package for gpfs on aarch64? >> >> >> Kind regards >> >> Philipp Rehs >> >> --------------------------- >> >> Zentrum f?r Informations- und Medientechnologie >> Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern >> >> Heinrich-Heine-Universit?t D?sseldorf >> Universit?tsstr. 1 >> Raum 25.41.00.51 >> 40225 D?sseldorf / Germany >> Tel: +49-211-81-15557 >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 65, Issue 17 > ********************************************** > From daniel.kidger at uk.ibm.com Thu Jun 8 11:54:04 2017 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 8 Jun 2017 10:54:04 +0000 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: Message-ID: I often hear requests for Spectrum Scale on ARM. It is always for clients. In general people are happy to have their NSD servers, etc. on x86 or POWER. It is also an anomaly that for a HPC cluster, IBM supports LSF on ARM v7/v8 but not Spectrum Scale on ARM. Daniel Daniel Kidger Technical Sales Specialist, IBM UK IBM Spectrum Storage Software daniel.kidger at uk.ibm.com +44 (0)7818 522266 > On 8 Jun 2017, at 10:54, Luis Bolinches wrote: > > And Linux on Z/VM > > If interested feel free to open a RFE > > -- > Cheers > >> On 8 Jun 2017, at 12.46, Andrew Beattie wrote: >> >> Philipp, >> >> Not to my knowledge, >> >> AIX >> Linux on x86 ( RHEL / SUSE / Ubuntu) >> Linux on Power (RHEL / SUSE) >> WIndows >> >> are the current supported platforms >> Andrew Beattie >> Software Defined Storage - IT Specialist >> Phone: 614-2133-7927 >> E-mail: abeattie at au1.ibm.com >> >> >> ----- Original message ----- >> From: Philipp Helo Rehs >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: >> Subject: [gpfsug-discuss] GPFS for aarch64? >> Date: Thu, Jun 8, 2017 7:36 PM >> >> Hello, >> >> we got a Cavium ThunderX-based Server and would like to use GPFS on it. >> >> Are the any package for gpfs on aarch64? >> >> >> Kind regards >> >> Philipp Rehs >> >> --------------------------- >> >> Zentrum f?r Informations- und Medientechnologie >> Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern >> >> Heinrich-Heine-Universit?t D?sseldorf >> Universit?tsstr. 1 >> Raum 25.41.00.51 >> 40225 D?sseldorf / Germany >> Tel: +49-211-81-15557 >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From duersch at us.ibm.com Thu Jun 8 15:09:03 2017 From: duersch at us.ibm.com (Steve Duersch) Date: Thu, 8 Jun 2017 10:09:03 -0400 Subject: [gpfsug-discuss] Upgrade with architecture change In-Reply-To: References: Message-ID: We have not tested such a procedure. The only route that we have done is a complete mmdelnode/mmaddnode scenario. This would mean an mmdeldisk. It would be more time consuming since data has to move. Operating in a mixed architecture environment is not a problem. We have tested and support that. Steve Duersch Spectrum Scale 845-433-7902 IBM Poughkeepsie, New York > > Message: 1 > Date: Wed, 7 Jun 2017 14:04:28 +0000 > From: "Sundermann, Jan Erik (SCC)" > To: "gpfsug-discuss at spectrumscale.org" > > Subject: [gpfsug-discuss] Upgrade with architecture change > Message-ID: > Content-Type: text/plain; charset="utf-8" > > Hi, > > we are operating a small Spectrum Scale cluster with about 100 > clients and 6 NSD servers. The cluster is FPO-enabled. For > historical reasons the NSD servers are running on ppc64 while the > clients are a mixture of ppc64le and x86_64 machines. Most machines > are running Red Hat Enterprise Linux 7 but we also have few machines > running AIX. > > At the moment we have installed Spectrum Scale version 4.1.1 but > would like to do an upgrade to 4.2.3. In the course of the upgrade > we would like to change the architecture of all NSD servers and > reinstall them with ppc64le instead of ppc64. > > From what I?ve learned so far it should be possible to upgrade > directly from 4.1.1 to 4.2.3. Before doing the upgrade we would like > to ask for some advice on the best strategy. > > For the NSD servers, one by one, we are thinking about doing the following: > > 1) Disable auto recovery > 2) Unmount GPFS file system > 3) Suspend disks > 4) Shutdown gpfs > 5) Reboot and reinstall with changed architecture ppc64le > 6) Install gpfs 4.2.3 > 7) Recover cluster config using mmsdrrestore > 8) Resume and start disks > 9) Reenable auto recovery > > Can GPFS handle the change of the NSD server?s architecture and > would it be fine to operate a mixture of different architectures for > the NSD servers? > > > Thanks, > Jan Erik > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Thu Jun 8 17:01:07 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 8 Jun 2017 12:01:07 -0400 Subject: [gpfsug-discuss] Upgrade with architecture change In-Reply-To: References: Message-ID: If you proceed carefully, it should not be necessary to mmdeldisk and mmadddisks. Although we may not have tested your exact scenario, GPFS does support fiber channel disks attached to multiple nodes. So the same disk can be attached to multiple GPFS nodes - and those nodes can be running different OSes and different GPFS versions. (That's something we do actually test!) Since GPFS can handle that with several nodes simultaneously active -- it can also handle the case when nodes come and go... Or in your case are killed and then reborn with new software... The key is to be careful... You want to unmount the file system and not re-mount until all of the disks become available again via one or more (NSD) nodes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Thu Jun 8 22:34:15 2017 From: Mark.Bush at siriuscom.com (Mark Bush) Date: Thu, 8 Jun 2017 21:34:15 +0000 Subject: [gpfsug-discuss] LROC/HAWC for CES nodes? Message-ID: <288751C9-7CB6-48E6-968E-938A4E56E786@siriuscom.com> I?m looking to improve performance of the SMB stack. My workload unfortunately has smallish files but in total it will still be large amount. I?m wondering if LROC/HAWC would be one way to speed things up. Is there a precedent for using this with protocol nodes in a cluster? Anyone else thinking/doing this? Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From TROPPENS at de.ibm.com Fri Jun 9 08:44:57 2017 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Fri, 9 Jun 2017 09:44:57 +0200 Subject: [gpfsug-discuss] ISC 2017 - Agenda User Meeting In-Reply-To: References: Message-ID: There is an update of the agenda for the User Meeting at ISC. We have added a Pawsey Site Report by Chris Schlipalius. Monday June 19, 2016 - 12:00-14:30 - Conference Room Konstant 12:00-12:10 ?[10 min] ?Opening 12:10-12:25 ?[15 min] ?Spectrum Scale Support for Docker - Olaf Weiser (IBM) 12:25-13:05 ?[40 min] ?IBM Spectrum LSF family update - Bill McMillan (IBM) 13:05-13:25 ?[20 min] ?Driving Operational Efficiencies with the IBM Spectrum LSF & Ellexus Mistral - Dr. Rosemary Francis (Ellexus) 13:25-13:40 [15 min] Pawsey Site Report - Chris Schlipalius (Pawsey) 13:40-13:55 ?[15 min] ?IBM Elastic Storage Server (ESS) Update - John Sing (IBM) 13:55-14:20 ?[25 min] ?IBM Spectrum Scale Enhancements for CORAL - Sven Oehme (IBM) 14:20-14:30 ?[10 min] ?Question & Answers -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Ulf Troppens" To: gpfsug-discuss at spectrumscale.org Cc: Fabienne Wegener Date: 07.06.2017 10:30 Subject: [gpfsug-discuss] ISC 2017 - Agenda User Meeting Sent by: gpfsug-discuss-bounces at spectrumscale.org Greetings: IBM is happy to announce the agenda for the joint IBM Spectrum Scale and IBM Spectrum LSF User Meeting at ISC. As with other user meetings, the agenda includes user stories, updates on IBM Spectrum Scale and IBM Spectrum LSF, and access to IBM experts and your peers. Please join us! To attend, please email Fabienne.Wegener at de.ibm.com so we can have an accurate count of attendees. Monday June 17, 2016 - 12:00-14:30 - Conference Room Konstant 12:00-12:10 [10 min] Opening 12:10-12:30 [20 min] Spectrum Scale Support for Docker - Olaf Weiser (IBM) 12:30-13:10 [40 min] IBM Spectrum LSF family update - Bill McMillan (IBM) 13:10-13:30 [20 min] Driving Operational Efficiencies with the IBM Spectrum LSF & Ellexus Mistral - Dr. Rosemary Francis (Ellexus) 13:30-13:50 [20 min] IBM Elastic Storage Server (ESS) Update - John Sing (IBM) 13:50-14:20 [30 min] IBM Spectrum Scale Enhancements for CORAL - Sven Oehme (IBM) 14:20-14:30 [10 min] Question & Answers Looking forward to seeing you there! -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Fri Jun 9 09:38:01 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 9 Jun 2017 08:38:01 +0000 Subject: [gpfsug-discuss] LROC/HAWC for CES nodes? In-Reply-To: <288751C9-7CB6-48E6-968E-938A4E56E786@siriuscom.com> References: <288751C9-7CB6-48E6-968E-938A4E56E786@siriuscom.com> Message-ID: I?m wary of spending a lot of money on LROC devices when I don?t know what return I will get.. that said I think the main bottleneck for any SMB installation is samba itself, not the disks, so I remain largely unconvinced that LROC will help much. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mark Bush Sent: 08 June 2017 22:34 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] LROC/HAWC for CES nodes? I?m looking to improve performance of the SMB stack. My workload unfortunately has smallish files but in total it will still be large amount. I?m wondering if LROC/HAWC would be one way to speed things up. Is there a precedent for using this with protocol nodes in a cluster? Anyone else thinking/doing this? Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Fri Jun 9 10:31:45 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 9 Jun 2017 09:31:45 +0000 Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS In-Reply-To: References: Message-ID: Thanks Mark, didn?t mean to wait so long to reply. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Marc A Kaplan Sent: 02 June 2017 17:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] TSM/SP compatibility with GPFS Upgrading from GPFS 4.2.x to GPFS 4.2.y should not "break" TSM. If it does, someone goofed, that would be a bug. (My opinion) Think of it this way. TSM is an application that uses the OS and the FileSystem(s). TSM can't verify it will work with all future versions of OS and Filesystems, and the releases can't be in lock step. Having said that, 4.2.3 has been "out" for a while, so if there were a TSM incompatibility, someone would have likely hit it or will before July... Trust but verify... From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 06/02/2017 11:51 AM Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi all, Where should I start looking for a compatibility matrix between TSM and GPFS? Specifically, we are currently running TSM 7.1.6-2 and GPFS 4.2.1-2 with the intent to upgrade to GPFS 4.2.3-latest in early July. I?ve spent 30 minutes looking over various documents and the best I can find is this: http://www-01.ibm.com/support/docview.wss?uid=swg21248771 ..which talks about TSM in a Space Management context and would suggest that we need to upgrade to Spectrum Protect i.e. 8.1 and that GPFS 4.2.2.x is the maximum supported version? Cheers Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.tower at outlook.com Sat Jun 10 11:55:54 2017 From: frank.tower at outlook.com (Frank Tower) Date: Sat, 10 Jun 2017 10:55:54 +0000 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found Message-ID: Hi everybody, I don't get why one of our compute node cannot start GPFS over IB. I have the following error: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). [I] VERBS RDMA parse verbsPorts mlx4_0/1 [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found [I] VERBS RDMA library libibverbs.so unloaded. [E] VERBS RDMA failed to start, no valid verbsPorts defined. I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. I have 2 infinibands card, both have an IP and working well. [root at rdx110 ~]# ibstat -l mlx4_0 mlx4_1 [root at rdx110 ~]# I tried configuration with both card, and no one work with GPFS. I also tried with mlx4_0/1, but same problem. Someone already have the issue ? Kind Regards, Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Sat Jun 10 13:05:04 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Sat, 10 Jun 2017 08:05:04 -0400 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found In-Reply-To: References: Message-ID: Out of curiosity could you send us the output of "ibv_devinfo -v"? -Aaron Sent from my iPhone > On Jun 10, 2017, at 06:55, Frank Tower wrote: > > Hi everybody, > > > I don't get why one of our compute node cannot start GPFS over IB. > > > I have the following error: > > > [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes > [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. > [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). > > [I] VERBS RDMA parse verbsPorts mlx4_0/1 > > [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found > > [I] VERBS RDMA library libibverbs.so unloaded. > > [E] VERBS RDMA failed to start, no valid verbsPorts defined. > > > > I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. > > > I have 2 infinibands card, both have an IP and working well. > > > [root at rdx110 ~]# ibstat -l > > mlx4_0 > > mlx4_1 > > [root at rdx110 ~]# > > I tried configuration with both card, and no one work with GPFS. > > > > I also tried with mlx4_0/1, but same problem. > > > > Someone already have the issue ? > > > Kind Regards, > > Frank > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Mon Jun 12 20:41:17 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Mon, 12 Jun 2017 15:41:17 -0400 Subject: [gpfsug-discuss] 'mmces address move' weirdness? Message-ID: <18719.1497296477@turing-police.cc.vt.edu> So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Mon Jun 12 21:01:44 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 12 Jun 2017 20:01:44 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: <18719.1497296477@turing-police.cc.vt.edu> References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of valdis.kletnieks at vt.edu Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Jun 12 21:06:09 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Mon, 12 Jun 2017 20:06:09 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkr at lbl.gov Mon Jun 12 21:17:08 2017 From: kkr at lbl.gov (Kristy Kallback-Rose) Date: Mon, 12 Jun 2017 16:17:08 -0400 Subject: [gpfsug-discuss] Meaning of API Stats Category Message-ID: Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? Category 1, GPFSFileSystemAPI: This metrics gives the following information for each file system (application view). For example: myMachine|GPFSFilesystemAPI|myCluster|myFilesystem|gpfs_fis_bytes_read . gpfs_fis_bytes_read Number of bytes read. gpfs_fis_bytes_written Number of bytes written. gpfs_fis_close_calls Number of close calls. gpfs_fis_disks Number of disks in the file system. gpfs_fis_inodes_written Number of inode updates to disk. gpfs_fis_open_calls Number of open calls. gpfs_fis_read_calls Number of read calls. gpfs_fis_readdir_calls Number of readdir calls. gpfs_fis_write_calls Number of write calls. Category 2, GPFSNodeAPI: This metrics gives the following information for a particular node from its application point of view. For example: myMachine|GPFSNodeAPI|gpfs_is_bytes_read . gpfs_is_bytes_read Number of bytes read. gpfs_is_bytes_written Number of bytes written. gpfs_is_close_calls Number of close calls. gpfs_is_inodes_written Number of inode updates to disk. gpfs_is_open_calls Number of open calls. gpfs_is_readDir_calls Number of readdir calls. gpfs_is_read_calls Number of read calls. gpfs_is_write_calls Number of write calls. Thanks, Kristy -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 12 21:42:47 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 12 Jun 2017 20:42:47 +0000 Subject: [gpfsug-discuss] Meaning of API Stats Category Message-ID: Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Mon Jun 12 22:01:36 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 12 Jun 2017 17:01:36 -0400 Subject: [gpfsug-discuss] Meaning of API Stats Category In-Reply-To: References: Message-ID: Hello Kristy, The GPFSFileSystemAPI and GPFSNodeAPI sensor metrics are from the point of view of "applications" in the sense that they provide stats about I/O requests made to files in GPFS file systems from user level applications using POSIX interfaces like open(), close(), read(), write(), etc. This is in contrast to similarly named sensors without the "API" suffix, like GPFSFilesystem and GPFSNode. Those sensors provide stats about I/O requests made by the GPFS code to NSDs (disks) making up GPFS file systems. The relationship between application I/O and disk I/O might or might not be obvious. Consider some examples. An application that starts sequentially reading a file might, at least initially, cause more disk I/O than expected because GPFS has decided to prefetch data. An application write() might not immediately cause a the writing of disk blocks due to the operation of the pagepool. Ultimately, application write()s might cause twice as much data written to disk due to the replication factor of the file system. Application I/O concerns itself with user data; disk I/O might have to occur to handle the user data and associated file system metadata (like inodes and indirect blocks). The difference between GPFSFileSystemAPI and GPFSNodeAPI: GPFSFileSystemAPI reports stats for application I/O per filesystem per node; GPFSNodeAPI reports application I/O stats per node. Similarly, GPFSFilesystem reports stats for disk I/O per filesystem per node; GPFSNode reports disk I/O stats per node. I hope this helps. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/12/2017 04:43 PM Subject: Re: [gpfsug-discuss] Meaning of API Stats Category Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 12 23:50:44 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 12 Jun 2017 22:50:44 +0000 Subject: [gpfsug-discuss] Meaning of API Stats Category Message-ID: <163FC574-4191-4C20-A4C7-E66DB1868BF3@nuance.com> Can you tell me how LROC plays into this? I?m trying to understand if the difference between gpfs_ns_bytes_read and gpfs_is_bytes_read on a cluster-wide basis reflects the amount of data that is recalled from pagepool+LROC (assuming the majority of the nodes have LROC. Any insight on LROC stats would helpful as well. [cid:image001.png at 01D2E3A4.63CEE1D0] Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of IBM Spectrum Scale Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 4:01 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Meaning of API Stats Category Hello Kristy, The GPFSFileSystemAPI and GPFSNodeAPI sensor metrics are from the point of view of "applications" in the sense that they provide stats about I/O requests made to files in GPFS file systems from user level applications using POSIX interfaces like open(), close(), read(), write(), etc. This is in contrast to similarly named sensors without the "API" suffix, like GPFSFilesystem and GPFSNode. Those sensors provide stats about I/O requests made by the GPFS code to NSDs (disks) making up GPFS file systems. The relationship between application I/O and disk I/O might or might not be obvious. Consider some examples. An application that starts sequentially reading a file might, at least initially, cause more disk I/O than expected because GPFS has decided to prefetch data. An application write() might not immediately cause a the writing of disk blocks due to the operation of the pagepool. Ultimately, application write()s might cause twice as much data written to disk due to the replication factor of the file system. Application I/O concerns itself with user data; disk I/O might have to occur to handle the user data and associated file system metadata (like inodes and indirect blocks). The difference between GPFSFileSystemAPI and GPFSNodeAPI: GPFSFileSystemAPI reports stats for application I/O per filesystem per node; GPFSNodeAPI reports application I/O stats per node. Similarly, GPFSFilesystem reports stats for disk I/O per filesystem per node; GPFSNode reports disk I/O stats per node. I hope this helps. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/12/2017 04:43 PM Subject: Re: [gpfsug-discuss] Meaning of API Stats Category Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 124065 bytes Desc: image001.png URL: From valdis.kletnieks at vt.edu Tue Jun 13 05:21:26 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 13 Jun 2017 00:21:26 -0400 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: <15827.1497327686@turing-police.cc.vt.edu> On Mon, 12 Jun 2017 20:06:09 -0000, "Simon Thompson (IT Research Support)" said: > mmces node suspend -N > > Is what you want. This will move the address and stop it being assigned one, > otherwise the rebalance will occur. Yeah, I figured that part out. What I couldn't wrap my brain around was what the purpose of 'mmces address move' is if mmsysmon is going to just put it back... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From janfrode at tanso.net Tue Jun 13 05:42:21 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Tue, 13 Jun 2017 04:42:21 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: <15827.1497327686@turing-police.cc.vt.edu> References: <18719.1497296477@turing-police.cc.vt.edu> <15827.1497327686@turing-police.cc.vt.edu> Message-ID: Switch to node affinity policy, and it will stick to where you move it. "mmces address policy node-affinity". -jf tir. 13. jun. 2017 kl. 06.21 skrev : > On Mon, 12 Jun 2017 20:06:09 -0000, "Simon Thompson (IT Research Support)" > said: > > > mmces node suspend -N > > > > Is what you want. This will move the address and stop it being assigned > one, > > otherwise the rebalance will occur. > > Yeah, I figured that part out. What I couldn't wrap my brain around was > what the purpose of 'mmces address move' is if mmsysmon is going to just > put it back... > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Jun 13 09:08:52 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 13 Jun 2017 08:08:52 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: Yes, suspending the node would do it, but in the case where you want to remove a node from service but keep it running for testing it's not ideal. I think you can set the IP address balancing policy to none which might do what we want. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Jun 13 09:12:13 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 13 Jun 2017 08:12:13 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> <15827.1497327686@turing-police.cc.vt.edu> Message-ID: Or this ? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: 13 June 2017 05:42 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Switch to node affinity policy, and it will stick to where you move it. "mmces address policy node-affinity". -jf tir. 13. jun. 2017 kl. 06.21 skrev >: On Mon, 12 Jun 2017 20:06:09 -0000, "Simon Thompson (IT Research Support)" said: > mmces node suspend -N > > Is what you want. This will move the address and stop it being assigned one, > otherwise the rebalance will occur. Yeah, I figured that part out. What I couldn't wrap my brain around was what the purpose of 'mmces address move' is if mmsysmon is going to just put it back... _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Tue Jun 13 09:28:25 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Tue, 13 Jun 2017 08:28:25 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: Suspending the node doesn't stop the services though, we've done a bunch of testing by connecting to the "real" IP on the box we wanted to test and that works fine. OK, so you end up connecting to shares like \\192.168.1.20\sharename, but its perfectly fine for testing purposes. In our experience, suspending the node has been fine for this as it moves the IP to a "working" node and keeps user service running whilst we test. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 13 June 2017 at 09:08 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Yes, suspending the node would do it, but in the case where you want to remove a node from service but keep it running for testing it?s not ideal. I think you can set the IP address balancing policy to none which might do what we want. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From:gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Jun 13 09:30:18 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 13 Jun 2017 08:30:18 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: Oh? Nice to know - thanks - will try that method next. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 13 June 2017 09:28 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Suspending the node doesn't stop the services though, we've done a bunch of testing by connecting to the "real" IP on the box we wanted to test and that works fine. OK, so you end up connecting to shares like \\192.168.1.20\sharename, but its perfectly fine for testing purposes. In our experience, suspending the node has been fine for this as it moves the IP to a "working" node and keeps user service running whilst we test. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 13 June 2017 at 09:08 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Yes, suspending the node would do it, but in the case where you want to remove a node from service but keep it running for testing it's not ideal. I think you can set the IP address balancing policy to none which might do what we want. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From:gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Tue Jun 13 14:36:30 2017 From: john.hearns at asml.com (John Hearns) Date: Tue, 13 Jun 2017 13:36:30 +0000 Subject: [gpfsug-discuss] Infiniband Quality of Service settings? Message-ID: I am investigating setting up Quality of Service parameters on an Infiniband fabric. The specific goal is to reduce the bandwidth which certain servers can use, ie if there are untested or development codes running on these servers in our cluster then they cannot adversely affect production users. I hope I do not show too much of my ignorance here. Perhaps out of date, but I find that Lustre does have a facility for setting the port range and hence associating with an ULP in Infiniband http://www.spinics.net/lists/linux-rdma/msg02150.html https://community.mellanox.com/thread/3660 (There. I said the L word. Is a quick soaping to the mouth needed?) Can anyone comment what the Infiniband Service ID for GPFS traffic is please? If the answer is blindingly obvious and is displayed by a Bat signal in the clouds above every datacenter containing GPFS then I am suitably apologetic. If it is buried in a footnote in a Redbook then a bit less apologetic. If you are familiar with Appendix A of the IBTA Architecture Release then it is truly a joy. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Tue Jun 13 15:10:43 2017 From: john.hearns at asml.com (John Hearns) Date: Tue, 13 Jun 2017 14:10:43 +0000 Subject: [gpfsug-discuss] Infiniband Quality of Service settings? In-Reply-To: References: Message-ID: Having the bad manners to answer my own question: Example If you define a service level of 2 for GPFS in the InfiniBand subnet manager set verbsRdmaQpRtrSl to 2. mmchconfig verbsRdmaQpRtrSl=2 I guess though that I still need the service ID to set the Service Level in qos-policy.conf From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of John Hearns Sent: Tuesday, June 13, 2017 3:37 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Infiniband Quality of Service settings? I am investigating setting up Quality of Service parameters on an Infiniband fabric. The specific goal is to reduce the bandwidth which certain servers can use, ie if there are untested or development codes running on these servers in our cluster then they cannot adversely affect production users. I hope I do not show too much of my ignorance here. Perhaps out of date, but I find that Lustre does have a facility for setting the port range and hence associating with an ULP in Infiniband http://www.spinics.net/lists/linux-rdma/msg02150.html https://community.mellanox.com/thread/3660 (There. I said the L word. Is a quick soaping to the mouth needed?) Can anyone comment what the Infiniband Service ID for GPFS traffic is please? If the answer is blindingly obvious and is displayed by a Bat signal in the clouds above every datacenter containing GPFS then I am suitably apologetic. If it is buried in a footnote in a Redbook then a bit less apologetic. If you are familiar with Appendix A of the IBTA Architecture Release then it is truly a joy. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From SAnderson at convergeone.com Tue Jun 13 17:31:31 2017 From: SAnderson at convergeone.com (Shaun Anderson) Date: Tue, 13 Jun 2017 16:31:31 +0000 Subject: [gpfsug-discuss] Difference between mmcesnfscrexport and 'mmnfs export add' commands. Message-ID: <2990f67cded849e8b82a4c5d2ac50d5c@NACR502.nacr.com> ?I see both of these, but only the mmnfs command is documented. Is one a wrapper of the other? SHAUN ANDERSON STORAGE ARCHITECT O 208.577.2112 M 214.263.7014 NOTICE: This email message and any attachments here to may contain confidential information. Any unauthorized review, use, disclosure, or distribution of such information is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy the original message and all copies of it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Wed Jun 14 01:50:24 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 13 Jun 2017 20:50:24 -0400 Subject: [gpfsug-discuss] Meaning of API Stats Category In-Reply-To: <163FC574-4191-4C20-A4C7-E66DB1868BF3@nuance.com> References: <163FC574-4191-4C20-A4C7-E66DB1868BF3@nuance.com> Message-ID: Hello Bob, Right. Within some observation interval, bytes read by an application will be reflected in gpfs_is_bytes_read, regardless of how the byte values were obtained (by reading from "disk", fetching from pagepool, or fetching from LROC). gpfs_ns_bytes_read is only going to reflect bytes read from "disk" within that observation interval. "mmdiag --lroc" provides some LROC stats. There is also a GPFSLROC sensor; it does not appear to be documented at this point, so I hope I haven't spoken out of turn. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Cc: "scale at us.ibm.com" Date: 06/12/2017 06:50 PM Subject: Re: Meaning of API Stats Category Can you tell me how LROC plays into this? I?m trying to understand if the difference between gpfs_ns_bytes_read and gpfs_is_bytes_read on a cluster-wide basis reflects the amount of data that is recalled from pagepool+LROC (assuming the majority of the nodes have LROC. Any insight on LROC stats would helpful as well. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of IBM Spectrum Scale Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 4:01 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Meaning of API Stats Category Hello Kristy, The GPFSFileSystemAPI and GPFSNodeAPI sensor metrics are from the point of view of "applications" in the sense that they provide stats about I/O requests made to files in GPFS file systems from user level applications using POSIX interfaces like open(), close(), read(), write(), etc. This is in contrast to similarly named sensors without the "API" suffix, like GPFSFilesystem and GPFSNode. Those sensors provide stats about I/O requests made by the GPFS code to NSDs (disks) making up GPFS file systems. The relationship between application I/O and disk I/O might or might not be obvious. Consider some examples. An application that starts sequentially reading a file might, at least initially, cause more disk I/O than expected because GPFS has decided to prefetch data. An application write() might not immediately cause a the writing of disk blocks due to the operation of the pagepool. Ultimately, application write()s might cause twice as much data written to disk due to the replication factor of the file system. Application I/O concerns itself with user data; disk I/O might have to occur to handle the user data and associated file system metadata (like inodes and indirect blocks). The difference between GPFSFileSystemAPI and GPFSNodeAPI: GPFSFileSystemAPI reports stats for application I/O per filesystem per node; GPFSNodeAPI reports application I/O stats per node. Similarly, GPFSFilesystem reports stats for disk I/O per filesystem per node; GPFSNode reports disk I/O stats per node. I hope this helps. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/12/2017 04:43 PM Subject: Re: [gpfsug-discuss] Meaning of API Stats Category Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 124065 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jun 14 17:11:33 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 14 Jun 2017 16:11:33 +0000 Subject: [gpfsug-discuss] 4.2.3.x and sub-block size Message-ID: Hi All, Back at SC16 I was told that GPFS 4.2.3.x would remove the ?a sub-block is 1/32nd of the block size? restriction. However, I have installed GPFS 4.2.3.1 on my test cluster and in the man page for mmcrfs I still see: 2. The GPFS block size determines: * The minimum disk space allocation unit. The minimum amount of space that file data can occupy is a sub?block. A sub?block is 1/32 of the block size. So has the restriction been removed? If not, is there an update on which version of GPFS will remove it? If so, can the documentation be updated to reflect the change and how to take advantage of it? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kums at us.ibm.com Wed Jun 14 18:15:27 2017 From: kums at us.ibm.com (Kumaran Rajaram) Date: Wed, 14 Jun 2017 13:15:27 -0400 Subject: [gpfsug-discuss] 4.2.3.x and sub-block size In-Reply-To: References: Message-ID: Hi, >>Back at SC16 I was told that GPFS 4.2.3.x would remove the ?a sub-block is 1/32nd of the block size? restriction. However, I have installed GPFS 4.2.3.1 on my test cluster and in the man page for mmcrfs I still see: >>So has the restriction been removed? If not, is there an update on which version of GPFS will remove it? If so, can the documentation be updated to reflect the change and how to take advantage of it? Thanks? Based on the current plan, this ?a sub-block is 1/32nd of the block size? restriction will be removed in the upcoming GPFS version 4.2.4 (Please NOTE: Support for >32 subblocks per block may subject to be delayed based on internal qualification/validation efforts). Regards, -Kums From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/14/2017 12:12 PM Subject: [gpfsug-discuss] 4.2.3.x and sub-block size Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, Back at SC16 I was told that GPFS 4.2.3.x would remove the ?a sub-block is 1/32nd of the block size? restriction. However, I have installed GPFS 4.2.3.1 on my test cluster and in the man page for mmcrfs I still see: 2. The GPFS block size determines: * The minimum disk space allocation unit. The minimum amount of space that file data can occupy is a sub?block. A sub?block is 1/32 of the block size. So has the restriction been removed? If not, is there an update on which version of GPFS will remove it? If so, can the documentation be updated to reflect the change and how to take advantage of it? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Schlipalius at pawsey.org.au Thu Jun 15 03:30:27 2017 From: Chris.Schlipalius at pawsey.org.au (Chris Schlipalius) Date: Thu, 15 Jun 2017 10:30:27 +0800 Subject: [gpfsug-discuss] Perth Australia - Spectrum Scale User Group event in August 2017 announced - Pawsey Supercomputing Centre Message-ID: Hi please find the eventbrite link (text as http/s links are usually stripped). www.eventbrite.com/e/spectrum-scale-user-group-perth-australia-gpfsugaus-au gust-2017-tickets-35227460282 Please register and let me know if you are keen to present. I have a special group booking offer on accomodation for attendees, well below usually rack rack. I will announce this Usergroup meeting on spectrumscle.org shortly. This event is on the same week and at the same location as HPC Advisory Council also being held in Perth Australia. (Call for papers is now out - I can supply the HPC AC invite separately if you wish to email me directly). If you want to know more in person and you are at ISC2017 next week I will be at the Spectrum Scale Usergroup that Ulf announced or you can catch me on the Pawsey Supercomputing Centre booth. Regards, Chris Schlipalius Senior Storage Infrastructure Specialist/Team Leader Pawsey Supercomputing Centre 12 Burvill Court Kensington WA 6151 Australia Tel +61 8 6436 8815 Email chris.schlipalius at pawsey.org.au Web www.pawsey.org.au From Kevin.Buterbaugh at Vanderbilt.Edu Thu Jun 15 21:00:47 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Thu, 15 Jun 2017 20:00:47 +0000 Subject: [gpfsug-discuss] SAN problem ... multipathd ... mmunlinkfileset ... ??? Message-ID: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> Hi All, I?ve got some very weird problems going on here (and I do have a PMR open with IBM). On Monday I attempted to unlink a fileset, something that I?ve done many times with no issues. This time, however, it hung up the filesystem. I was able to clear things up by shutting down GPFS on the filesystem manager for that filesystem and restarting it. The very next morning we awoke to problems with GPFS. I noticed in my messages file on all my NSD servers I had messages like: Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Write Protect is off Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Asking for cache data failed Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Assuming drive cache: write through Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Attached SCSI disk Jun 12 22:03:32 nsd32 multipathd: sdab: add path (uevent) Jun 12 22:03:32 nsd32 multipathd: sdab: failed to get path uid Jun 12 22:03:32 nsd32 multipathd: uevent trigger error Jun 12 22:03:42 nsd32 kernel: rport-0:0-4: blocked FC remote port time out: removing target and saving binding Since we use an FC SAN and Linux multi-pathing I was expecting some sort of problem with the switches. Now on the switches I see messages like: [114][Thu Jun 15 19:02:05.411 UTC 2017][I][8600.0020][Port][Port: 9][SYNC_LOSS] [115][Thu Jun 15 19:03:49.988 UTC 2017][I][8600.001F][Port][Port: 9][SYNC_ACQ] Which (while not in this example) do correlate time-wise with the multi path messages on the servers. So it?s not a GPFS problem and I shouldn?t be bugging this list about this EXCEPT? These issues only started on Monday after I ran the mmunlinkfileset command. That?s right ? NO such errors prior to then. And literally NOTHING changed on Monday with my SAN environment (nothing had changed there for months actually). Nothing added to nor removed from the SAN. No changes until today when, in an attempt to solve this issue, I updated the switch firmware on all switches one at a time. I also yum updated to the latest RHEL 7 version of the multipathd packages. I?ve been Googling and haven?t found anything useful on those SYNC_LOSS messages on the QLogic SANbox 5800 switches. Anybody out there happen to have any knowledge of them and what could be causing them? Oh, I?m investigating this now ? but it?s not all ports that are throwing the errors. And the ports that are seem to be random and don?t have one specific type of hardware plugged in ? i.e. some ports have NSD servers plugged in, others have storage arrays. I understand that it makes no sense that mmunlinkfileset hanging would cause problems with my SAN ? but I also don?t believe in coincidences! I?m running GPFS 4.2.2.3. Any help / suggestions apprecaiated! ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Thu Jun 15 21:50:10 2017 From: ewahl at osc.edu (Edward Wahl) Date: Thu, 15 Jun 2017 16:50:10 -0400 Subject: [gpfsug-discuss] SAN problem ... multipathd ... mmunlinkfileset ... ??? In-Reply-To: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> References: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> Message-ID: <20170615165010.6241c6d3@osc.edu> On Thu, 15 Jun 2017 20:00:47 +0000 "Buterbaugh, Kevin L" wrote: > Hi All, > > I?ve got some very weird problems going on here (and I do have a PMR open > with IBM). On Monday I attempted to unlink a fileset, something that I?ve > done many times with no issues. This time, however, it hung up the > filesystem. I was able to clear things up by shutting down GPFS on the > filesystem manager for that filesystem and restarting it. > > The very next morning we awoke to problems with GPFS. I noticed in my > messages file on all my NSD servers I had messages like: > > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Write Protect is off > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Asking for cache data failed > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Assuming drive cache: write > through Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline > device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Attached SCSI disk > Jun 12 22:03:32 nsd32 multipathd: sdab: add path (uevent) > Jun 12 22:03:32 nsd32 multipathd: sdab: failed to get path uid > Jun 12 22:03:32 nsd32 multipathd: uevent trigger error > Jun 12 22:03:42 nsd32 kernel: rport-0:0-4: blocked FC remote port time out: > removing target and saving binding > > Since we use an FC SAN and Linux multi-pathing I was expecting some sort of > problem with the switches. Now on the switches I see messages like: > > [114][Thu Jun 15 19:02:05.411 UTC 2017][I][8600.0020][Port][Port: > 9][SYNC_LOSS] [115][Thu Jun 15 19:03:49.988 UTC > 2017][I][8600.001F][Port][Port: 9][SYNC_ACQ] > > Which (while not in this example) do correlate time-wise with the multi path > messages on the servers. So it?s not a GPFS problem and I shouldn?t be > bugging this list about this EXCEPT? > > These issues only started on Monday after I ran the mmunlinkfileset command. > That?s right ? NO such errors prior to then. And literally NOTHING changed > on Monday with my SAN environment (nothing had changed there for months > actually). Nothing added to nor removed from the SAN. No changes until > today when, in an attempt to solve this issue, I updated the switch firmware > on all switches one at a time. I also yum updated to the latest RHEL 7 > version of the multipathd packages. > > I?ve been Googling and haven?t found anything useful on those SYNC_LOSS > messages on the QLogic SANbox 5800 switches. Anybody out there happen to > have any knowledge of them and what could be causing them? Oh, I?m > investigating this now ? but it?s not all ports that are throwing the > errors. And the ports that are seem to be random and don?t have one specific > type of hardware plugged in ? i.e. some ports have NSD servers plugged in, > others have storage arrays. I have a half dozen of the Sanbox 5802 switches, but no GPFS devices going through them any longer. Used to though. We do see that exact same messages when the FC interface on a device goes bad (SFP, HCA, etc) or someone moving cables. This happens when the device cannot properly join the loop with it's login. I've NEVER seen them randomly though. Nor has this been a bad cable type error. I don't recall why, but I froze our Sanbox's at : V7.4.0.16.0 I'm sure I have notes on it somewhere. I've got one right now, in fact, with a bad ancient LTO4 drive. [8124][Thu Jun 15 12:46:00.190 EDT 2017][I][8600.001F][Port][Port: 4][SYNC_ACQ] [8125][Thu Jun 15 12:49:20.920 EDT 2017][I][8600.0020][Port][Port: 4][SYNC_LOSS] Sounds like the sanbox itself is having an issue perhaps? "Show alarm" clean on the sanbox? Array has a bad HCA? 'Show port 9' errors not crazy? All power supplies working? > I understand that it makes no sense that mmunlinkfileset hanging would cause > problems with my SAN ? but I also don?t believe in coincidences! > > I?m running GPFS 4.2.2.3. Any help / suggestions apprecaiated! This does seem like QUITE the coincidence. Increased traffic on the device triggered a failure? (The fear of all RAID users!) Multipath is working properly though? Sounds like mmlsdisk would have shown devices not in 'ready'. We mysteriously lost a MD disk during a recent downtime and it caused an MMFSCK to not run properly until we figured it out. 4.2.2.3 as well. MD Replication is NOT helpful in that case. Ed > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu - > (615)875-9633 > > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From Kevin.Buterbaugh at Vanderbilt.Edu Thu Jun 15 22:14:33 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Thu, 15 Jun 2017 21:14:33 +0000 Subject: [gpfsug-discuss] SAN problem ... multipathd ... mmunlinkfileset ... ??? In-Reply-To: <20170615165010.6241c6d3@osc.edu> References: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> <20170615165010.6241c6d3@osc.edu> Message-ID: <91D4DDD0-BF4C-4F73-B369-A91C032B0FCD@vanderbilt.edu> Hi Ed, others, I have spent the intervening time since sending my original e-mail taking the logs from the SAN switches and putting them into text files where they can be sorted and grep?d ? and something potentially interesting has come to light ? While there are a number of ports on all switches that have one or two SYNC_LOSS errors on them, on two of the switches port 9 has dozens of SYNC_LOSS errors (looking at the raw logs with other messages interspersed that wasn?t obvious). Turns out that one particular dual-controller storage array is plugged into those ports and - in a stroke of good luck which usually manages to avoid me - that particular storage array is no longer in use! It, and a few others still in use, are older and about to be life-cycled. Since it?s no longer in use, I have unplugged it from the SAN and am monitoring to see if my problems now go away. Yes, correlation is not causation. And sometimes coincidences do happen. I?ll monitor to see if this is one of those occasions. Thanks? Kevin > On Jun 15, 2017, at 3:50 PM, Edward Wahl wrote: > > On Thu, 15 Jun 2017 20:00:47 +0000 > "Buterbaugh, Kevin L" wrote: > >> Hi All, >> >> I?ve got some very weird problems going on here (and I do have a PMR open >> with IBM). On Monday I attempted to unlink a fileset, something that I?ve >> done many times with no issues. This time, however, it hung up the >> filesystem. I was able to clear things up by shutting down GPFS on the >> filesystem manager for that filesystem and restarting it. >> >> The very next morning we awoke to problems with GPFS. I noticed in my >> messages file on all my NSD servers I had messages like: >> >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Write Protect is off >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Asking for cache data failed >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Assuming drive cache: write >> through Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline >> device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Attached SCSI disk >> Jun 12 22:03:32 nsd32 multipathd: sdab: add path (uevent) >> Jun 12 22:03:32 nsd32 multipathd: sdab: failed to get path uid >> Jun 12 22:03:32 nsd32 multipathd: uevent trigger error >> Jun 12 22:03:42 nsd32 kernel: rport-0:0-4: blocked FC remote port time out: >> removing target and saving binding >> >> Since we use an FC SAN and Linux multi-pathing I was expecting some sort of >> problem with the switches. Now on the switches I see messages like: >> >> [114][Thu Jun 15 19:02:05.411 UTC 2017][I][8600.0020][Port][Port: >> 9][SYNC_LOSS] [115][Thu Jun 15 19:03:49.988 UTC >> 2017][I][8600.001F][Port][Port: 9][SYNC_ACQ] >> >> Which (while not in this example) do correlate time-wise with the multi path >> messages on the servers. So it?s not a GPFS problem and I shouldn?t be >> bugging this list about this EXCEPT? >> >> These issues only started on Monday after I ran the mmunlinkfileset command. >> That?s right ? NO such errors prior to then. And literally NOTHING changed >> on Monday with my SAN environment (nothing had changed there for months >> actually). Nothing added to nor removed from the SAN. No changes until >> today when, in an attempt to solve this issue, I updated the switch firmware >> on all switches one at a time. I also yum updated to the latest RHEL 7 >> version of the multipathd packages. >> >> I?ve been Googling and haven?t found anything useful on those SYNC_LOSS >> messages on the QLogic SANbox 5800 switches. Anybody out there happen to >> have any knowledge of them and what could be causing them? Oh, I?m >> investigating this now ? but it?s not all ports that are throwing the >> errors. And the ports that are seem to be random and don?t have one specific >> type of hardware plugged in ? i.e. some ports have NSD servers plugged in, >> others have storage arrays. > > I have a half dozen of the Sanbox 5802 switches, but no GPFS devices going > through them any longer. Used to though. We do see that exact same messages > when the FC interface on a device goes bad (SFP, HCA, etc) or someone moving > cables. This happens when the device cannot properly join the loop with it's > login. I've NEVER seen them randomly though. Nor has this been a bad cable type > error. I don't recall why, but I froze our Sanbox's at : V7.4.0.16.0 I'm sure > I have notes on it somewhere. > > I've got one right now, in fact, with a bad ancient LTO4 drive. > [8124][Thu Jun 15 12:46:00.190 EDT 2017][I][8600.001F][Port][Port: 4][SYNC_ACQ] > [8125][Thu Jun 15 12:49:20.920 EDT 2017][I][8600.0020][Port][Port: 4][SYNC_LOSS] > > > Sounds like the sanbox itself is having an issue perhaps? "Show alarm" clean on > the sanbox? Array has a bad HCA? 'Show port 9' errors not crazy? All power > supplies working? > > > >> I understand that it makes no sense that mmunlinkfileset hanging would cause >> problems with my SAN ? but I also don?t believe in coincidences! >> >> I?m running GPFS 4.2.2.3. Any help / suggestions apprecaiated! > > This does seem like QUITE the coincidence. Increased traffic on the > device triggered a failure? (The fear of all RAID users!) Multipath is working > properly though? Sounds like mmlsdisk would have shown devices not in 'ready'. > We mysteriously lost a MD disk during a recent downtime and it caused an MMFSCK > to not run properly until we figured it out. 4.2.2.3 as well. MD Replication > is NOT helpful in that case. > > Ed > > > >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and Education >> Kevin.Buterbaugh at vanderbilt.edu - >> (615)875-9633 >> >> >> > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 From frank.tower at outlook.com Sun Jun 18 05:57:57 2017 From: frank.tower at outlook.com (Frank Tower) Date: Sun, 18 Jun 2017 04:57:57 +0000 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found In-Reply-To: References: , Message-ID: Hi, You were right, ibv_devinfo -v doesn't return something if both card are connected. I didn't checked ibv_* tools, I supposed once IP stack and ibstat OK, the rest should work. I'm stupid ? Anyway, once I disconnect one card, ibv_devinfo show me input but with both cards, I don't have any input except "device not found". And what is weird here, it's that it work only when one card are connected, no matter the card (both are similar: model, firmware, revision, company)... Really strange, I will dig more about the issue. Stupid and bad workaround: connected a dual port Infiniband. But production system doesn't wait.. Thank for your help, Frank ________________________________ From: Aaron Knister Sent: Saturday, June 10, 2017 2:05 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Out of curiosity could you send us the output of "ibv_devinfo -v"? -Aaron Sent from my iPhone On Jun 10, 2017, at 06:55, Frank Tower > wrote: Hi everybody, I don't get why one of our compute node cannot start GPFS over IB. I have the following error: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). [I] VERBS RDMA parse verbsPorts mlx4_0/1 [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found [I] VERBS RDMA library libibverbs.so unloaded. [E] VERBS RDMA failed to start, no valid verbsPorts defined. I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. I have 2 infinibands card, both have an IP and working well. [root at rdx110 ~]# ibstat -l mlx4_0 mlx4_1 [root at rdx110 ~]# I tried configuration with both card, and no one work with GPFS. I also tried with mlx4_0/1, but same problem. Someone already have the issue ? Kind Regards, Frank _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.tower at outlook.com Sun Jun 18 06:06:30 2017 From: frank.tower at outlook.com (Frank Tower) Date: Sun, 18 Jun 2017 05:06:30 +0000 Subject: [gpfsug-discuss] Protocol node: active directory authentication Message-ID: Hi, We finally received protocols node following the recommendations some here provided and the help of the wiki. Now we would like to use kerberized NFS, we dig into spectrumscale documentations and wiki but we would like to know if anyone is using such configuration ? Do you also have any performance issue (vs NFSv4/NFSv3 with sec=sys) ? We will also use Microsoft Active Directory and we are willing to populate all our users with UID/GID, summer is coming, we will have some spare time ? Someone is using kerberized NFSv4 with Microsoft Active Directory ? Thank by advance for your feedback. Kind Regards, Frank. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcatana at gmail.com Sun Jun 18 16:30:55 2017 From: jcatana at gmail.com (Josh Catana) Date: Sun, 18 Jun 2017 11:30:55 -0400 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found In-Reply-To: References: Message-ID: Are any cards VPI that can do both eth and ib? I remember reading in documentation that that there is a bus order to having mixed media with mellanox cards. There is a module setting during init where you can set eth ib or auto detect. If the card is on auto it might be coming up eth and making the driver flake out because it's in the wrong order. Responding from my phone so I can't really look it up myself right now about what the proper order is, but maybe this might be some help troubleshooting. On Jun 18, 2017 12:58 AM, "Frank Tower" wrote: > Hi, > > > You were right, ibv_devinfo -v doesn't return something if both card are > connected. I didn't checked ibv_* tools, I supposed once IP stack and > ibstat OK, the rest should work. I'm stupid ? > > > Anyway, once I disconnect one card, ibv_devinfo show me input but with > both cards, I don't have any input except "device not found". > > And what is weird here, it's that it work only when one card are > connected, no matter the card (both are similar: model, firmware, revision, > company)... Really strange, I will dig more about the issue. > > > Stupid and bad workaround: connected a dual port Infiniband. But > production system doesn't wait.. > > > Thank for your help, > Frank > > ------------------------------ > *From:* Aaron Knister > *Sent:* Saturday, June 10, 2017 2:05 PM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found > > Out of curiosity could you send us the output of "ibv_devinfo -v"? > > -Aaron > > Sent from my iPhone > > On Jun 10, 2017, at 06:55, Frank Tower wrote: > > Hi everybody, > > > I don't get why one of our compute node cannot start GPFS over IB. > > > I have the following error: > > > [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no > verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes > > [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and > initialized. > > [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match > (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). > > [I] VERBS RDMA parse verbsPorts mlx4_0/1 > > [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device > mlx4_0 not found > > [I] VERBS RDMA library libibverbs.so unloaded. > > [E] VERBS RDMA failed to start, no valid verbsPorts defined. > > > > I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. > > > I have 2 infinibands card, both have an IP and working well. > > > [root at rdx110 ~]# ibstat -l > > mlx4_0 > > mlx4_1 > > [root at rdx110 ~]# > > > I tried configuration with both card, and no one work with GPFS. > > > I also tried with mlx4_0/1, but same problem. > > > Someone already have the issue ? > > > Kind Regards, > > Frank > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Sun Jun 18 18:53:28 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Sun, 18 Jun 2017 17:53:28 +0000 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found Message-ID: There used to be issues with the CX-3 cards and specific ports for if you wanted to use IB and Eth, but that went away in later firmwares, as did a whole load of bits with it being slow to detect media type, so see if you are running an up to date Mellanox firmware (assuming it's a VPI card). On CX-4 there is no auto detect media, but default is IB unless you changed it. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of jcatana at gmail.com [jcatana at gmail.com] Sent: 18 June 2017 16:30 To: gpfsug main discussion list Subject: ?spam? Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Are any cards VPI that can do both eth and ib? I remember reading in documentation that that there is a bus order to having mixed media with mellanox cards. There is a module setting during init where you can set eth ib or auto detect. If the card is on auto it might be coming up eth and making the driver flake out because it's in the wrong order. Responding from my phone so I can't really look it up myself right now about what the proper order is, but maybe this might be some help troubleshooting. On Jun 18, 2017 12:58 AM, "Frank Tower" > wrote: Hi, You were right, ibv_devinfo -v doesn't return something if both card are connected. I didn't checked ibv_* tools, I supposed once IP stack and ibstat OK, the rest should work. I'm stupid ? Anyway, once I disconnect one card, ibv_devinfo show me input but with both cards, I don't have any input except "device not found". And what is weird here, it's that it work only when one card are connected, no matter the card (both are similar: model, firmware, revision, company)... Really strange, I will dig more about the issue. Stupid and bad workaround: connected a dual port Infiniband. But production system doesn't wait.. Thank for your help, Frank ________________________________ From: Aaron Knister > Sent: Saturday, June 10, 2017 2:05 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Out of curiosity could you send us the output of "ibv_devinfo -v"? -Aaron Sent from my iPhone On Jun 10, 2017, at 06:55, Frank Tower > wrote: Hi everybody, I don't get why one of our compute node cannot start GPFS over IB. I have the following error: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). [I] VERBS RDMA parse verbsPorts mlx4_0/1 [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found [I] VERBS RDMA library libibverbs.so unloaded. [E] VERBS RDMA failed to start, no valid verbsPorts defined. I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. I have 2 infinibands card, both have an IP and working well. [root at rdx110 ~]# ibstat -l mlx4_0 mlx4_1 [root at rdx110 ~]# I tried configuration with both card, and no one work with GPFS. I also tried with mlx4_0/1, but same problem. Someone already have the issue ? Kind Regards, Frank _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From eric.wonderley at vt.edu Tue Jun 20 17:03:40 2017 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Tue, 20 Jun 2017 12:03:40 -0400 Subject: [gpfsug-discuss] gui related connection fail in gpfs logs Message-ID: These type messages repeat often in our logs: 017-06-20_09:25:13.676-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_09:25:24.292-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_10:00:25.935-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 Is there any way to tell if it is a misconfiguration or communications issue? -------------- next part -------------- An HTML attachment was scrubbed... URL: From NSCHULD at de.ibm.com Wed Jun 21 08:12:36 2017 From: NSCHULD at de.ibm.com (Norbert Schuld) Date: Wed, 21 Jun 2017 09:12:36 +0200 Subject: [gpfsug-discuss] gui related connection fail in gpfs logs In-Reply-To: References: Message-ID: This happens if a the mmhealth system of a node can not forward an event to the GUI - typically on some other node. Resons could be: - Gui is not running - Firewall on used port 80 for older versions of spectrum scale or 443 for newer Mit freundlichen Gr??en / Kind regards Norbert Schuld Dr. IBM Systems Group M925: IBM Spectrum Scale Norbert Software Development Schuld IBM Deutschland R&D GmbH Phone: +49-160 70 70 335 Am Weiher 24 E-Mail: nschuld at de.ibm.com 65451 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina Koederitz /Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "J. Eric Wonderley" To: gpfsug main discussion list Date: 20/06/2017 18:04 Subject: [gpfsug-discuss] gui related connection fail in gpfs logs Sent by: gpfsug-discuss-bounces at spectrumscale.org These type messages repeat often in our logs: 017-06-20_09:25:13.676-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_09:25:24.292-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_10:00:25.935-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 Is there any way to tell if it is a misconfiguration or communications issue? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 16690161.gif Type: image/gif Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From ewahl at osc.edu Thu Jun 22 20:37:12 2017 From: ewahl at osc.edu (Edward Wahl) Date: Thu, 22 Jun 2017 15:37:12 -0400 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: <20170622153712.052d312c@osc.edu> Is there a command to show existing node Address Policy? Or are we left with grep "affinity" on /var/mmfs/gen/mmsdrfs? Ed On Tue, 13 Jun 2017 08:30:18 +0000 "Sobey, Richard A" wrote: > Oh? Nice to know - thanks - will try that method next. > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson > (IT Research Support) Sent: 13 June 2017 09:28 To: gpfsug main discussion > list Subject: Re: [gpfsug-discuss] 'mmces > address move' weirdness? > > Suspending the node doesn't stop the services though, we've done a bunch of > testing by connecting to the "real" IP on the box we wanted to test and that > works fine. > > OK, so you end up connecting to shares like > \\192.168.1.20\sharename, but its perfectly > fine for testing purposes. > > In our experience, suspending the node has been fine for this as it moves the > IP to a "working" node and keeps user service running whilst we test. > > Simon > > From: > > > on behalf of "Sobey, Richard A" > > Reply-To: > "gpfsug-discuss at spectrumscale.org" > > > Date: Tuesday, 13 June 2017 at 09:08 To: > "gpfsug-discuss at spectrumscale.org" > > > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? > > Yes, suspending the node would do it, but in the case where you want to > remove a node from service but keep it running for testing it's not ideal. > > I think you can set the IP address balancing policy to none which might do > what we want. From: > gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson > (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion > list > > > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? > > mmces node suspend -N > > Is what you want. This will move the address and stop it being assigned one, > otherwise the rebalance will occur. I think you can change the way it > balances, but the default is to distribute. > > Simon > > From: > > > on behalf of "Sobey, Richard A" > > Reply-To: > "gpfsug-discuss at spectrumscale.org" > > > Date: Monday, 12 June 2017 at 21:01 To: > "gpfsug-discuss at spectrumscale.org" > > > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? > > > I think it's intended but I don't know why. The AUTH service became unhealthy > on one of our CES nodes (SMB only) and we moved its float address elsewhere. > CES decided to move it back again moments later despite the node not being > fit. > > > > Sorry that doesn't really help but at least you're not alone! > > ________________________________ > From:gpfsug-discuss-bounces at spectrumscale.org > > > on behalf of valdis.kletnieks at vt.edu > > Sent: 12 June 2017 > 20:41 To: > gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] 'mmces address move' weirdness? > > So here's our address setup: > > mmces address list > > Address Node Group Attribute > ------------------------------------------------------------------------- > 172.28.45.72 arproto1.ar.nis.isb.internal isb none > 172.28.45.73 arproto2.ar.nis.isb.internal isb none > 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none > 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none > > Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to > move the address over to its pair so I can look around without impacting > users. However, seems like something insists on moving it right back 60 > seconds later... > > Question 1: Is this expected behavior? > Question 2: If it is, what use is 'mmces address move' if it just gets > undone a few seconds later... > > (running on arproto2.ar.nis.vtc.internal): > > ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 > --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr > show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 > 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global > secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 > Mon Jun 12 15:34:42 EDT 2017 > Mon Jun 12 15:34:43 EDT 2017 > (skipped) > Mon Jun 12 15:35:44 EDT 2017 > Mon Jun 12 15:35:45 EDT 2017 > inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 > Mon Jun 12 15:35:46 EDT 2017 > inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 > Mon Jun 12 15:35:47 EDT 2017 > inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 > ^C -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From laurence at qsplace.co.uk Thu Jun 22 23:05:49 2017 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Thu, 22 Jun 2017 23:05:49 +0100 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: <20170622153712.052d312c@osc.edu> References: <18719.1497296477@turing-police.cc.vt.edu> <20170622153712.052d312c@osc.edu> Message-ID: <6B843B3B-4C07-459D-B905-10B16E3590A0@qsplace.co.uk> "mmlscluster --ces" will show the address distribution policy. -- Lauz On 22 June 2017 20:37:12 BST, Edward Wahl wrote: > >Is there a command to show existing node Address Policy? >Or are we left with grep "affinity" on /var/mmfs/gen/mmsdrfs? > >Ed > > >On Tue, 13 Jun 2017 08:30:18 +0000 >"Sobey, Richard A" wrote: > >> Oh? Nice to know - thanks - will try that method next. >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon >Thompson >> (IT Research Support) Sent: 13 June 2017 09:28 To: gpfsug main >discussion >> list Subject: Re: [gpfsug-discuss] >'mmces >> address move' weirdness? >> >> Suspending the node doesn't stop the services though, we've done a >bunch of >> testing by connecting to the "real" IP on the box we wanted to test >and that >> works fine. >> >> OK, so you end up connecting to shares like >> \\192.168.1.20\sharename, but its >perfectly >> fine for testing purposes. >> >> In our experience, suspending the node has been fine for this as it >moves the >> IP to a "working" node and keeps user service running whilst we test. >> >> Simon >> >> From: >> >> >> on behalf of "Sobey, Richard A" >> > Reply-To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Date: Tuesday, 13 June 2017 at 09:08 To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? >> >> Yes, suspending the node would do it, but in the case where you want >to >> remove a node from service but keep it running for testing it's not >ideal. >> >> I think you can set the IP address balancing policy to none which >might do >> what we want. From: >> >gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon >Thompson >> (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main >discussion >> list >> >> >> Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? >> >> mmces node suspend -N >> >> Is what you want. This will move the address and stop it being >assigned one, >> otherwise the rebalance will occur. I think you can change the way it >> balances, but the default is to distribute. >> >> Simon >> >> From: >> >> >> on behalf of "Sobey, Richard A" >> > Reply-To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Date: Monday, 12 June 2017 at 21:01 To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? >> >> >> I think it's intended but I don't know why. The AUTH service became >unhealthy >> on one of our CES nodes (SMB only) and we moved its float address >elsewhere. >> CES decided to move it back again moments later despite the node not >being >> fit. >> >> >> >> Sorry that doesn't really help but at least you're not alone! >> >> ________________________________ >> >From:gpfsug-discuss-bounces at spectrumscale.org >> >> >> on behalf of valdis.kletnieks at vt.edu >> > Sent: 12 >June 2017 >> 20:41 To: >> >gpfsug-discuss at spectrumscale.org >> Subject: [gpfsug-discuss] 'mmces address move' weirdness? >> >> So here's our address setup: >> >> mmces address list >> >> Address Node Group >Attribute >> >------------------------------------------------------------------------- >> 172.28.45.72 arproto1.ar.nis.isb.internal isb none >> 172.28.45.73 arproto2.ar.nis.isb.internal isb none >> 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none >> 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none >> >> Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so >I try to >> move the address over to its pair so I can look around without >impacting >> users. However, seems like something insists on moving it right back >60 >> seconds later... >> >> Question 1: Is this expected behavior? >> Question 2: If it is, what use is 'mmces address move' if it just >gets >> undone a few seconds later... >> >> (running on arproto2.ar.nis.vtc.internal): >> >> ## (date; ip addr show | grep '\.72';mmces address move --ces-ip >172.28.46.72 >> --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; >ip addr >> show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon >Jun 12 >> 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global >> secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 >EDT 2017 >> Mon Jun 12 15:34:42 EDT 2017 >> Mon Jun 12 15:34:43 EDT 2017 >> (skipped) >> Mon Jun 12 15:35:44 EDT 2017 >> Mon Jun 12 15:35:45 EDT 2017 >> inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary >bond1:0 >> Mon Jun 12 15:35:46 EDT 2017 >> inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary >bond1:0 >> Mon Jun 12 15:35:47 EDT 2017 >> inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary >bond1:0 >> ^C > > > >-- > >Ed Wahl >Ohio Supercomputer Center >614-292-9302 >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Fri Jun 23 09:13:38 2017 From: john.hearns at asml.com (John Hearns) Date: Fri, 23 Jun 2017 08:13:38 +0000 Subject: [gpfsug-discuss] IO prioritisation / throttling? Message-ID: I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO 'runs wild' it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Fri Jun 23 09:57:46 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Fri, 23 Jun 2017 04:57:46 -0400 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: References: Message-ID: <11158C89-C712-4C79-8B1D-CAA9D3D8641F@gmail.com> I unfortunately don't have an answer other than to perhaps check out this presentation from a recent users group meeting: http://files.gpfsug.org/presentations/2017/Manchester/05_Ellexus_SSUG_Manchester.pdf I've never used the product but it might be able to do what you're asking. Sent from my iPhone > On Jun 23, 2017, at 04:13, John Hearns wrote: > > I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. > We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. > The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down > the production storage. If anyone has a setup like this I would be interested in chatting with you. > Is it feasible to create filesets which have higher/lower priority than others? > > Thankyou for any insights or feedback > John Hearns > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Fri Jun 23 12:04:23 2017 From: john.hearns at asml.com (John Hearns) Date: Fri, 23 Jun 2017 11:04:23 +0000 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: <11158C89-C712-4C79-8B1D-CAA9D3D8641F@gmail.com> References: <11158C89-C712-4C79-8B1D-CAA9D3D8641F@gmail.com> Message-ID: Aaron, thankyou. I know Rosemary and that is a good company! From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Friday, June 23, 2017 10:58 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] IO prioritisation / throttling? I unfortunately don't have an answer other than to perhaps check out this presentation from a recent users group meeting: http://files.gpfsug.org/presentations/2017/Manchester/05_Ellexus_SSUG_Manchester.pdf I've never used the product but it might be able to do what you're asking. Sent from my iPhone On Jun 23, 2017, at 04:13, John Hearns > wrote: I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kums at us.ibm.com Fri Jun 23 14:36:34 2017 From: kums at us.ibm.com (Kumaran Rajaram) Date: Fri, 23 Jun 2017 09:36:34 -0400 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: References: Message-ID: Hi John, >>We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. >>The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down >>the production storage. You may use the Spectrum Scale Quality of Service for I/O "mmchqos" command (details in link below) to define IOPS limits for the "others" as well as the "maintenance" class for the Dev/Test file-system "pools" (for e.g., mmchqos tds_fs --enable pool=*,other=10000IOPS, maintenance=5000IOPS). This way, the Test and Dev file-system/storage-pools IOPS can be limited/controlled to specified IOPS , giving higher priority to the production GPFS file-system/storage (with production_fs pool=* other=unlimited,maintenance=unlimited - which is the default). https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_mmchqos.htm https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_qosio_describe.htm#qosio_describe My two cents. Regards, -Kums From: John Hearns To: gpfsug main discussion list Date: 06/23/2017 04:14 AM Subject: [gpfsug-discuss] IO prioritisation / throttling? Sent by: gpfsug-discuss-bounces at spectrumscale.org I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Fri Jun 23 15:23:19 2017 From: john.hearns at asml.com (John Hearns) Date: Fri, 23 Jun 2017 14:23:19 +0000 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: References: Message-ID: Thankyou to Kumaran and Aaaron for your help. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Kumaran Rajaram Sent: Friday, June 23, 2017 3:37 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] IO prioritisation / throttling? Hi John, >>We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. >>The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down >>the production storage. You may use the Spectrum Scale Quality of Service for I/O "mmchqos" command (details in link below) to define IOPS limits for the "others" as well as the "maintenance" class for the Dev/Test file-system "pools" (for e.g., mmchqos tds_fs --enable pool=*,other=10000IOPS, maintenance=5000IOPS). This way, the Test and Dev file-system/storage-pools IOPS can be limited/controlled to specified IOPS , giving higher priority to the production GPFS file-system/storage (with production_fs pool=* other=unlimited,maintenance=unlimited - which is the default). https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_mmchqos.htm https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_qosio_describe.htm#qosio_describe My two cents. Regards, -Kums From: John Hearns > To: gpfsug main discussion list > Date: 06/23/2017 04:14 AM Subject: [gpfsug-discuss] IO prioritisation / throttling? Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 23 17:06:51 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 23 Jun 2017 16:06:51 +0000 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy Message-ID: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> Hi All, I haven?t been able to find this explicitly documented, so I?m just wanting to confirm that the behavior that I?m expecting is what GPFS is going to do in this scenario? I have a filesystem with data replication set to two. I?m creating a capacity type pool for it right now which will be used to migrate old files to. I only want to use replication of one on the capacity pool. My policy file has two rules, one to move files with an atime > 30 days to the capacity pool, to which I?ve included ?REPLICATE(1)?. The other rule is to move files from the capacity pool back to the system pool if the atime < 14 days. Since data replication is set to two, I am thinking that I do not need to explicitly have a ?REPLICATE(2)? as part of that rule ? is that correct? I.e., I?m wanting to make sure that a file moved to the capacity pool which therefore has its? replication set to one doesn?t keep that same setting even if moved back out of the capacity pool. Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 23 17:58:23 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 23 Jun 2017 12:58:23 -0400 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy In-Reply-To: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> References: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> Message-ID: I believe that is correct. If not, let us know! To recap... when running mmapplypolicy with rules like: ... MIGRATE ... REPLICATE(x) ... will change the replication factor to x, for each file selected by this rule and chosen for execution. ... MIGRATE ... /* no REPLICATE keyword */ will not mess with the replication factor From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/23/2017 12:07 PM Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I haven?t been able to find this explicitly documented, so I?m just wanting to confirm that the behavior that I?m expecting is what GPFS is going to do in this scenario? I have a filesystem with data replication set to two. I?m creating a capacity type pool for it right now which will be used to migrate old files to. I only want to use replication of one on the capacity pool. My policy file has two rules, one to move files with an atime > 30 days to the capacity pool, to which I?ve included ?REPLICATE(1)?. The other rule is to move files from the capacity pool back to the system pool if the atime < 14 days. Since data replication is set to two, I am thinking that I do not need to explicitly have a ?REPLICATE(2)? as part of that rule ? is that correct? I.e., I?m wanting to make sure that a file moved to the capacity pool which therefore has its? replication set to one doesn?t keep that same setting even if moved back out of the capacity pool. Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Fri Jun 23 18:55:15 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Fri, 23 Jun 2017 13:55:15 -0400 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy In-Reply-To: References: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> Message-ID: <82717215-1C59-4B12-BAF6-09908044688D@ulmer.org> > On Jun 23, 2017, at 12:58 PM, Marc A Kaplan > wrote: > > I believe that is correct. If not, let us know! > > To recap... when running mmapplypolicy with rules like: > > ... MIGRATE ... REPLICATE(x) ... > > will change the replication factor to x, for each file selected by this rule and chosen for execution. > > ... MIGRATE ... /* no REPLICATE keyword */ > > will not mess with the replication factor > > I think I detect an impedance mismatch... By "not mess with the replication factor" do you mean that after the move: the file will have the default replication factor for the file system the file will retain a replication factor previously set on the file You told Kevin that he was correct and I think he meant the first one, but I read what you said as the second one. Liberty, -- Stephen -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 23 20:28:28 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 23 Jun 2017 15:28:28 -0400 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy In-Reply-To: <82717215-1C59-4B12-BAF6-09908044688D@ulmer.org> References: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> <82717215-1C59-4B12-BAF6-09908044688D@ulmer.org> Message-ID: Sorry for any confusion. MIGRATing a file does NOT change the replication factor, unless you explicitly use the keyword REPLICATE. The default replication factor, as set/displayed by mm[ch|ls]fs -r only applies at file creation time, unless overriden by a policy SET POOL ... REPLICATE(x) rule. From: Stephen Ulmer To: gpfsug main discussion list Date: 06/23/2017 01:55 PM Subject: Re: [gpfsug-discuss] Replication settings when running mmapplypolicy Sent by: gpfsug-discuss-bounces at spectrumscale.org On Jun 23, 2017, at 12:58 PM, Marc A Kaplan wrote: I believe that is correct. If not, let us know! To recap... when running mmapplypolicy with rules like: ... MIGRATE ... REPLICATE(x) ... will change the replication factor to x, for each file selected by this rule and chosen for execution. ... MIGRATE ... /* no REPLICATE keyword */ will not mess with the replication factor I think I detect an impedance mismatch... By "not mess with the replication factor" do you mean that after the move: the file will have the default replication factor for the file system the file will retain a replication factor previously set on the file You told Kevin that he was correct and I think he meant the first one, but I read what you said as the second one. Liberty, -- Stephen _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From ncapit at atos.net Mon Jun 26 08:49:28 2017 From: ncapit at atos.net (CAPIT, NICOLAS) Date: Mon, 26 Jun 2017 07:49:28 +0000 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads Message-ID: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Hello, I don't know if this behavior/bug was already reported on this ML, so in doubt. Context: - SpectrumScale 4.2.2-3 - client node with 64 cores - OS: RHEL7.3 When a MPI job with 64 processes is launched on the node with 64 cores then the FS freezed (only the output log file of the MPI job is put on the GPFS; so it may be related to the 64 processes writing in a same file???). strace -p 3105 # mmfsd pid stucked Process 3105 attached wait4(-1, # stucked at this point strace ls /gpfs stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC # stucked at this point I have no problem with the other nodes of 28 cores. The GPFS command mmgetstate is working and I am able to use mmshutdown to recover the node. If I put workerThreads=72 on the 64 core node then I am not able to reproduce the freeze and I get the right behavior. Is this a known bug with a number of cores > workerThreads? Best regards, [--] Nicolas Capit -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jun 27 00:57:57 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 26 Jun 2017 19:57:57 -0400 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Message-ID: <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> That's a fascinating bug. When the node is locked up what does "mmdiag --waiters" show from the node in question? I suspect there's more low-level diagnostic data that's helpful for the gurus at IBM but I'm just curious what the waiters look like. -Aaron On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > Hello, > > I don't know if this behavior/bug was already reported on this ML, so in > doubt. > > Context: > > - SpectrumScale 4.2.2-3 > - client node with 64 cores > - OS: RHEL7.3 > > When a MPI job with 64 processes is launched on the node with 64 cores > then the FS freezed (only the output log file of the MPI job is put on > the GPFS; so it may be related to the 64 processes writing in a same > file???). > > strace -p 3105 # mmfsd pid stucked > Process 3105 attached > wait4(-1, # stucked at this point > > strace ls /gpfs > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > # stucked at this point > > I have no problem with the other nodes of 28 cores. > The GPFS command mmgetstate is working and I am able to use mmshutdown > to recover the node. > > > If I put workerThreads=72 on the 64 core node then I am not able to > reproduce the freeze and I get the right behavior. > > Is this a known bug with a number of cores > workerThreads? > > Best regards, > -- > *Nicolas Capit* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From ncapit at atos.net Tue Jun 27 07:59:19 2017 From: ncapit at atos.net (CAPIT, NICOLAS) Date: Tue, 27 Jun 2017 06:59:19 +0000 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net>, <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> Message-ID: <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Hello, When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters"). In the GPFS log file "/var/mmfs/gen/mmfslog" there is nothing and nothing in the dmesg output or system log. The "mmgetstate" command says that the node is "active". The only thing is the freeze of the FS. Best regards, Nicolas Capit ________________________________________ De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de Aaron Knister [aaron.s.knister at nasa.gov] Envoy? : mardi 27 juin 2017 01:57 ? : gpfsug-discuss at spectrumscale.org Objet : Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads That's a fascinating bug. When the node is locked up what does "mmdiag --waiters" show from the node in question? I suspect there's more low-level diagnostic data that's helpful for the gurus at IBM but I'm just curious what the waiters look like. -Aaron On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > Hello, > > I don't know if this behavior/bug was already reported on this ML, so in > doubt. > > Context: > > - SpectrumScale 4.2.2-3 > - client node with 64 cores > - OS: RHEL7.3 > > When a MPI job with 64 processes is launched on the node with 64 cores > then the FS freezed (only the output log file of the MPI job is put on > the GPFS; so it may be related to the 64 processes writing in a same > file???). > > strace -p 3105 # mmfsd pid stucked > Process 3105 attached > wait4(-1, # stucked at this point > > strace ls /gpfs > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > # stucked at this point > > I have no problem with the other nodes of 28 cores. > The GPFS command mmgetstate is working and I am able to use mmshutdown > to recover the node. > > > If I put workerThreads=72 on the 64 core node then I am not able to > reproduce the freeze and I get the right behavior. > > Is this a known bug with a number of cores > workerThreads? > > Best regards, > -- > *Nicolas Capit* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From stuartb at 4gh.net Tue Jun 27 22:34:34 2017 From: stuartb at 4gh.net (Stuart Barkley) Date: Tue, 27 Jun 2017 17:34:34 -0400 (EDT) Subject: [gpfsug-discuss] express edition vs standard edition Message-ID: Does anyone know what controls whether GPFS (4.1.1) thinks it is Express Edition versus Standard Edition? While rebuilding an old cluster from scratch we are getting the message: mmcrfs: Storage pools are not available in the GPFS Express Edition. The Problem Determination Guide says to "Install the GPFS Standard Edition on all nodes in the cluster" which we think we have done. The cluster is just 3 servers and no clients at this point. We have verified that our purchased license is for Standard Version, but have not been able to figure out what controls the GPFS view of this. mmlslicense tells us that we have Express Edition installed. mmchlicense sets server vs client license information, but does not seem to be able to control the edition. Our normal install process is to install gpfs.base-4.1.1-0.x86_64.rpm first and then install gpfs.base-4.1.1-15.x86_64.update.rpm followed by the other needed 4.1.1-15 rpms. I thought maybe we had the wrong gpfs.base and we have re-downloaded Standard Edition RPM files from IBM in case we had the wrong version. However, reinstalling and recreating the cluster does not seem to have addressed this issue. We must be doing something stupid during our install, but I'm pretty sure we used only Standard Edition rpms for our latest attempt. Thanks, Stuart Barkley -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From knop at us.ibm.com Tue Jun 27 22:54:35 2017 From: knop at us.ibm.com (Felipe Knop) Date: Tue, 27 Jun 2017 17:54:35 -0400 Subject: [gpfsug-discuss] express edition vs standard edition In-Reply-To: References: Message-ID: Stuart, I believe you will need to install the gpfs.ext RPMs , otherwise the daemons and commands will think only the Express edition is installed. Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 From: Stuart Barkley To: gpfsug-discuss at spectrumscale.org Date: 06/27/2017 05:35 PM Subject: [gpfsug-discuss] express edition vs standard edition Sent by: gpfsug-discuss-bounces at spectrumscale.org Does anyone know what controls whether GPFS (4.1.1) thinks it is Express Edition versus Standard Edition? While rebuilding an old cluster from scratch we are getting the message: mmcrfs: Storage pools are not available in the GPFS Express Edition. The Problem Determination Guide says to "Install the GPFS Standard Edition on all nodes in the cluster" which we think we have done. The cluster is just 3 servers and no clients at this point. We have verified that our purchased license is for Standard Version, but have not been able to figure out what controls the GPFS view of this. mmlslicense tells us that we have Express Edition installed. mmchlicense sets server vs client license information, but does not seem to be able to control the edition. Our normal install process is to install gpfs.base-4.1.1-0.x86_64.rpm first and then install gpfs.base-4.1.1-15.x86_64.update.rpm followed by the other needed 4.1.1-15 rpms. I thought maybe we had the wrong gpfs.base and we have re-downloaded Standard Edition RPM files from IBM in case we had the wrong version. However, reinstalling and recreating the cluster does not seem to have addressed this issue. We must be doing something stupid during our install, but I'm pretty sure we used only Standard Edition rpms for our latest attempt. Thanks, Stuart Barkley -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuartb at 4gh.net Tue Jun 27 23:48:53 2017 From: stuartb at 4gh.net (Stuart Barkley) Date: Tue, 27 Jun 2017 18:48:53 -0400 (EDT) Subject: [gpfsug-discuss] express edition vs standard edition In-Reply-To: References: Message-ID: On Tue, 27 Jun 2017 at 17:54 -0000, Felipe Knop wrote: > I believe you will need to install the gpfs.ext RPMs , otherwise the > daemons and commands will think only the Express edition is > installed. Yes. This appears to have been my problem. Thanks, Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From scale at us.ibm.com Fri Jun 30 07:57:49 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 30 Jun 2017 14:57:49 +0800 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net>, <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Message-ID: I'm not aware this kind of defects, seems it should not. but lack of data, we don't know what happened. I suggest you can open a PMR for your issue. Thanks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "CAPIT, NICOLAS" To: gpfsug main discussion list Date: 06/27/2017 02:59 PM Subject: Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters"). In the GPFS log file "/var/mmfs/gen/mmfslog" there is nothing and nothing in the dmesg output or system log. The "mmgetstate" command says that the node is "active". The only thing is the freeze of the FS. Best regards, Nicolas Capit ________________________________________ De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de Aaron Knister [aaron.s.knister at nasa.gov] Envoy? : mardi 27 juin 2017 01:57 ? : gpfsug-discuss at spectrumscale.org Objet : Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads That's a fascinating bug. When the node is locked up what does "mmdiag --waiters" show from the node in question? I suspect there's more low-level diagnostic data that's helpful for the gurus at IBM but I'm just curious what the waiters look like. -Aaron On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > Hello, > > I don't know if this behavior/bug was already reported on this ML, so in > doubt. > > Context: > > - SpectrumScale 4.2.2-3 > - client node with 64 cores > - OS: RHEL7.3 > > When a MPI job with 64 processes is launched on the node with 64 cores > then the FS freezed (only the output log file of the MPI job is put on > the GPFS; so it may be related to the 64 processes writing in a same > file???). > > strace -p 3105 # mmfsd pid stucked > Process 3105 attached > wait4(-1, # stucked at this point > > strace ls /gpfs > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > # stucked at this point > > I have no problem with the other nodes of 28 cores. > The GPFS command mmgetstate is working and I am able to use mmshutdown > to recover the node. > > > If I put workerThreads=72 on the 64 core node then I am not able to > reproduce the freeze and I get the right behavior. > > Is this a known bug with a number of cores > workerThreads? > > Best regards, > -- > *Nicolas Capit* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From heiner.billich at psi.ch Fri Jun 30 11:07:10 2017 From: heiner.billich at psi.ch (Billich Heinrich Rainer (PSI)) Date: Fri, 30 Jun 2017 10:07:10 +0000 Subject: [gpfsug-discuss] AFM - how to update directories with deleted files during prefetch Message-ID: <76D2B41B-37A6-410B-9ECE-F5FA4C7FF1EE@psi.ch> Hello I have a short question about AFM prefetch and some more remarks regarding AFM and it?s use for data migration. I understand that many of you have done this for very large amounts of data and number of files. I would welcome an input, comments or remarks. Sorry if this is a bit too long for a mailing list. Short: How can I tell an AFM cache to update a directory when I do prefetch? I know about ?find .? or ?ls ?lsR? but this really is no option for us as it takes too long. Mostly I want to update the directories to make AFM cache aware of file deletions on home. On home I can use a policy run to find all directories which changed since the last update and pass them to prefetch on AFM cache. I know that I can find some workaround based on the directory list, like an ?ls ?lsa? just for those directories, but this doesn?t sound very efficient. And depending on cache effects and timeout settings it may work or not (o.k. ? it will work most time). We do regular file deletions and will accumulated millions of deleted files on cache over time if we don?t update the directories to make AFM cache aware of the deletion. Background: We will use AFM to migrate data on filesets to another cluster. We have to do this several times in the next few months, hence I want to get a reliable and easy to use procedure. The old system is home, the new system is a read-only AFM cache. We want to use ?mmafmctl prefetch? to move the data. Home will be in use while we run the migration. Once almost all data is moved we do a (short) break for a last sync and make the read-only AFM cache a ?normal? fileset. During the break I want to use policy runs and prefetch only and no time consuming ?ls ?lsr? or ?find .? I don?t want to use this metadata intensive posix operation during operation, either. More general: AFM can be used for data migration. But I don?t see how to use it efficiently. How to do incremental transfers, how to ensure that the we really have identical copies before we switch and how to keep the switch time short , i.e. the time when both old and new aren?t accessible for clients, Wish ? maybe an RFE? I can use policy runs to collect all changed items on home since the last update. I wish that I can pass this list to afm prefetch to do all updates on AFM cache, too. Same as backup tools use the list to do incremental backups. And a tool to create policy lists of home and cache and to compare the lists would be nice, too. As you do this during the break/switch it should be fast and reliable and leave no doubts. Kind regards, Heiner From vpuvvada at in.ibm.com Fri Jun 30 13:35:18 2017 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Fri, 30 Jun 2017 18:05:18 +0530 Subject: [gpfsug-discuss] AFM - how to update directories with deleted files during prefetch In-Reply-To: <76D2B41B-37A6-410B-9ECE-F5FA4C7FF1EE@psi.ch> References: <76D2B41B-37A6-410B-9ECE-F5FA4C7FF1EE@psi.ch> Message-ID: What is the version of GPFS ? >Mostly I want to update the directories to make AFM cache aware of file deletions on home. On home I can use a policy run to find all directories which changed since the last >update and pass them to prefetch on AFM cache. AFM prefetch has undocumented option to delete files from cache without doing lookup from home. It supports all types of list files. Find all deleted file from home and prefetch at cache to delete them. mmafmctl device prefetch -j fileset --list-file --delete >Wish ? maybe an RFE? >I can use policy runs to collect all changed items on home since the last update. I wish that I can pass this list to afm prefetch to do all updates on AFM cache, too. Same >as backup tools use the list to do incremental backups. This feature support was already added but undocumented today. This feature will be externalized in future releases. ~Venkat (vpuvvada at in.ibm.com) From: "Billich Heinrich Rainer (PSI)" To: "gpfsug-discuss at spectrumscale.org" Date: 06/30/2017 03:37 PM Subject: [gpfsug-discuss] AFM - how to update directories with deleted files during prefetch Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello I have a short question about AFM prefetch and some more remarks regarding AFM and it?s use for data migration. I understand that many of you have done this for very large amounts of data and number of files. I would welcome an input, comments or remarks. Sorry if this is a bit too long for a mailing list. Short: How can I tell an AFM cache to update a directory when I do prefetch? I know about ?find .? or ?ls ?lsR? but this really is no option for us as it takes too long. Mostly I want to update the directories to make AFM cache aware of file deletions on home. On home I can use a policy run to find all directories which changed since the last update and pass them to prefetch on AFM cache. I know that I can find some workaround based on the directory list, like an ?ls ?lsa? just for those directories, but this doesn?t sound very efficient. And depending on cache effects and timeout settings it may work or not (o.k. ? it will work most time). We do regular file deletions and will accumulated millions of deleted files on cache over time if we don?t update the directories to make AFM cache aware of the deletion. Background: We will use AFM to migrate data on filesets to another cluster. We have to do this several times in the next few months, hence I want to get a reliable and easy to use procedure. The old system is home, the new system is a read-only AFM cache. We want to use ?mmafmctl prefetch? to move the data. Home will be in use while we run the migration. Once almost all data is moved we do a (short) break for a last sync and make the read-only AFM cache a ?normal? fileset. During the break I want to use policy runs and prefetch only and no time consuming ?ls ?lsr? or ?find .? I don?t want to use this metadata intensive posix operation during operation, either. More general: AFM can be used for data migration. But I don?t see how to use it efficiently. How to do incremental transfers, how to ensure that the we really have identical copies before we switch and how to keep the switch time short , i.e. the time when both old and new aren?t accessible for clients, Wish ? maybe an RFE? I can use policy runs to collect all changed items on home since the last update. I wish that I can pass this list to afm prefetch to do all updates on AFM cache, too. Same as backup tools use the list to do incremental backups. And a tool to create policy lists of home and cache and to compare the lists would be nice, too. As you do this during the break/switch it should be fast and reliable and leave no doubts. Kind regards, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From hpc-luke at uconn.edu Fri Jun 30 16:20:27 2017 From: hpc-luke at uconn.edu (hpc-luke at uconn.edu) Date: Fri, 30 Jun 2017 11:20:27 -0400 Subject: [gpfsug-discuss] Mass UID migration suggestions Message-ID: <59566c3b.kqOSy9e2kg80XZ/k%hpc-luke@uconn.edu> Hello, We're trying to change most of our users uids, is there a clean way to migrate all of one users files with say `mmapplypolicy`? We have to change the owner of around 273539588 files, and my estimates for runtime are around 6 days. What we've been doing is indexing all of the files and splitting them up by owner which takes around an hour, and then we were locking the user out while we chown their files. I made it multi threaded as it weirdly gave a 10% speedup despite my expectation that multi threading access from a single node would not give any speedup. Generally I'm looking for advice on how to make the chowning faster. Would spreading the chowning processes over multiple nodes improve performance? Should I not stat the files before running lchown on them, since lchown checks the file before changing it? I saw mention of inodescan(), in an old gpfsug email, which speeds up disk read access, by not guaranteeing that the data is up to date. We have a maintenance day coming up where all users will be locked out, so the file handles(?) from GPFS's perspective will not be able to go stale. Is there a function with similar constraints to inodescan that I can use to speed up this process? Thank you for your time, Luke Storrs-HPC University of Connecticut From aaron.knister at gmail.com Fri Jun 30 16:47:40 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Fri, 30 Jun 2017 11:47:40 -0400 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Message-ID: Nicolas, By chance do you have a skylake or kabylake based CPU? Sent from my iPhone > On Jun 30, 2017, at 02:57, IBM Spectrum Scale wrote: > > I'm not aware this kind of defects, seems it should not. but lack of data, we don't know what happened. I suggest you can open a PMR for your issue. Thanks. > > Regards, The Spectrum Scale (GPFS) team > > ------------------------------------------------------------------------------------------------------------------ > If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. > > If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. > > The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. > > "CAPIT, NICOLAS" ---06/27/2017 02:59:59 PM---Hello, When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters") > > From: "CAPIT, NICOLAS" > To: gpfsug main discussion list > Date: 06/27/2017 02:59 PM > Subject: Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > Hello, > > When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters"). > In the GPFS log file "/var/mmfs/gen/mmfslog" there is nothing and nothing in the dmesg output or system log. > The "mmgetstate" command says that the node is "active". > The only thing is the freeze of the FS. > > Best regards, > Nicolas Capit > ________________________________________ > De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de Aaron Knister [aaron.s.knister at nasa.gov] > Envoy? : mardi 27 juin 2017 01:57 > ? : gpfsug-discuss at spectrumscale.org > Objet : Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads > > That's a fascinating bug. When the node is locked up what does "mmdiag > --waiters" show from the node in question? I suspect there's more > low-level diagnostic data that's helpful for the gurus at IBM but I'm > just curious what the waiters look like. > > -Aaron > > On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > > Hello, > > > > I don't know if this behavior/bug was already reported on this ML, so in > > doubt. > > > > Context: > > > > - SpectrumScale 4.2.2-3 > > - client node with 64 cores > > - OS: RHEL7.3 > > > > When a MPI job with 64 processes is launched on the node with 64 cores > > then the FS freezed (only the output log file of the MPI job is put on > > the GPFS; so it may be related to the 64 processes writing in a same > > file???). > > > > strace -p 3105 # mmfsd pid stucked > > Process 3105 attached > > wait4(-1, # stucked at this point > > > > strace ls /gpfs > > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > > # stucked at this point > > > > I have no problem with the other nodes of 28 cores. > > The GPFS command mmgetstate is working and I am able to use mmshutdown > > to recover the node. > > > > > > If I put workerThreads=72 on the 64 core node then I am not able to > > reproduce the freeze and I get the right behavior. > > > > Is this a known bug with a number of cores > workerThreads? > > > > Best regards, > > -- > > *Nicolas Capit* > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Jun 30 18:14:07 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 30 Jun 2017 13:14:07 -0400 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: <487469581.449569.1498832342497.JavaMail.webinst@w30112> References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: I'm curious to know why this doesn't affect GSS/ESS? Is it a feature of the additional check-summing done on those platforms? -------- Forwarded Message -------- Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) Date: Fri, 30 Jun 2017 14:19:02 +0000 From: IBM My Notifications To: aaron.s.knister at nasa.gov My Notifications for Storage - 30 Jun 2017 Dear Subscriber (aaron.s.knister at nasa.gov), Here are your updates from IBM My Notifications. Your support Notifications display in English by default. Machine translation based on your IBM profile language setting is added if you specify this option in My defaults within My Notifications. (Note: Not all languages are available at this time, and the English version always takes precedence over the machine translated version.) ------------------------------------------------------------------------------ 1. IBM Spectrum Scale - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error - URL: http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM Spectrum Scale versions where the NSD server is enabled to use RDMA for file IO and the storage used in your GPFS cluster accessed via NSD servers (not fully SAN accessible) includes anything other than IBM Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under these conditions, when the RDMA-enabled network adapter fails, the issue may result in undetected data corruption for file write or read operations. ------------------------------------------------------------------------------ Manage your My Notifications subscriptions, or send questions and comments. - Subscribe or Unsubscribe - https://www.ibm.com/support/mynotifications - Feedback - https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html - Follow us on Twitter - https://twitter.com/IBMStorageSupt To ensure proper delivery please add mynotify at stg.events.ihost.com to your address book. You received this email because you are subscribed to IBM My Notifications as: aaron.s.knister at nasa.gov Please do not reply to this message as it is generated by an automated service machine. (C) International Business Machines Corporation 2017. All rights reserved. From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 30 18:25:56 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 30 Jun 2017 17:25:56 +0000 Subject: [gpfsug-discuss] Mass UID migration suggestions In-Reply-To: <59566c3b.kqOSy9e2kg80XZ/k%hpc-luke@uconn.edu> References: <59566c3b.kqOSy9e2kg80XZ/k%hpc-luke@uconn.edu> Message-ID: <1901859B-7620-42E2-9064-14930AC50EE3@vanderbilt.edu> Hi Luke, I?ve got an off the wall suggestion for you, which may or may not work depending on whether or not you have any UID conflicts with old and new UIDs ? this won?t actually speed things up but it will eliminate the ?downtime? for your users. And the big caveat is that there can?t be any UID conflicts ? i.e. someone?s new UID can?t be someone else?s old UID. Given that ? what if you set an ACL to allow access to both their old and new UIDs ? then change their UID to the new UID ? then chown the files to the new UID and remove the ACL? More work for you, but no downtime for them. We actually may need to do something similar as we will need to change Windows-assigned UID?s based on SIDs to ?correct? UIDs at some point in the future on one of our storage systems. If someone has a better way to solve your problem, I hope they?ll post it to the list, as it may help us as well. HTHAL. Thanks? Kevin On Jun 30, 2017, at 10:20 AM, hpc-luke at uconn.edu wrote: Hello, We're trying to change most of our users uids, is there a clean way to migrate all of one users files with say `mmapplypolicy`? We have to change the owner of around 273539588 files, and my estimates for runtime are around 6 days. What we've been doing is indexing all of the files and splitting them up by owner which takes around an hour, and then we were locking the user out while we chown their files. I made it multi threaded as it weirdly gave a 10% speedup despite my expectation that multi threading access from a single node would not give any speedup. Generally I'm looking for advice on how to make the chowning faster. Would spreading the chowning processes over multiple nodes improve performance? Should I not stat the files before running lchown on them, since lchown checks the file before changing it? I saw mention of inodescan(), in an old gpfsug email, which speeds up disk read access, by not guaranteeing that the data is up to date. We have a maintenance day coming up where all users will be locked out, so the file handles(?) from GPFS's perspective will not be able to go stale. Is there a function with similar constraints to inodescan that I can use to speed up this process? Thank you for your time, Luke Storrs-HPC University of Connecticut _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Fri Jun 30 18:37:30 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Fri, 30 Jun 2017 19:37:30 +0200 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Fri Jun 30 18:41:43 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Fri, 30 Jun 2017 13:41:43 -0400 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: Thanks Olaf, that's good to know (and is kind of what I suspected). I've requested a number of times this capability for those of us who can't use or aren't using GNR and the answer is effectively "no". This response is curious to me because I'm sure IBM doesn't believe that data integrity is only important and of value to customers who purchase their hardware *and* software. -Aaron On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser wrote: > yes.. in case of GNR (GPFS native raid) .. we do end-to-end check-summing > ... client --> server --> downToDisk > GNR writes down a chksum to disk (to all pdisks /all "raid" segments ) so > that dropped writes can be detected as well as miss-done writes (bit > flips..) > > > > From: Aaron Knister > To: gpfsug main discussion list > Date: 06/30/2017 07:15 PM > Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): > RDMA-enabled network adapter failure on the NSD server may result in file > IO error (2017.06.30) > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > I'm curious to know why this doesn't affect GSS/ESS? Is it a feature of > the additional check-summing done on those platforms? > > > -------- Forwarded Message -------- > Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled > network adapter > failure on the NSD server may result in file IO error (2017.06.30) > Date: Fri, 30 Jun 2017 14:19:02 +0000 > From: IBM My Notifications > > To: aaron.s.knister at nasa.gov > > > > > My Notifications for Storage - 30 Jun 2017 > > Dear Subscriber (aaron.s.knister at nasa.gov), > > Here are your updates from IBM My Notifications. > > Your support Notifications display in English by default. Machine > translation based on your IBM profile > language setting is added if you specify this option in My defaults > within My Notifications. > (Note: Not all languages are available at this time, and the English > version always takes precedence > over the machine translated version.) > > ------------------------------------------------------------ > ------------------ > 1. IBM Spectrum Scale > > - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure > on the NSD server may result in file IO error > - URL: > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233& > myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_- > OCSTXKQY-OCSWJ00-_-E > - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM > Spectrum Scale versions where the NSD server is enabled to use RDMA for > file IO and the storage used in your GPFS cluster accessed via NSD > servers (not fully SAN accessible) includes anything other than IBM > Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under these > conditions, when the RDMA-enabled network adapter fails, the issue may > result in undetected data corruption for file write or read operations. > > ------------------------------------------------------------ > ------------------ > Manage your My Notifications subscriptions, or send questions and comments. > - Subscribe or Unsubscribe - https://www.ibm.com/support/mynotifications > - Feedback - > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotif > ications.html > > - Follow us on Twitter - https://twitter.com/IBMStorageSupt > > > > To ensure proper delivery please add mynotify at stg.events.ihost.com to > your address book. > You received this email because you are subscribed to IBM My > Notifications as: > aaron.s.knister at nasa.gov > > Please do not reply to this message as it is generated by an automated > service machine. > > (C) International Business Machines Corporation 2017. All rights reserved. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Jun 30 18:53:16 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 30 Jun 2017 13:53:16 -0400 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: <2689cf86-eca2-dab6-c6aa-7fc54d923e55@nasa.gov> In fact the answer was quite literally "no": https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523 (the RFE was declined and the answer was that the "function is already available in GNR environments"). Regarding GNR, see this RFE request https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090 requesting the use of GNR outside of an ESS/GSS environment. It's interesting to note this is the highest voted Public RFE for GPFS that I can see, at least. It too was declined. -Aaron On 6/30/17 1:41 PM, Aaron Knister wrote: > Thanks Olaf, that's good to know (and is kind of what I suspected). I've > requested a number of times this capability for those of us who can't > use or aren't using GNR and the answer is effectively "no". This > response is curious to me because I'm sure IBM doesn't believe that data > integrity is only important and of value to customers who purchase their > hardware *and* software. > > -Aaron > > On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser > wrote: > > yes.. in case of GNR (GPFS native raid) .. we do end-to-end > check-summing ... client --> server --> downToDisk > GNR writes down a chksum to disk (to all pdisks /all "raid" segments > ) so that dropped writes can be detected as well as miss-done > writes (bit flips..) > > > > From: Aaron Knister > > To: gpfsug main discussion list > > Date: 06/30/2017 07:15 PM > Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): > RDMA-enabled network adapter failure on the NSD server may result in > file IO error (2017.06.30) > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------ > > > > I'm curious to know why this doesn't affect GSS/ESS? Is it a feature of > the additional check-summing done on those platforms? > > > -------- Forwarded Message -------- > Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network > adapter > failure on the NSD server may result in file IO error (2017.06.30) > Date: Fri, 30 Jun 2017 14:19:02 +0000 > From: IBM My Notifications > > > To: aaron.s.knister at nasa.gov > > > > > My Notifications for Storage - 30 Jun 2017 > > Dear Subscriber (aaron.s.knister at nasa.gov > ), > > Here are your updates from IBM My Notifications. > > Your support Notifications display in English by default. Machine > translation based on your IBM profile > language setting is added if you specify this option in My defaults > within My Notifications. > (Note: Not all languages are available at this time, and the English > version always takes precedence > over the machine translated version.) > > ------------------------------------------------------------------------------ > 1. IBM Spectrum Scale > > - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter > failure > on the NSD server may result in file IO error > - URL: > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E > > - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM > Spectrum Scale versions where the NSD server is enabled to use RDMA for > file IO and the storage used in your GPFS cluster accessed via NSD > servers (not fully SAN accessible) includes anything other than IBM > Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under these > conditions, when the RDMA-enabled network adapter fails, the issue may > result in undetected data corruption for file write or read operations. > > ------------------------------------------------------------------------------ > Manage your My Notifications subscriptions, or send questions and > comments. > - Subscribe or Unsubscribe - > https://www.ibm.com/support/mynotifications > > - Feedback - > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html > > > - Follow us on Twitter - https://twitter.com/IBMStorageSupt > > > > > To ensure proper delivery please add mynotify at stg.events.ihost.com > to > your address book. > You received this email because you are subscribed to IBM My > Notifications as: > aaron.s.knister at nasa.gov > > Please do not reply to this message as it is generated by an automated > service machine. > > (C) International Business Machines Corporation 2017. All rights > reserved. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From oehmes at gmail.com Fri Jun 30 19:25:28 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 30 Jun 2017 18:25:28 +0000 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: <2689cf86-eca2-dab6-c6aa-7fc54d923e55@nasa.gov> References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> <2689cf86-eca2-dab6-c6aa-7fc54d923e55@nasa.gov> Message-ID: end-to-end data integrity is very important and the reason it hasn't been done in Scale is not because its not important, its because its very hard to do without impacting performance in a very dramatic way. imagine your raid controller blocksize is 1mb and your filesystem blocksize is 1MB . if your application does a 1 MB write this ends up being a perfect full block , full track de-stage to your raid layer and everything works fine and fast. as soon as you add checksum support you need to add data somehow into this, means your 1MB is no longer 1 MB but 1 MB+checksum. to store this additional data you have multiple options, inline , outside the data block or some combination ,the net is either you need to do more physical i/o's to different places to get both the data and the corresponding checksum or your per block on disc structure becomes bigger than than what your application reads/or writes, both put massive burden on the Storage layer as e.g. a 1 MB write will now, even the blocks are all aligned from the application down to the raid layer, cause a read/modify/write on the raid layer as the data is bigger than the physical track size. so to get end-to-end checksum in Scale outside of ESS the best way is to get GNR as SW to run on generic HW, this is what people should vote for as RFE if they need that functionality. beside end-to-end checksums you get read/write cache and acceleration , fast rebuild and many other goodies as a added bonus. Sven On Fri, Jun 30, 2017 at 10:53 AM Aaron Knister wrote: > In fact the answer was quite literally "no": > > https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523 > (the RFE was declined and the answer was that the "function is already > available in GNR environments"). > > Regarding GNR, see this RFE request > https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090 > requesting the use of GNR outside of an ESS/GSS environment. It's > interesting to note this is the highest voted Public RFE for GPFS that I > can see, at least. It too was declined. > > -Aaron > > On 6/30/17 1:41 PM, Aaron Knister wrote: > > Thanks Olaf, that's good to know (and is kind of what I suspected). I've > > requested a number of times this capability for those of us who can't > > use or aren't using GNR and the answer is effectively "no". This > > response is curious to me because I'm sure IBM doesn't believe that data > > integrity is only important and of value to customers who purchase their > > hardware *and* software. > > > > -Aaron > > > > On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser > > wrote: > > > > yes.. in case of GNR (GPFS native raid) .. we do end-to-end > > check-summing ... client --> server --> downToDisk > > GNR writes down a chksum to disk (to all pdisks /all "raid" segments > > ) so that dropped writes can be detected as well as miss-done > > writes (bit flips..) > > > > > > > > From: Aaron Knister > > > > To: gpfsug main discussion list > > > > Date: 06/30/2017 07:15 PM > > Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): > > RDMA-enabled network adapter failure on the NSD server may result in > > file IO error (2017.06.30) > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > ------------------------------------------------------------------------ > > > > > > > > I'm curious to know why this doesn't affect GSS/ESS? Is it a feature > of > > the additional check-summing done on those platforms? > > > > > > -------- Forwarded Message -------- > > Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network > > adapter > > failure on the NSD server may result in file IO error (2017.06.30) > > Date: Fri, 30 Jun 2017 14:19:02 +0000 > > From: IBM My Notifications > > >> > > To: aaron.s.knister at nasa.gov > > > > > > > > > > My Notifications for Storage - 30 Jun 2017 > > > > Dear Subscriber (aaron.s.knister at nasa.gov > > ), > > > > Here are your updates from IBM My Notifications. > > > > Your support Notifications display in English by default. Machine > > translation based on your IBM profile > > language setting is added if you specify this option in My defaults > > within My Notifications. > > (Note: Not all languages are available at this time, and the English > > version always takes precedence > > over the machine translated version.) > > > > > ------------------------------------------------------------------------------ > > 1. IBM Spectrum Scale > > > > - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter > > failure > > on the NSD server may result in file IO error > > - URL: > > > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E > > < > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E > > > > - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM > > Spectrum Scale versions where the NSD server is enabled to use RDMA > for > > file IO and the storage used in your GPFS cluster accessed via NSD > > servers (not fully SAN accessible) includes anything other than IBM > > Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under > these > > conditions, when the RDMA-enabled network adapter fails, the issue > may > > result in undetected data corruption for file write or read > operations. > > > > > ------------------------------------------------------------------------------ > > Manage your My Notifications subscriptions, or send questions and > > comments. > > - Subscribe or Unsubscribe - > > https://www.ibm.com/support/mynotifications > > > > - Feedback - > > > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html > > < > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html > > > > > > - Follow us on Twitter - https://twitter.com/IBMStorageSupt > > > > > > > > > > To ensure proper delivery please add mynotify at stg.events.ihost.com > > to > > your address book. > > You received this email because you are subscribed to IBM My > > Notifications as: > > aaron.s.knister at nasa.gov > > > > Please do not reply to this message as it is generated by an > automated > > service machine. > > > > (C) International Business Machines Corporation 2017. All rights > > reserved. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alandhae at gmx.de Thu Jun 1 10:35:37 2017 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Thu, 1 Jun 2017 11:35:37 +0200 (CEST) Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup Message-ID: Hello all out there, customer wants to receive periodical reports on the filesystem heat (relatively age) of files. We already switched on fileheat using mmchconfig. mmchconfig fileheatlosspercent=10,fileHeatPeriodMinutes=1440 for the reports, I think I need to know the file usage in a given time period. Are there any how-to for obtaining this information, examples for ILM policies to be used as a start? any help will be highly appreciated. Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From olaf.weiser at de.ibm.com Thu Jun 1 12:15:57 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 1 Jun 2017 13:15:57 +0200 Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Thu Jun 1 14:40:30 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 1 Jun 2017 09:40:30 -0400 Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup In-Reply-To: References: Message-ID: To generate a list of files and their file heat values... define([TS],esyscmd(date +%Y-%m-%d-%H-%M | tr -d '\n')) RULE 'x1' EXTERNAL LIST 'heat.TS' EXEC '' RULE 'x2' LIST 'heat.TS' SHOW(FILE_HEAT) WEIGHT(FILE_HEAT) /* use a WHERE clause to select or exclude files */ mmapplypolicy /gpfs-path-of-interest -I defer -P policy-rules-shown-above -f /path-for-result ... other good options are -N nodes -g /shared-temp To do it periodically use crontab. I defined the TS macro so each time you run it, you get a different filename. WEIGHT clause will cause "hottest" files to be listed firstmost. Notice that FILE_HEAT "shows" as a floating point number. --marc From: "Olaf Weiser" To: Andreas Landh?u?er , gpfsug main discussion list Date: 06/01/2017 07:17 AM Subject: Re: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Andreas, one could use the WEIGHT statement ... a simple policy for e.g. rule ?repack? MIGRATE FROM POOL ?xxxxxx? TO POOL ?xxxx? WEIGHT(FILE_HEAT) and then the -I prepare to see, what would be done by policy.. or you use the LIST function .. or .. and so on .. From: Andreas Landh?u?er To: gpfsug-discuss at spectrumscale.org Date: 06/01/2017 11:36 AM Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all out there, customer wants to receive periodical reports on the filesystem heat (relatively age) of files. We already switched on fileheat using mmchconfig. mmchconfig fileheatlosspercent=10,fileHeatPeriodMinutes=1440 for the reports, I think I need to know the file usage in a given time period. Are there any how-to for obtaining this information, examples for ILM policies to be used as a start? any help will be highly appreciated. Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Fri Jun 2 04:10:44 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Fri, 2 Jun 2017 03:10:44 +0000 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - Space Management (GPFS HSM) Message-ID: An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Fri Jun 2 10:28:36 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 02 Jun 2017 05:28:36 -0400 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - Space Management (GPFS HSM) In-Reply-To: References: Message-ID: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca> We have that situation. Users don't need to login to NSD's What you need is to add at least one gpfs client to the cluster (or multi-cluster), mount the DMAPI enabled file system, and use that node as a gateway for end-users. They can access the contents on the mount point with their own underprivileged accounts. Whether or not on a schedule, the moment an application or linux command (such as cp, cat, vi, etc) accesses a stub, the file will be staged. Jaime Quoting "Andrew Beattie" : > Quick question, Does anyone have a Scale / GPFS environment (HPC) > where users need the ability to recall data sets after they have been > stubbed, but only System Administrators are permitted to log onto the > NSD servers for security purposes. And if so how do you provide the > ability for the users to schedule their data set recalls? Regards, > Andrew Beattie Software Defined Storage - IT Specialist Phone: > 614-2133-7927 E-mail: abeattie at au1.ibm.com[1] > > > Links: > ------ > [1] mailto:abeattie at au1.ibm.com > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From abeattie at au1.ibm.com Fri Jun 2 10:48:11 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Fri, 2 Jun 2017 09:48:11 +0000 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) In-Reply-To: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca> References: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca>, Message-ID: An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Fri Jun 2 16:12:41 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 02 Jun 2017 11:12:41 -0400 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) In-Reply-To: References: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca>, Message-ID: <20170602111241.56882fx2qr2yz2ax@support.scinet.utoronto.ca> It has been a while since I used HSM with GPFS via TSM, but as far as I can remember, unprivileged users can run dsmmigrate and dsmrecall. Based on the instructions on the link, dsmrecall may now leverage the Recommended Access Order (RAO) available on enterprise drives, however root would have to be the one to invoke that feature. In that case we may have to develop a middleware/wrapper for dsmrecall that will run as root and act on behalf of the user when optimization is requested. Someone here more familiar with the latest version of TSM-HSM may be able to give us some hints on how people are doing this in practice. Jaime Quoting "Andrew Beattie" : > Thanks Jaime, How do you get around Optimised recalls? from what I > can see the optimised recall process needs a root level account to > retrieve a list of files > https://www.ibm.com/support/knowledgecenter/SSSR2R_7.1.1/com.ibm.itsm.hsmul.doc/c_recall_optimized_tape.html[1] > Regards, Andrew Beattie Software Defined Storage - IT Specialist > Phone: 614-2133-7927 E-mail: abeattie at au1.ibm.com[2] ----- > Original message ----- > From: "Jaime Pinto" > To: "gpfsug main discussion list" , > "Andrew Beattie" > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - > Space Management (GPFS HSM) > Date: Fri, Jun 2, 2017 7:28 PM > We have that situation. > Users don't need to login to NSD's > > What you need is to add at least one gpfs client to the cluster (or > multi-cluster), mount the DMAPI enabled file system, and use that > node > as a gateway for end-users. They can access the contents on the mount > > point with their own underprivileged accounts. > > Whether or not on a schedule, the moment an application or linux > command (such as cp, cat, vi, etc) accesses a stub, the file will be > > staged. > > Jaime > > Quoting "Andrew Beattie" : > >> Quick question, Does anyone have a Scale / GPFS environment (HPC) >> where users need the ability to recall data sets after they have > been >> stubbed, but only System Administrators are permitted to log onto > the >> NSD servers for security purposes. And if so how do you provide > the >> ability for the users to schedule their data set recalls? > Regards, >> Andrew Beattie Software Defined Storage - IT Specialist Phone: >> 614-2133-7927 E-mail: abeattie at au1.ibm.com[1] >> >> >> Links: >> ------ >> [1] mailto:abeattie at au1.ibm.com[3] >> > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials[4] > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From r.sobey at imperial.ac.uk Fri Jun 2 16:51:12 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 2 Jun 2017 15:51:12 +0000 Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS Message-ID: Hi all, Where should I start looking for a compatibility matrix between TSM and GPFS? Specifically, we are currently running TSM 7.1.6-2 and GPFS 4.2.1-2 with the intent to upgrade to GPFS 4.2.3-latest in early July. I've spent 30 minutes looking over various documents and the best I can find is this: http://www-01.ibm.com/support/docview.wss?uid=swg21248771 ..which talks about TSM in a Space Management context and would suggest that we need to upgrade to Spectrum Protect i.e. 8.1 and that GPFS 4.2.2.x is the maximum supported version... Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 2 17:40:11 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 2 Jun 2017 12:40:11 -0400 Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS In-Reply-To: References: Message-ID: Upgrading from GPFS 4.2.x to GPFS 4.2.y should not "break" TSM. If it does, someone goofed, that would be a bug. (My opinion) Think of it this way. TSM is an application that uses the OS and the FileSystem(s). TSM can't verify it will work with all future versions of OS and Filesystems, and the releases can't be in lock step. Having said that, 4.2.3 has been "out" for a while, so if there were a TSM incompatibility, someone would have likely hit it or will before July... Trust but verify... From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 06/02/2017 11:51 AM Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi all, Where should I start looking for a compatibility matrix between TSM and GPFS? Specifically, we are currently running TSM 7.1.6-2 and GPFS 4.2.1-2 with the intent to upgrade to GPFS 4.2.3-latest in early July. I?ve spent 30 minutes looking over various documents and the best I can find is this: http://www-01.ibm.com/support/docview.wss?uid=swg21248771 ..which talks about TSM in a Space Management context and would suggest that we need to upgrade to Spectrum Protect i.e. 8.1 and that GPFS 4.2.2.x is the maximum supported version? Cheers Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 08:51:10 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 08:51:10 +0100 Subject: [gpfsug-discuss] NSD access routes Message-ID: Morning all, Just a quick one about NSD access and read only disks. Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? Cheers, ----------------------------- Dave Goodbourn Head of Systems Milk Visual Effects Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Mon Jun 5 08:52:39 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Mon, 5 Jun 2017 07:52:39 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: Message-ID: Hi Have you look at LROC instead? Might fit in simpler way to what your are describing. -- Cheers > On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: > > Morning all, > > Just a quick one about NSD access and read only disks. > > Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. > > Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? > > Cheers, > ----------------------------- > Dave Goodbourn > > Head of Systems > Milk Visual Effects > Tel: +44 (0)20 3697 8448 > Mob: +44 (0)7917 411 069 Ellei edell?? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 09:02:16 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 09:02:16 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: Yeah, that was my back up plan but would be more costly in the cloud. Read only is a limitation of most cloud providers not something that I "want". Just trying to move a network bottleneck. Cheers, ----------------------------- Dave Goodbourn Head of Systems Milk Visual Effects Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 > On 5 Jun 2017, at 08:52, Luis Bolinches wrote: > > Hi > > Have you look at LROC instead? Might fit in simpler way to what your are describing. > > -- > Cheers > >> On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: >> >> Morning all, >> >> Just a quick one about NSD access and read only disks. >> >> Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. >> >> Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? >> >> Cheers, >> ----------------------------- >> Dave Goodbourn >> >> Head of Systems >> Milk Visual Effects >> Tel: +44 (0)20 3697 8448 >> Mob: +44 (0)7917 411 069 > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 13:19:47 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 13:19:47 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: OK scrap my first question, I can't do what I wanted to do anyway! I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. Cheers, ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 08:52, Luis Bolinches wrote: > Hi > > Have you look at LROC instead? Might fit in simpler way to what your are > describing. > > -- > Cheers > > On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: > > Morning all, > > Just a quick one about NSD access and read only disks. > > Can you have 2 NSD servers, one with read/write access to a disk and one > with just read only access to the same disk? I know you can write to a disk > over the network via another NSD server but can you mount the disk in read > only mode to increase the read performance? This is all virtual/cloud based. > > Is GPFS clever enough (or can it be configured) to know to read from the > locally attached read only disk but write back via another NSD server over > the GPFS network? > > Cheers, > ----------------------------- > Dave Goodbourn > > Head of Systems > Milk Visual Effects > Tel: +44 (0)20 3697 8448 <+44%2020%203697%208448> > Mob: +44 (0)7917 411 069 <+44%207917%20411069> > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Jun 5 13:24:27 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Mon, 5 Jun 2017 12:24:27 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: mmdiag --lroc ? From: > on behalf of "dave at milk-vfx.com" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 5 June 2017 at 13:19 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] NSD access routes OK scrap my first question, I can't do what I wanted to do anyway! I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. Cheers, ---------------------------------------------------- Dave Goodbourn Head of Systems MILK VISUAL EFFECTS [http://www.milk-vfx.com/src/milk_email_logo.jpg] 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 On 5 June 2017 at 08:52, Luis Bolinches > wrote: Hi Have you look at LROC instead? Might fit in simpler way to what your are describing. -- Cheers On 5 Jun 2017, at 10.51, Dave Goodbourn > wrote: Morning all, Just a quick one about NSD access and read only disks. Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? Cheers, ----------------------------- Dave Goodbourn Head of Systems Milk Visual Effects Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 5 13:48:48 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 5 Jun 2017 12:48:48 +0000 Subject: [gpfsug-discuss] NSD access routes Message-ID: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Hi Dave I?ve done a large-scale (600 node) LROC deployment here - feel free to reach out if you have questions. mmdiag --lroc is about all there is but it does give you a pretty good idea how the cache is performing but you can?t tell which files are cached. Also, watch out that the LROC cached will steal pagepool memory (1% of the LROC cache size) Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Dave Goodbourn Reply-To: gpfsug main discussion list Date: Monday, June 5, 2017 at 7:19 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] NSD access routes I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 14:10:21 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 14:10:21 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: Ah yep, thanks a lot. ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 13:24, Simon Thompson (IT Research Support) < S.J.Thompson at bham.ac.uk> wrote: > mmdiag --lroc > > ? > > > From: on behalf of " > dave at milk-vfx.com" > Reply-To: "gpfsug-discuss at spectrumscale.org" < > gpfsug-discuss at spectrumscale.org> > Date: Monday, 5 June 2017 at 13:19 > To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] NSD access routes > > OK scrap my first question, I can't do what I wanted to do anyway! > > I'm testing out the LROC idea. All seems to be working well, but, is there > anyway to monitor what's cached? How full it might be? The performance etc?? > > I can see some stats in mmfsadm dump lroc but that's about it. > > Cheers, > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 08:52, Luis Bolinches wrote: > >> Hi >> >> Have you look at LROC instead? Might fit in simpler way to what your are >> describing. >> >> -- >> Cheers >> >> On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: >> >> Morning all, >> >> Just a quick one about NSD access and read only disks. >> >> Can you have 2 NSD servers, one with read/write access to a disk and one >> with just read only access to the same disk? I know you can write to a disk >> over the network via another NSD server but can you mount the disk in read >> only mode to increase the read performance? This is all virtual/cloud based. >> >> Is GPFS clever enough (or can it be configured) to know to read from the >> locally attached read only disk but write back via another NSD server over >> the GPFS network? >> >> Cheers, >> ----------------------------- >> Dave Goodbourn >> >> Head of Systems >> Milk Visual Effects >> Tel: +44 (0)20 3697 8448 <+44%2020%203697%208448> >> Mob: +44 (0)7917 411 069 <+44%207917%20411069> >> >> >> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >> Oy IBM Finland Ab >> PL 265, 00101 Helsinki, Finland >> Business ID, Y-tunnus: 0195876-3 >> Registered in Finland >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 14:49:55 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 14:49:55 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: Thanks Bob, That pagepool comment has just answered my next question! But it doesn't seem to be working. Here's my mmdiag output: === mmdiag: lroc === LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' status Running Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 Max capacity: 1151997 MB, currently in use: 0 MB Statistics from: Mon Jun 5 13:40:50 2017 Total objects stored 0 (0 MB) recalled 0 (0 MB) objects failed to store 0 failed to recall 0 failed to inval 0 objects queried 0 (0 MB) not found 0 = 0.00 % objects invalidated 0 (0 MB) Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Inode objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Directory objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Data objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 agent inserts=0, reads=0 response times (usec): insert min/max/avg=0/0/0 read min/max/avg=0/0/0 ssd writeIOs=0, writePages=0 readIOs=0, readPages=0 response times (usec): write min/max/avg=0/0/0 read min/max/avg=0/0/0 I've restarted GPFS on that node just in case but that didn't seem to help. I have LROC on a node that DOESN'T have direct access to an NSD so will hopefully cache files that get requested over NFS. How often are these stats updated? The Statistics line doesn't seem to update when running the command again. Dave, ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 13:48, Oesterlin, Robert wrote: > Hi Dave > > > > I?ve done a large-scale (600 node) LROC deployment here - feel free to > reach out if you have questions. > > > > mmdiag --lroc is about all there is but it does give you a pretty good > idea how the cache is performing but you can?t tell which files are cached. > Also, watch out that the LROC cached will steal pagepool memory (1% of the > LROC cache size) > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > > > > > *From: * on behalf of Dave > Goodbourn > *Reply-To: *gpfsug main discussion list > *Date: *Monday, June 5, 2017 at 7:19 AM > *To: *gpfsug main discussion list > *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes > > > > I'm testing out the LROC idea. All seems to be working well, but, is there > anyway to monitor what's cached? How full it might be? The performance etc?? > > > > I can see some stats in mmfsadm dump lroc but that's about it. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 14:55:22 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 14:55:22 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: OK slightly ignore that last email. It's still not updating the output but I realise the Stats from line is when they started so probably won't update! :( Still nothing seems to being cached though. ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 14:49, Dave Goodbourn wrote: > Thanks Bob, > > That pagepool comment has just answered my next question! > > But it doesn't seem to be working. Here's my mmdiag output: > > === mmdiag: lroc === > LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' > status Running > Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 > Max capacity: 1151997 MB, currently in use: 0 MB > Statistics from: Mon Jun 5 13:40:50 2017 > > Total objects stored 0 (0 MB) recalled 0 (0 MB) > objects failed to store 0 failed to recall 0 failed to inval 0 > objects queried 0 (0 MB) not found 0 = 0.00 % > objects invalidated 0 (0 MB) > > Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Inode objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Directory objects failed to store 0 failed to recall 0 failed to > query 0 failed to inval 0 > > Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Data objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > agent inserts=0, reads=0 > response times (usec): > insert min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > ssd writeIOs=0, writePages=0 > readIOs=0, readPages=0 > response times (usec): > write min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > > I've restarted GPFS on that node just in case but that didn't seem to > help. I have LROC on a node that DOESN'T have direct access to an NSD so > will hopefully cache files that get requested over NFS. > > How often are these stats updated? The Statistics line doesn't seem to > update when running the command again. > > Dave, > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 13:48, Oesterlin, Robert > wrote: > >> Hi Dave >> >> >> >> I?ve done a large-scale (600 node) LROC deployment here - feel free to >> reach out if you have questions. >> >> >> >> mmdiag --lroc is about all there is but it does give you a pretty good >> idea how the cache is performing but you can?t tell which files are cached. >> Also, watch out that the LROC cached will steal pagepool memory (1% of the >> LROC cache size) >> >> >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> >> >> >> >> *From: * on behalf of Dave >> Goodbourn >> *Reply-To: *gpfsug main discussion list > > >> *Date: *Monday, June 5, 2017 at 7:19 AM >> *To: *gpfsug main discussion list >> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >> >> >> >> I'm testing out the LROC idea. All seems to be working well, but, is >> there anyway to monitor what's cached? How full it might be? The >> performance etc?? >> >> >> >> I can see some stats in mmfsadm dump lroc but that's about it. >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Jun 5 14:59:07 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Mon, 5 Jun 2017 13:59:07 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> , Message-ID: We've seen exactly this behaviour. Removing and readding the lroc nsd device worked for us. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of dave at milk-vfx.com [dave at milk-vfx.com] Sent: 05 June 2017 14:55 To: Oesterlin, Robert Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] NSD access routes OK slightly ignore that last email. It's still not updating the output but I realise the Stats from line is when they started so probably won't update! :( Still nothing seems to being cached though. ---------------------------------------------------- Dave Goodbourn Head of Systems MILK VISUAL EFFECTS [http://www.milk-vfx.com/src/milk_email_logo.jpg] 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 On 5 June 2017 at 14:49, Dave Goodbourn > wrote: Thanks Bob, That pagepool comment has just answered my next question! But it doesn't seem to be working. Here's my mmdiag output: === mmdiag: lroc === LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' status Running Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 Max capacity: 1151997 MB, currently in use: 0 MB Statistics from: Mon Jun 5 13:40:50 2017 Total objects stored 0 (0 MB) recalled 0 (0 MB) objects failed to store 0 failed to recall 0 failed to inval 0 objects queried 0 (0 MB) not found 0 = 0.00 % objects invalidated 0 (0 MB) Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Inode objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Directory objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Data objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 agent inserts=0, reads=0 response times (usec): insert min/max/avg=0/0/0 read min/max/avg=0/0/0 ssd writeIOs=0, writePages=0 readIOs=0, readPages=0 response times (usec): write min/max/avg=0/0/0 read min/max/avg=0/0/0 I've restarted GPFS on that node just in case but that didn't seem to help. I have LROC on a node that DOESN'T have direct access to an NSD so will hopefully cache files that get requested over NFS. How often are these stats updated? The Statistics line doesn't seem to update when running the command again. Dave, ---------------------------------------------------- Dave Goodbourn Head of Systems MILK VISUAL EFFECTS [http://www.milk-vfx.com/src/milk_email_logo.jpg] 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 On 5 June 2017 at 13:48, Oesterlin, Robert > wrote: Hi Dave I?ve done a large-scale (600 node) LROC deployment here - feel free to reach out if you have questions. mmdiag --lroc is about all there is but it does give you a pretty good idea how the cache is performing but you can?t tell which files are cached. Also, watch out that the LROC cached will steal pagepool memory (1% of the LROC cache size) Bob Oesterlin Sr Principal Storage Engineer, Nuance From: > on behalf of Dave Goodbourn > Reply-To: gpfsug main discussion list > Date: Monday, June 5, 2017 at 7:19 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] NSD access routes I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. From oehmes at gmail.com Mon Jun 5 14:59:44 2017 From: oehmes at gmail.com (Sven Oehme) Date: Mon, 05 Jun 2017 13:59:44 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: if you are using O_DIRECT calls they will be ignored by default for LROC, same for encrypted data. how exactly are you testing this? On Mon, Jun 5, 2017 at 6:50 AM Dave Goodbourn wrote: > Thanks Bob, > > That pagepool comment has just answered my next question! > > But it doesn't seem to be working. Here's my mmdiag output: > > === mmdiag: lroc === > LROC Device(s): > '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' > status Running > Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 > Max capacity: 1151997 MB, currently in use: 0 MB > Statistics from: Mon Jun 5 13:40:50 2017 > > Total objects stored 0 (0 MB) recalled 0 (0 MB) > objects failed to store 0 failed to recall 0 failed to inval 0 > objects queried 0 (0 MB) not found 0 = 0.00 % > objects invalidated 0 (0 MB) > > Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Inode objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Directory objects failed to store 0 failed to recall 0 failed to > query 0 failed to inval 0 > > Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Data objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > agent inserts=0, reads=0 > response times (usec): > insert min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > ssd writeIOs=0, writePages=0 > readIOs=0, readPages=0 > response times (usec): > write min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > > I've restarted GPFS on that node just in case but that didn't seem to > help. I have LROC on a node that DOESN'T have direct access to an NSD so > will hopefully cache files that get requested over NFS. > > How often are these stats updated? The Statistics line doesn't seem to > update when running the command again. > > Dave, > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 13:48, Oesterlin, Robert > wrote: > >> Hi Dave >> >> >> >> I?ve done a large-scale (600 node) LROC deployment here - feel free to >> reach out if you have questions. >> >> >> >> mmdiag --lroc is about all there is but it does give you a pretty good >> idea how the cache is performing but you can?t tell which files are cached. >> Also, watch out that the LROC cached will steal pagepool memory (1% of the >> LROC cache size) >> >> >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> >> >> >> >> *From: * on behalf of Dave >> Goodbourn >> *Reply-To: *gpfsug main discussion list > > >> *Date: *Monday, June 5, 2017 at 7:19 AM >> *To: *gpfsug main discussion list >> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >> >> >> >> I'm testing out the LROC idea. All seems to be working well, but, is >> there anyway to monitor what's cached? How full it might be? The >> performance etc?? >> >> >> >> I can see some stats in mmfsadm dump lroc but that's about it. >> >> >> >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 15:00:45 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 15:00:45 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: OK I'm going to hang my head in the corner...RTFM...I've not filled the memory buffer pool yet so I doubt it will have anything in it yet!! :( ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 14:55, Dave Goodbourn wrote: > OK slightly ignore that last email. It's still not updating the output but > I realise the Stats from line is when they started so probably won't > update! :( > > Still nothing seems to being cached though. > > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 14:49, Dave Goodbourn wrote: > >> Thanks Bob, >> >> That pagepool comment has just answered my next question! >> >> But it doesn't seem to be working. Here's my mmdiag output: >> >> === mmdiag: lroc === >> LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF >> 0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' status Running >> Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 >> Max capacity: 1151997 MB, currently in use: 0 MB >> Statistics from: Mon Jun 5 13:40:50 2017 >> >> Total objects stored 0 (0 MB) recalled 0 (0 MB) >> objects failed to store 0 failed to recall 0 failed to inval 0 >> objects queried 0 (0 MB) not found 0 = 0.00 % >> objects invalidated 0 (0 MB) >> >> Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >> Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >> Inode objects failed to store 0 failed to recall 0 failed to query >> 0 failed to inval 0 >> >> Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >> Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >> Directory objects failed to store 0 failed to recall 0 failed to >> query 0 failed to inval 0 >> >> Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >> Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >> Data objects failed to store 0 failed to recall 0 failed to query 0 >> failed to inval 0 >> >> agent inserts=0, reads=0 >> response times (usec): >> insert min/max/avg=0/0/0 >> read min/max/avg=0/0/0 >> >> ssd writeIOs=0, writePages=0 >> readIOs=0, readPages=0 >> response times (usec): >> write min/max/avg=0/0/0 >> read min/max/avg=0/0/0 >> >> >> I've restarted GPFS on that node just in case but that didn't seem to >> help. I have LROC on a node that DOESN'T have direct access to an NSD so >> will hopefully cache files that get requested over NFS. >> >> How often are these stats updated? The Statistics line doesn't seem to >> update when running the command again. >> >> Dave, >> ---------------------------------------------------- >> *Dave Goodbourn* >> Head of Systems >> *MILK VISUAL EFFECTS* >> >> 5th floor, Threeways House, >> 40-44 Clipstone Street London, W1W 5DW >> Tel: *+44 (0)20 3697 8448* >> Mob: *+44 (0)7917 411 069* >> >> On 5 June 2017 at 13:48, Oesterlin, Robert >> wrote: >> >>> Hi Dave >>> >>> >>> >>> I?ve done a large-scale (600 node) LROC deployment here - feel free to >>> reach out if you have questions. >>> >>> >>> >>> mmdiag --lroc is about all there is but it does give you a pretty good >>> idea how the cache is performing but you can?t tell which files are cached. >>> Also, watch out that the LROC cached will steal pagepool memory (1% of the >>> LROC cache size) >>> >>> >>> >>> Bob Oesterlin >>> Sr Principal Storage Engineer, Nuance >>> >>> >>> >>> >>> >>> >>> >>> *From: * on behalf of Dave >>> Goodbourn >>> *Reply-To: *gpfsug main discussion list >> org> >>> *Date: *Monday, June 5, 2017 at 7:19 AM >>> *To: *gpfsug main discussion list >>> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >>> >>> >>> >>> I'm testing out the LROC idea. All seems to be working well, but, is >>> there anyway to monitor what's cached? How full it might be? The >>> performance etc?? >>> >>> >>> >>> I can see some stats in mmfsadm dump lroc but that's about it. >>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Mon Jun 5 15:03:28 2017 From: oehmes at gmail.com (Sven Oehme) Date: Mon, 05 Jun 2017 14:03:28 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: yes as long as you haven't pushed anything to it (means pagepool got under enough pressure to free up space) you won't see anything in the stats :-) sven On Mon, Jun 5, 2017 at 7:00 AM Dave Goodbourn wrote: > OK I'm going to hang my head in the corner...RTFM...I've not filled the > memory buffer pool yet so I doubt it will have anything in it yet!! :( > > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 14:55, Dave Goodbourn wrote: > >> OK slightly ignore that last email. It's still not updating the output >> but I realise the Stats from line is when they started so probably won't >> update! :( >> >> Still nothing seems to being cached though. >> >> ---------------------------------------------------- >> *Dave Goodbourn* >> Head of Systems >> *MILK VISUAL EFFECTS* >> >> 5th floor, Threeways House, >> 40-44 Clipstone Street London, W1W 5DW >> Tel: *+44 (0)20 3697 8448* >> Mob: *+44 (0)7917 411 069* >> >> On 5 June 2017 at 14:49, Dave Goodbourn wrote: >> >>> Thanks Bob, >>> >>> That pagepool comment has just answered my next question! >>> >>> But it doesn't seem to be working. Here's my mmdiag output: >>> >>> === mmdiag: lroc === >>> LROC Device(s): >>> '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' >>> status Running >>> Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 >>> Max capacity: 1151997 MB, currently in use: 0 MB >>> Statistics from: Mon Jun 5 13:40:50 2017 >>> >>> Total objects stored 0 (0 MB) recalled 0 (0 MB) >>> objects failed to store 0 failed to recall 0 failed to inval 0 >>> objects queried 0 (0 MB) not found 0 = 0.00 % >>> objects invalidated 0 (0 MB) >>> >>> Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>> Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>> Inode objects failed to store 0 failed to recall 0 failed to query >>> 0 failed to inval 0 >>> >>> Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>> Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>> Directory objects failed to store 0 failed to recall 0 failed to >>> query 0 failed to inval 0 >>> >>> Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>> Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>> Data objects failed to store 0 failed to recall 0 failed to query >>> 0 failed to inval 0 >>> >>> agent inserts=0, reads=0 >>> response times (usec): >>> insert min/max/avg=0/0/0 >>> read min/max/avg=0/0/0 >>> >>> ssd writeIOs=0, writePages=0 >>> readIOs=0, readPages=0 >>> response times (usec): >>> write min/max/avg=0/0/0 >>> read min/max/avg=0/0/0 >>> >>> >>> I've restarted GPFS on that node just in case but that didn't seem to >>> help. I have LROC on a node that DOESN'T have direct access to an NSD so >>> will hopefully cache files that get requested over NFS. >>> >>> How often are these stats updated? The Statistics line doesn't seem to >>> update when running the command again. >>> >>> Dave, >>> ---------------------------------------------------- >>> *Dave Goodbourn* >>> Head of Systems >>> *MILK VISUAL EFFECTS* >>> >>> 5th floor, Threeways House, >>> 40-44 Clipstone Street London, W1W 5DW >>> Tel: *+44 (0)20 3697 8448* >>> Mob: *+44 (0)7917 411 069* >>> >>> On 5 June 2017 at 13:48, Oesterlin, Robert >>> wrote: >>> >>>> Hi Dave >>>> >>>> >>>> >>>> I?ve done a large-scale (600 node) LROC deployment here - feel free to >>>> reach out if you have questions. >>>> >>>> >>>> >>>> mmdiag --lroc is about all there is but it does give you a pretty good >>>> idea how the cache is performing but you can?t tell which files are cached. >>>> Also, watch out that the LROC cached will steal pagepool memory (1% of the >>>> LROC cache size) >>>> >>>> >>>> >>>> Bob Oesterlin >>>> Sr Principal Storage Engineer, Nuance >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *From: * on behalf of Dave >>>> Goodbourn >>>> *Reply-To: *gpfsug main discussion list < >>>> gpfsug-discuss at spectrumscale.org> >>>> *Date: *Monday, June 5, 2017 at 7:19 AM >>>> *To: *gpfsug main discussion list >>>> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >>>> >>>> >>>> >>>> I'm testing out the LROC idea. All seems to be working well, but, is >>>> there anyway to monitor what's cached? How full it might be? The >>>> performance etc?? >>>> >>>> >>>> >>>> I can see some stats in mmfsadm dump lroc but that's about it. >>>> >>>> >>>> >>>> >>> >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 15:15:00 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 15:15:00 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: Ha! A quick shrink of the pagepool and we're in action! Thanks all. Dave. ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 15:03, Sven Oehme wrote: > yes as long as you haven't pushed anything to it (means pagepool got under > enough pressure to free up space) you won't see anything in the stats :-) > > sven > > > On Mon, Jun 5, 2017 at 7:00 AM Dave Goodbourn wrote: > >> OK I'm going to hang my head in the corner...RTFM...I've not filled the >> memory buffer pool yet so I doubt it will have anything in it yet!! :( >> >> ---------------------------------------------------- >> *Dave Goodbourn* >> Head of Systems >> *MILK VISUAL EFFECTS* >> >> 5th floor, Threeways House, >> 40-44 Clipstone Street London, W1W 5DW >> Tel: *+44 (0)20 3697 8448* >> Mob: *+44 (0)7917 411 069* >> >> On 5 June 2017 at 14:55, Dave Goodbourn wrote: >> >>> OK slightly ignore that last email. It's still not updating the output >>> but I realise the Stats from line is when they started so probably won't >>> update! :( >>> >>> Still nothing seems to being cached though. >>> >>> ---------------------------------------------------- >>> *Dave Goodbourn* >>> Head of Systems >>> *MILK VISUAL EFFECTS* >>> >>> 5th floor, Threeways House, >>> 40-44 Clipstone Street London, W1W 5DW >>> Tel: *+44 (0)20 3697 8448* >>> Mob: *+44 (0)7917 411 069* >>> >>> On 5 June 2017 at 14:49, Dave Goodbourn wrote: >>> >>>> Thanks Bob, >>>> >>>> That pagepool comment has just answered my next question! >>>> >>>> But it doesn't seem to be working. Here's my mmdiag output: >>>> >>>> === mmdiag: lroc === >>>> LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' >>>> status Running >>>> Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 >>>> Max capacity: 1151997 MB, currently in use: 0 MB >>>> Statistics from: Mon Jun 5 13:40:50 2017 >>>> >>>> Total objects stored 0 (0 MB) recalled 0 (0 MB) >>>> objects failed to store 0 failed to recall 0 failed to inval 0 >>>> objects queried 0 (0 MB) not found 0 = 0.00 % >>>> objects invalidated 0 (0 MB) >>>> >>>> Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>>> Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>>> Inode objects failed to store 0 failed to recall 0 failed to >>>> query 0 failed to inval 0 >>>> >>>> Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>>> Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>>> Directory objects failed to store 0 failed to recall 0 failed to >>>> query 0 failed to inval 0 >>>> >>>> Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>>> Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>>> Data objects failed to store 0 failed to recall 0 failed to query >>>> 0 failed to inval 0 >>>> >>>> agent inserts=0, reads=0 >>>> response times (usec): >>>> insert min/max/avg=0/0/0 >>>> read min/max/avg=0/0/0 >>>> >>>> ssd writeIOs=0, writePages=0 >>>> readIOs=0, readPages=0 >>>> response times (usec): >>>> write min/max/avg=0/0/0 >>>> read min/max/avg=0/0/0 >>>> >>>> >>>> I've restarted GPFS on that node just in case but that didn't seem to >>>> help. I have LROC on a node that DOESN'T have direct access to an NSD so >>>> will hopefully cache files that get requested over NFS. >>>> >>>> How often are these stats updated? The Statistics line doesn't seem to >>>> update when running the command again. >>>> >>>> Dave, >>>> ---------------------------------------------------- >>>> *Dave Goodbourn* >>>> Head of Systems >>>> *MILK VISUAL EFFECTS* >>>> >>>> 5th floor, Threeways House, >>>> 40-44 Clipstone Street London, W1W 5DW >>>> Tel: *+44 (0)20 3697 8448* >>>> Mob: *+44 (0)7917 411 069* >>>> >>>> On 5 June 2017 at 13:48, Oesterlin, Robert >>> > wrote: >>>> >>>>> Hi Dave >>>>> >>>>> >>>>> >>>>> I?ve done a large-scale (600 node) LROC deployment here - feel free to >>>>> reach out if you have questions. >>>>> >>>>> >>>>> >>>>> mmdiag --lroc is about all there is but it does give you a pretty good >>>>> idea how the cache is performing but you can?t tell which files are cached. >>>>> Also, watch out that the LROC cached will steal pagepool memory (1% of the >>>>> LROC cache size) >>>>> >>>>> >>>>> >>>>> Bob Oesterlin >>>>> Sr Principal Storage Engineer, Nuance >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *From: * on behalf of Dave >>>>> Goodbourn >>>>> *Reply-To: *gpfsug main discussion list >>>> org> >>>>> *Date: *Monday, June 5, 2017 at 7:19 AM >>>>> *To: *gpfsug main discussion list >>>>> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >>>>> >>>>> >>>>> >>>>> I'm testing out the LROC idea. All seems to be working well, but, is >>>>> there anyway to monitor what's cached? How full it might be? The >>>>> performance etc?? >>>>> >>>>> >>>>> >>>>> I can see some stats in mmfsadm dump lroc but that's about it. >>>>> >>>>> >>>>> >>>>> >>>> >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 5 16:54:09 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 5 Jun 2017 15:54:09 +0000 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add Message-ID: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> Our node build process re-adds a node to the cluster and then does a ?service gpfs start?, but GPFS doesn?t start. From the build log: + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' + rc=0 + chkconfig gpfs on + service gpfs start The ?service gpfs start? command hangs and never seems to return. If I look at the process tree: [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12292 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e s/\.num$// 21639 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 This is GPFS 4.2.2-1 This seems to occur only on the initial startup after build - if I try to start GPFS again, it works just fine - any ideas on what it?s sitting here waiting? Nothing in mmfslog (does not exist) Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Mon Jun 5 20:54:31 2017 From: ewahl at osc.edu (Edward Wahl) Date: Mon, 5 Jun 2017 15:54:31 -0400 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add In-Reply-To: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> References: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> Message-ID: <20170605155431.75b42322@osc.edu> Just a thought, as we noticed the EXACT opposite of this, and what I think is new behavior in either mmmount or mmfsfuncs.. Does the file system exist in your /etc/fstab (or AIX equiv) yet? Ed On Mon, 5 Jun 2017 15:54:09 +0000 "Oesterlin, Robert" wrote: > Our node build process re-adds a node to the cluster and then does a ?service > gpfs start?, but GPFS doesn?t start. From the build log: > > + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com > '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' > + rc=0 > + chkconfig gpfs on > + service gpfs start > > The ?service gpfs start? command hangs and never seems to return. > > If I look at the process tree: > > [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" > 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post > 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 > 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? > S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S > 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > 12292 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num > 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e > s/\.num$// 21639 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 > > This is GPFS 4.2.2-1 > > This seems to occur only on the initial startup after build - if I try to > start GPFS again, it works just fine - any ideas on what it?s sitting here > waiting? Nothing in mmfslog (does not exist) > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From ewahl at osc.edu Mon Jun 5 20:56:55 2017 From: ewahl at osc.edu (Edward Wahl) Date: Mon, 5 Jun 2017 15:56:55 -0400 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add In-Reply-To: <20170605155431.75b42322@osc.edu> References: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> <20170605155431.75b42322@osc.edu> Message-ID: <20170605155655.3ce54084@osc.edu> On Mon, 5 Jun 2017 15:54:31 -0400 Edward Wahl wrote: > Just a thought, as we noticed the EXACT opposite of this, and what I think is > new behavior in either mmmount or .. Does the file system exist in > your /etc/fstab (or AIX equiv) yet? Apologies, I meant mmsdrfsdef, not mmfsfuncs. Ed > > Ed > > On Mon, 5 Jun 2017 15:54:09 +0000 > "Oesterlin, Robert" wrote: > > > Our node build process re-adds a node to the cluster and then does a > > ?service gpfs start?, but GPFS doesn?t start. >From the build log: > > > > + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com > > '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' > > + rc=0 > > + chkconfig gpfs on > > + service gpfs start > > > > The ?service gpfs start? command hangs and never seems to return. > > > > If I look at the process tree: > > > > [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" > > 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post > > 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 > > 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? > > S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S > > 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > > 12292 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > > 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num > > 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e > > s/\.num$// 21639 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 > > > > This is GPFS 4.2.2-1 > > > > This seems to occur only on the initial startup after build - if I try to > > start GPFS again, it works just fine - any ideas on what it?s sitting here > > waiting? Nothing in mmfslog (does not exist) > > > > Bob Oesterlin > > Sr Principal Storage Engineer, Nuance > > 507-269-0413 > > > > > > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From scale at us.ibm.com Mon Jun 5 22:49:23 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 5 Jun 2017 17:49:23 -0400 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add In-Reply-To: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> References: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> Message-ID: Looks like a bug in the code. The command hung in grep command. It has missing argument. Please open a PMR to have this fix. Instead of "service gpfs start", can you use mmstartup? You can also try to run mm list command before service gpfs start as a workaround. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/05/2017 11:54 AM Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add Sent by: gpfsug-discuss-bounces at spectrumscale.org Our node build process re-adds a node to the cluster and then does a ?service gpfs start?, but GPFS doesn?t start. From the build log: + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' + rc=0 + chkconfig gpfs on + service gpfs start The ?service gpfs start? command hangs and never seems to return. If I look at the process tree: [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12292 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e s/\.num$// 21639 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 This is GPFS 4.2.2-1 This seems to occur only on the initial startup after build - if I try to start GPFS again, it works just fine - any ideas on what it?s sitting here waiting? Nothing in mmfslog (does not exist) Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From stijn.deweirdt at ugent.be Tue Jun 6 08:05:06 2017 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Tue, 6 Jun 2017 09:05:06 +0200 Subject: [gpfsug-discuss] gpfs waiters debugging Message-ID: hi all, we have recently been hit by quite a few cases that triggered long waiters. we are aware of the excellent slides http://files.gpfsug.org/presentations/2017/NERSC/GPFS-Troubleshooting-Apr-2017.pdf but we are wondering if and how we can cause those waiters ourself, so we can train ourself in debugging and resolving them (either on test system or in controlled environment on the production clusters). all hints welcome. stijn From Robert.Oesterlin at nuance.com Tue Jun 6 12:44:31 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 6 Jun 2017 11:44:31 +0000 Subject: [gpfsug-discuss] gpfs waiters debugging Message-ID: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> Hi Stijn You need to provide some more details on the type and duration of the waiters before the group can offer some advice. Bob Oesterlin Sr Principal Storage Engineer, Nuance On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Stijn De Weirdt" wrote: but we are wondering if and how we can cause those waiters ourself, so we can train ourself in debugging and resolving them (either on test system or in controlled environment on the production clusters). all hints welcome. stijn _______________________________________________ From stijn.deweirdt at ugent.be Tue Jun 6 13:29:43 2017 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Tue, 6 Jun 2017 14:29:43 +0200 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> Message-ID: <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> hi bob, waiters from RPC replies and/or threads waiting on mutex are most "popular". but my question is not how to resolve them, the question is how to create such a waiter so we can train ourself in grep and mmfsadm etc etc we want to recreate the waiters a few times, try out some things and either script or at least put instructions on our internal wiki what to do. the instructions in the slides are clear enough, but there are a lot of slides, and typically when this occurs offshift, you don't want to start with rereading the slides and wondering what to do next; let alone debug scripts ;) thanks, stijn On 06/06/2017 01:44 PM, Oesterlin, Robert wrote: > Hi Stijn > > You need to provide some more details on the type and duration of the waiters before the group can offer some advice. > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Stijn De Weirdt" wrote: > > > but we are wondering if and how we can cause those waiters ourself, so > we can train ourself in debugging and resolving them (either on test > system or in controlled environment on the production clusters). > > all hints welcome. > > stijn > _______________________________________________ > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From stockf at us.ibm.com Tue Jun 6 13:57:00 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 6 Jun 2017 08:57:00 -0400 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> Message-ID: Realize that generally any waiter under 1 second should be ignored. In an active GPFS system there are always waiters and the greater the use of the system likely the more waiters you will see. The point is waiters themselves are not an indication your system is having problems. As for creating them any steady level of activity against the file system should cause waiters to appear, though most should be of a short duration. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: Stijn De Weirdt To: gpfsug-discuss at spectrumscale.org Date: 06/06/2017 08:31 AM Subject: Re: [gpfsug-discuss] gpfs waiters debugging Sent by: gpfsug-discuss-bounces at spectrumscale.org hi bob, waiters from RPC replies and/or threads waiting on mutex are most "popular". but my question is not how to resolve them, the question is how to create such a waiter so we can train ourself in grep and mmfsadm etc etc we want to recreate the waiters a few times, try out some things and either script or at least put instructions on our internal wiki what to do. the instructions in the slides are clear enough, but there are a lot of slides, and typically when this occurs offshift, you don't want to start with rereading the slides and wondering what to do next; let alone debug scripts ;) thanks, stijn On 06/06/2017 01:44 PM, Oesterlin, Robert wrote: > Hi Stijn > > You need to provide some more details on the type and duration of the waiters before the group can offer some advice. > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Stijn De Weirdt" wrote: > > > but we are wondering if and how we can cause those waiters ourself, so > we can train ourself in debugging and resolving them (either on test > system or in controlled environment on the production clusters). > > all hints welcome. > > stijn > _______________________________________________ > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Tue Jun 6 14:06:57 2017 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Tue, 6 Jun 2017 15:06:57 +0200 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> Message-ID: oh sure, i meant waiters that last > 300 seconds or so (something that could trigger deadlock). obviously we're not interested in debugging the short ones, it's not that gpfs doesn't work or anything ;) stijn On 06/06/2017 02:57 PM, Frederick Stock wrote: > Realize that generally any waiter under 1 second should be ignored. In an > active GPFS system there are always waiters and the greater the use of the > system likely the more waiters you will see. The point is waiters > themselves are not an indication your system is having problems. > > As for creating them any steady level of activity against the file system > should cause waiters to appear, though most should be of a short duration. > > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > stockf at us.ibm.com > > > > From: Stijn De Weirdt > To: gpfsug-discuss at spectrumscale.org > Date: 06/06/2017 08:31 AM > Subject: Re: [gpfsug-discuss] gpfs waiters debugging > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > hi bob, > > waiters from RPC replies and/or threads waiting on mutex are most > "popular". > > but my question is not how to resolve them, the question is how to > create such a waiter so we can train ourself in grep and mmfsadm etc etc > > we want to recreate the waiters a few times, try out some things and > either script or at least put instructions on our internal wiki what to > do. > > the instructions in the slides are clear enough, but there are a lot of > slides, and typically when this occurs offshift, you don't want to start > with rereading the slides and wondering what to do next; let alone debug > scripts ;) > > thanks, > > stijn > > On 06/06/2017 01:44 PM, Oesterlin, Robert wrote: >> Hi Stijn >> >> You need to provide some more details on the type and duration of the > waiters before the group can offer some advice. >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of Stijn De Weirdt" stijn.deweirdt at ugent.be> wrote: >> >> >> but we are wondering if and how we can cause those waiters ourself, > so >> we can train ourself in debugging and resolving them (either on test >> system or in controlled environment on the production clusters). >> >> all hints welcome. >> >> stijn >> _______________________________________________ >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From valdis.kletnieks at vt.edu Tue Jun 6 17:45:51 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 06 Jun 2017 12:45:51 -0400 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> Message-ID: <6873.1496767551@turing-police.cc.vt.edu> On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt said: > oh sure, i meant waiters that last > 300 seconds or so (something that > could trigger deadlock). obviously we're not interested in debugging the > short ones, it's not that gpfs doesn't work or anything ;) At least at one time, a lot of the mm(whatever) administrative commands would leave one dangling waiter for the duration of the command - which could be a while if the command was mmdeldisk or mmrestripefs. I admit not having specifically checked for gpfs 4.2, but it was true for 3.2 through 4.1.... And my addition to the collective debugging knowledge: A bash one-liner to dump all the waiters across a cluster, sorted by wait time. Note that our clusters tend to be 5-8 servers, this may be painful for those of you who have 400+ node clusters. :) ##!/bin/bash for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh $i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done | sort -n -r -k 3 -t' ' We've found it useful - if you have 1 waiter on one node that's 1278 seconds old, and 3 other nodes have waiters that are 1275 seconds old, it's a good chance the other 3 nodes waiters are waiting on the first node's waiter to resolve itself.... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From stockf at us.ibm.com Tue Jun 6 17:54:06 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 6 Jun 2017 12:54:06 -0400 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: <6873.1496767551@turing-police.cc.vt.edu> References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com><3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> <6873.1496767551@turing-police.cc.vt.edu> Message-ID: On recent releases you can accomplish the same with the command, "mmlsnode -N waiters -L". Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list Date: 06/06/2017 12:46 PM Subject: Re: [gpfsug-discuss] gpfs waiters debugging Sent by: gpfsug-discuss-bounces at spectrumscale.org On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt said: > oh sure, i meant waiters that last > 300 seconds or so (something that > could trigger deadlock). obviously we're not interested in debugging the > short ones, it's not that gpfs doesn't work or anything ;) At least at one time, a lot of the mm(whatever) administrative commands would leave one dangling waiter for the duration of the command - which could be a while if the command was mmdeldisk or mmrestripefs. I admit not having specifically checked for gpfs 4.2, but it was true for 3.2 through 4.1.... And my addition to the collective debugging knowledge: A bash one-liner to dump all the waiters across a cluster, sorted by wait time. Note that our clusters tend to be 5-8 servers, this may be painful for those of you who have 400+ node clusters. :) ##!/bin/bash for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh $i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done | sort -n -r -k 3 -t' ' We've found it useful - if you have 1 waiter on one node that's 1278 seconds old, and 3 other nodes have waiters that are 1275 seconds old, it's a good chance the other 3 nodes waiters are waiting on the first node's waiter to resolve itself.... [attachment "attltepl.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Tue Jun 6 19:05:15 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 6 Jun 2017 14:05:15 -0400 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) In-Reply-To: <20170602111241.56882fx2qr2yz2ax@support.scinet.utoronto.ca> References: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca>, <20170602111241.56882fx2qr2yz2ax@support.scinet.utoronto.ca> Message-ID: Hi, Just as Jaime has explained, any GPFS node in the cluster, can induce a recall (as he called "staged") by access to file data. It is not optimized by tape order, and a dynamic file access of any pattern, such as "find" or "cat *" will surely result in an inefficient processing of the data recall if all data lives in physical tape. But if migrated data lives on spinning disk on the TSM server, there is no harm in such a recall pattern because recalls from a disk pool incur no significant overhead or delay for tape loading and positioning. Unprivileged users may not run "dsmcrecall" because creating a DMAPI session as the dsmrecall program must do, requires admin user privilege on that node. You may be able to wrap dsmrecall in a set-uid wrapper if you want to permit users to run that, but of course that comes with the danger that a recall storm could monopolize resources on your cluster. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Jaime Pinto" To: "Andrew Beattie" Cc: gpfsug-discuss at spectrumscale.org Date: 06/02/2017 11:13 AM Subject: Re: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) Sent by: gpfsug-discuss-bounces at spectrumscale.org It has been a while since I used HSM with GPFS via TSM, but as far as I can remember, unprivileged users can run dsmmigrate and dsmrecall. Based on the instructions on the link, dsmrecall may now leverage the Recommended Access Order (RAO) available on enterprise drives, however root would have to be the one to invoke that feature. In that case we may have to develop a middleware/wrapper for dsmrecall that will run as root and act on behalf of the user when optimization is requested. Someone here more familiar with the latest version of TSM-HSM may be able to give us some hints on how people are doing this in practice. Jaime Quoting "Andrew Beattie" : > Thanks Jaime, How do you get around Optimised recalls? from what I > can see the optimised recall process needs a root level account to > retrieve a list of files > https://www.ibm.com/support/knowledgecenter/SSSR2R_7.1.1/com.ibm.itsm.hsmul.doc/c_recall_optimized_tape.html [1] > Regards, Andrew Beattie Software Defined Storage - IT Specialist > Phone: 614-2133-7927 E-mail: abeattie at au1.ibm.com[2] ----- > Original message ----- > From: "Jaime Pinto" > To: "gpfsug main discussion list" , > "Andrew Beattie" > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - > Space Management (GPFS HSM) > Date: Fri, Jun 2, 2017 7:28 PM > We have that situation. > Users don't need to login to NSD's > > What you need is to add at least one gpfs client to the cluster (or > multi-cluster), mount the DMAPI enabled file system, and use that > node > as a gateway for end-users. They can access the contents on the mount > > point with their own underprivileged accounts. > > Whether or not on a schedule, the moment an application or linux > command (such as cp, cat, vi, etc) accesses a stub, the file will be > > staged. > > Jaime > > Quoting "Andrew Beattie" : > >> Quick question, Does anyone have a Scale / GPFS environment (HPC) >> where users need the ability to recall data sets after they have > been >> stubbed, but only System Administrators are permitted to log onto > the >> NSD servers for security purposes. And if so how do you provide > the >> ability for the users to schedule their data set recalls? > Regards, >> Andrew Beattie Software Defined Storage - IT Specialist Phone: >> 614-2133-7927 E-mail: abeattie at au1.ibm.com[1] >> >> >> Links: >> ------ >> [1] mailto:abeattie at au1.ibm.com[3] >> > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials[4] > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jun 6 19:15:22 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 6 Jun 2017 18:15:22 +0000 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> <6873.1496767551@turing-police.cc.vt.edu> Message-ID: All, mmlsnode -N waiters is great ? I also appreciate the ?-s? option to it. Very helpful when you know the problem started say, slightly more than half an hour ago and you therefore don?t care about sub-1800 second waiters? Kevin On Jun 6, 2017, at 11:54 AM, Frederick Stock > wrote: On recent releases you can accomplish the same with the command, "mmlsnode -N waiters -L". Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list > Date: 06/06/2017 12:46 PM Subject: Re: [gpfsug-discuss] gpfs waiters debugging Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt said: > oh sure, i meant waiters that last > 300 seconds or so (something that > could trigger deadlock). obviously we're not interested in debugging the > short ones, it's not that gpfs doesn't work or anything ;) At least at one time, a lot of the mm(whatever) administrative commands would leave one dangling waiter for the duration of the command - which could be a while if the command was mmdeldisk or mmrestripefs. I admit not having specifically checked for gpfs 4.2, but it was true for 3.2 through 4.1.... And my addition to the collective debugging knowledge: A bash one-liner to dump all the waiters across a cluster, sorted by wait time. Note that our clusters tend to be 5-8 servers, this may be painful for those of you who have 400+ node clusters. :) ##!/bin/bash for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh $i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done | sort -n -r -k 3 -t' ' We've found it useful - if you have 1 waiter on one node that's 1278 seconds old, and 3 other nodes have waiters that are 1275 seconds old, it's a good chance the other 3 nodes waiters are waiting on the first node's waiter to resolve itself.... [attachment "attltepl.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Tue Jun 6 21:31:01 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 06 Jun 2017 16:31:01 -0400 Subject: [gpfsug-discuss] mmapplypolicy and ltfsee - identifying progress... Message-ID: <25944.1496781061@turing-police.cc.vt.edu> So I'm trying to get a handle on where exactly an mmapplypolicy that's doing a premigrate is in its progress. I've already determined that 'ltfsee info jobs' will only report where in the current batch it is, but that still leaves me unable to tell the difference between [I] 2017-06-05 at 17:31:47.995 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 10000 files dispatched. and [I] 2017-06-06 at 02:44:48.236 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 225000 files dispatched. Is there any better way to figure out where it is than writing the cron job to launch it as mmapplypolicy | tee /tmp/something and then go scraping the messages? (And yes, I know not all chunks of 1,000 files are created equal. Sometimes it's 1,000 C source files that total to less than a megabyte, other times it's 1,000 streaming video files that total to over a terabye - but even knowing it's 194,000 into 243,348 files is better than what I have now...) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jun 6 22:20:31 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 6 Jun 2017 21:20:31 +0000 Subject: [gpfsug-discuss] mmapplypolicy and ltfsee - identifying progress... In-Reply-To: <25944.1496781061@turing-police.cc.vt.edu> References: <25944.1496781061@turing-police.cc.vt.edu> Message-ID: <5987990A-39F6-47A5-981D-A34A3054E4D8@vanderbilt.edu> Hi Valdis, I?m not sure this is ?better?, but what I typically do is have mmapplypolicy running from a shell script launched by a cron job and redirecting output to a file in /tmp. Once the mmapplypolicy finishes the SysAdmin?s get the tmp file e-mailed to them and then it gets deleted. Of course, while the mmapplypolicy is running you can ?tail -f /tmp/mmapplypolicy.log? or grep it or whatever. HTHAL? Kevin On Jun 6, 2017, at 3:31 PM, valdis.kletnieks at vt.edu wrote: So I'm trying to get a handle on where exactly an mmapplypolicy that's doing a premigrate is in its progress. I've already determined that 'ltfsee info jobs' will only report where in the current batch it is, but that still leaves me unable to tell the difference between [I] 2017-06-05 at 17:31:47.995 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 10000 files dispatched. and [I] 2017-06-06 at 02:44:48.236 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 225000 files dispatched. Is there any better way to figure out where it is than writing the cron job to launch it as mmapplypolicy | tee /tmp/something and then go scraping the messages? (And yes, I know not all chunks of 1,000 files are created equal. Sometimes it's 1,000 C source files that total to less than a megabyte, other times it's 1,000 streaming video files that total to over a terabye - but even knowing it's 194,000 into 243,348 files is better than what I have now...) ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From TROPPENS at de.ibm.com Wed Jun 7 09:30:17 2017 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Wed, 7 Jun 2017 08:30:17 +0000 Subject: [gpfsug-discuss] ISC 2017 - Agenda User Meeting Message-ID: An HTML attachment was scrubbed... URL: From jan.sundermann at kit.edu Wed Jun 7 15:04:28 2017 From: jan.sundermann at kit.edu (Sundermann, Jan Erik (SCC)) Date: Wed, 7 Jun 2017 14:04:28 +0000 Subject: [gpfsug-discuss] Upgrade with architecture change Message-ID: Hi, we are operating a small Spectrum Scale cluster with about 100 clients and 6 NSD servers. The cluster is FPO-enabled. For historical reasons the NSD servers are running on ppc64 while the clients are a mixture of ppc64le and x86_64 machines. Most machines are running Red Hat Enterprise Linux 7 but we also have few machines running AIX. At the moment we have installed Spectrum Scale version 4.1.1 but would like to do an upgrade to 4.2.3. In the course of the upgrade we would like to change the architecture of all NSD servers and reinstall them with ppc64le instead of ppc64. From what I?ve learned so far it should be possible to upgrade directly from 4.1.1 to 4.2.3. Before doing the upgrade we would like to ask for some advice on the best strategy. For the NSD servers, one by one, we are thinking about doing the following: 1) Disable auto recovery 2) Unmount GPFS file system 3) Suspend disks 4) Shutdown gpfs 5) Reboot and reinstall with changed architecture ppc64le 6) Install gpfs 4.2.3 7) Recover cluster config using mmsdrrestore 8) Resume and start disks 9) Reenable auto recovery Can GPFS handle the change of the NSD server?s architecture and would it be fine to operate a mixture of different architectures for the NSD servers? Thanks, Jan Erik From tarak.patel at canada.ca Wed Jun 7 16:42:45 2017 From: tarak.patel at canada.ca (Patel, Tarak (SSC/SPC)) Date: Wed, 7 Jun 2017 15:42:45 +0000 Subject: [gpfsug-discuss] Remote cluster gpfs communication on IP different then one for Daemon or Admin node name. Message-ID: <50fd0dc6cf47485c8728fc09b7ae0263@PEVDACDEXC009.birch.int.bell.ca> Hi all, We've been experiencing issues with remote cluster node expelling CES nodes causing remote filesystems to unmount. The issue is related gpfs communication using Ethernet IP rather than IP defined on IB which is used for Daemon node name and Admin node name. So remote cluster is aware of IPs that are not defined in GPFS configuration as Admin/Daemon node name. The CES nodes are configure to have IB as well as Ethernet (for client interactive and NFS access). We've double checked /etc/hosts and DNS and all looks to be in order since the CES IPoIB IP is present in /etc/hosts of remote cluster. I'm unsure where cluster manager for remote cluster is getting the Ethernet IP if there is no mention of it in GPFS configuration. The CES nodes were added later therefore they are not listed as Contact Nodes in 'mmremotecluster show' output. The CES nodes use IP defined on IB for GPFS configuration and we also have Ethernet which has the default route defined. In order to ensure that all IB communication passes via IPoIB, we've even defined a static route so that all GPFS communication will use IPoIB (since we are dealing with a different fabric). 'mmfsadm dump tscomm' reports multiple IPs for CES nodes which includes the Ethernet and also the IPoIB. I'm unsure if there is a way to drop some connections on GPFS (cluster wide) after stopping a specific CES node and ensure that only IB is listed. I realize that one option would be to define subnet parameter for remote cluster which will require a downtime (solution to be explored at later date). Hope that someone can explain how or why remote cluster is picking IPs not used in GPFS config for remote nodes and how to ensure those IPs are not used in future. Thank you, Tarak -- Tarak Patel Chef d'?quipe, Integration HPC, Solution de calcul E-Science Service partag? Canada / Gouvernment du Canada tarak.patel at canada.ca 1-514-421-7299 Team Lead, HPC Integration, E-Science Computing Solution Shared Services Canada, Government of Canada tarak.patel at canada.ca 1-514-421-7299 -------------- next part -------------- An HTML attachment was scrubbed... URL: From chekh at stanford.edu Wed Jun 7 23:12:56 2017 From: chekh at stanford.edu (Alex Chekholko) Date: Wed, 7 Jun 2017 15:12:56 -0700 Subject: [gpfsug-discuss] Upgrade with architecture change In-Reply-To: References: Message-ID: Hi Jan, I don't have hands-on experience with FPO or ppc64 but your procedure sounds OK to me. How do you currently handle just shutting down an NSD node for maintenance? I guess you'd have the same process except skip 5,6,7 How do you currently handle OS rebuild on NSD node? Maybe try that first without the architecture change. But I don't see why it would matter so long as you don't touch the GPFS disks. Regards, Alex On 06/07/2017 07:04 AM, Sundermann, Jan Erik (SCC) wrote: > Hi, > > we are operating a small Spectrum Scale cluster with about 100 clients and 6 NSD servers. The cluster is FPO-enabled. For historical reasons the NSD servers are running on ppc64 while the clients are a mixture of ppc64le and x86_64 machines. Most machines are running Red Hat Enterprise Linux 7 but we also have few machines running AIX. > > At the moment we have installed Spectrum Scale version 4.1.1 but would like to do an upgrade to 4.2.3. In the course of the upgrade we would like to change the architecture of all NSD servers and reinstall them with ppc64le instead of ppc64. > > From what I?ve learned so far it should be possible to upgrade directly from 4.1.1 to 4.2.3. Before doing the upgrade we would like to ask for some advice on the best strategy. > > For the NSD servers, one by one, we are thinking about doing the following: > > 1) Disable auto recovery > 2) Unmount GPFS file system > 3) Suspend disks > 4) Shutdown gpfs > 5) Reboot and reinstall with changed architecture ppc64le > 6) Install gpfs 4.2.3 > 7) Recover cluster config using mmsdrrestore > 8) Resume and start disks > 9) Reenable auto recovery > > Can GPFS handle the change of the NSD server?s architecture and would it be fine to operate a mixture of different architectures for the NSD servers? > > > Thanks, > Jan Erik From Philipp.Rehs at uni-duesseldorf.de Thu Jun 8 10:35:57 2017 From: Philipp.Rehs at uni-duesseldorf.de (Philipp Helo Rehs) Date: Thu, 8 Jun 2017 11:35:57 +0200 Subject: [gpfsug-discuss] GPFS for aarch64? Message-ID: <5848d2c0-d526-3d81-a469-6b7a10b9bf3a@uni-duesseldorf.de> Hello, we got a Cavium ThunderX-based Server and would like to use GPFS on it. Are the any package for gpfs on aarch64? Kind regards Philipp Rehs --------------------------- Zentrum f?r Informations- und Medientechnologie Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern Heinrich-Heine-Universit?t D?sseldorf Universit?tsstr. 1 Raum 25.41.00.51 40225 D?sseldorf / Germany Tel: +49-211-81-15557 From abeattie at au1.ibm.com Thu Jun 8 10:45:38 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Thu, 8 Jun 2017 09:45:38 +0000 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: <5848d2c0-d526-3d81-a469-6b7a10b9bf3a@uni-duesseldorf.de> References: <5848d2c0-d526-3d81-a469-6b7a10b9bf3a@uni-duesseldorf.de> Message-ID: An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Thu Jun 8 10:54:15 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 8 Jun 2017 09:54:15 +0000 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: Message-ID: And Linux on Z/VM If interested feel free to open a RFE -- Cheers > On 8 Jun 2017, at 12.46, Andrew Beattie wrote: > > Philipp, > > Not to my knowledge, > > AIX > Linux on x86 ( RHEL / SUSE / Ubuntu) > Linux on Power (RHEL / SUSE) > WIndows > > are the current supported platforms > Andrew Beattie > Software Defined Storage - IT Specialist > Phone: 614-2133-7927 > E-mail: abeattie at au1.ibm.com > > > ----- Original message ----- > From: Philipp Helo Rehs > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [gpfsug-discuss] GPFS for aarch64? > Date: Thu, Jun 8, 2017 7:36 PM > > Hello, > > we got a Cavium ThunderX-based Server and would like to use GPFS on it. > > Are the any package for gpfs on aarch64? > > > Kind regards > > Philipp Rehs > > --------------------------- > > Zentrum f?r Informations- und Medientechnologie > Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern > > Heinrich-Heine-Universit?t D?sseldorf > Universit?tsstr. 1 > Raum 25.41.00.51 > 40225 D?sseldorf / Germany > Tel: +49-211-81-15557 > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From Philipp.Rehs at uni-duesseldorf.de Thu Jun 8 11:40:23 2017 From: Philipp.Rehs at uni-duesseldorf.de (Philipp Helo Rehs) Date: Thu, 8 Jun 2017 12:40:23 +0200 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: References: Message-ID: <9f47c897-74ff-9473-2ab3-343e4ce69d15@uni-duesseldorf.de> Thanks for the Information. I created an RFE: https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=106218 Kind regards, Philipp Rehs > Message: 6 > Date: Thu, 8 Jun 2017 09:54:15 +0000 > From: "Luis Bolinches" > To: "gpfsug main discussion list" > Subject: Re: [gpfsug-discuss] GPFS for aarch64? > Message-ID: > > > Content-Type: text/plain; charset="utf-8" > > And Linux on Z/VM > > If interested feel free to open a RFE > > -- > Cheers > >> On 8 Jun 2017, at 12.46, Andrew Beattie wrote: >> >> Philipp, >> >> Not to my knowledge, >> >> AIX >> Linux on x86 ( RHEL / SUSE / Ubuntu) >> Linux on Power (RHEL / SUSE) >> WIndows >> >> are the current supported platforms >> Andrew Beattie >> Software Defined Storage - IT Specialist >> Phone: 614-2133-7927 >> E-mail: abeattie at au1.ibm.com >> >> >> ----- Original message ----- >> From: Philipp Helo Rehs >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: >> Subject: [gpfsug-discuss] GPFS for aarch64? >> Date: Thu, Jun 8, 2017 7:36 PM >> >> Hello, >> >> we got a Cavium ThunderX-based Server and would like to use GPFS on it. >> >> Are the any package for gpfs on aarch64? >> >> >> Kind regards >> >> Philipp Rehs >> >> --------------------------- >> >> Zentrum f?r Informations- und Medientechnologie >> Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern >> >> Heinrich-Heine-Universit?t D?sseldorf >> Universit?tsstr. 1 >> Raum 25.41.00.51 >> 40225 D?sseldorf / Germany >> Tel: +49-211-81-15557 >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 65, Issue 17 > ********************************************** > From daniel.kidger at uk.ibm.com Thu Jun 8 11:54:04 2017 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 8 Jun 2017 10:54:04 +0000 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: Message-ID: I often hear requests for Spectrum Scale on ARM. It is always for clients. In general people are happy to have their NSD servers, etc. on x86 or POWER. It is also an anomaly that for a HPC cluster, IBM supports LSF on ARM v7/v8 but not Spectrum Scale on ARM. Daniel Daniel Kidger Technical Sales Specialist, IBM UK IBM Spectrum Storage Software daniel.kidger at uk.ibm.com +44 (0)7818 522266 > On 8 Jun 2017, at 10:54, Luis Bolinches wrote: > > And Linux on Z/VM > > If interested feel free to open a RFE > > -- > Cheers > >> On 8 Jun 2017, at 12.46, Andrew Beattie wrote: >> >> Philipp, >> >> Not to my knowledge, >> >> AIX >> Linux on x86 ( RHEL / SUSE / Ubuntu) >> Linux on Power (RHEL / SUSE) >> WIndows >> >> are the current supported platforms >> Andrew Beattie >> Software Defined Storage - IT Specialist >> Phone: 614-2133-7927 >> E-mail: abeattie at au1.ibm.com >> >> >> ----- Original message ----- >> From: Philipp Helo Rehs >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: >> Subject: [gpfsug-discuss] GPFS for aarch64? >> Date: Thu, Jun 8, 2017 7:36 PM >> >> Hello, >> >> we got a Cavium ThunderX-based Server and would like to use GPFS on it. >> >> Are the any package for gpfs on aarch64? >> >> >> Kind regards >> >> Philipp Rehs >> >> --------------------------- >> >> Zentrum f?r Informations- und Medientechnologie >> Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern >> >> Heinrich-Heine-Universit?t D?sseldorf >> Universit?tsstr. 1 >> Raum 25.41.00.51 >> 40225 D?sseldorf / Germany >> Tel: +49-211-81-15557 >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From duersch at us.ibm.com Thu Jun 8 15:09:03 2017 From: duersch at us.ibm.com (Steve Duersch) Date: Thu, 8 Jun 2017 10:09:03 -0400 Subject: [gpfsug-discuss] Upgrade with architecture change In-Reply-To: References: Message-ID: We have not tested such a procedure. The only route that we have done is a complete mmdelnode/mmaddnode scenario. This would mean an mmdeldisk. It would be more time consuming since data has to move. Operating in a mixed architecture environment is not a problem. We have tested and support that. Steve Duersch Spectrum Scale 845-433-7902 IBM Poughkeepsie, New York > > Message: 1 > Date: Wed, 7 Jun 2017 14:04:28 +0000 > From: "Sundermann, Jan Erik (SCC)" > To: "gpfsug-discuss at spectrumscale.org" > > Subject: [gpfsug-discuss] Upgrade with architecture change > Message-ID: > Content-Type: text/plain; charset="utf-8" > > Hi, > > we are operating a small Spectrum Scale cluster with about 100 > clients and 6 NSD servers. The cluster is FPO-enabled. For > historical reasons the NSD servers are running on ppc64 while the > clients are a mixture of ppc64le and x86_64 machines. Most machines > are running Red Hat Enterprise Linux 7 but we also have few machines > running AIX. > > At the moment we have installed Spectrum Scale version 4.1.1 but > would like to do an upgrade to 4.2.3. In the course of the upgrade > we would like to change the architecture of all NSD servers and > reinstall them with ppc64le instead of ppc64. > > From what I?ve learned so far it should be possible to upgrade > directly from 4.1.1 to 4.2.3. Before doing the upgrade we would like > to ask for some advice on the best strategy. > > For the NSD servers, one by one, we are thinking about doing the following: > > 1) Disable auto recovery > 2) Unmount GPFS file system > 3) Suspend disks > 4) Shutdown gpfs > 5) Reboot and reinstall with changed architecture ppc64le > 6) Install gpfs 4.2.3 > 7) Recover cluster config using mmsdrrestore > 8) Resume and start disks > 9) Reenable auto recovery > > Can GPFS handle the change of the NSD server?s architecture and > would it be fine to operate a mixture of different architectures for > the NSD servers? > > > Thanks, > Jan Erik > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Thu Jun 8 17:01:07 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 8 Jun 2017 12:01:07 -0400 Subject: [gpfsug-discuss] Upgrade with architecture change In-Reply-To: References: Message-ID: If you proceed carefully, it should not be necessary to mmdeldisk and mmadddisks. Although we may not have tested your exact scenario, GPFS does support fiber channel disks attached to multiple nodes. So the same disk can be attached to multiple GPFS nodes - and those nodes can be running different OSes and different GPFS versions. (That's something we do actually test!) Since GPFS can handle that with several nodes simultaneously active -- it can also handle the case when nodes come and go... Or in your case are killed and then reborn with new software... The key is to be careful... You want to unmount the file system and not re-mount until all of the disks become available again via one or more (NSD) nodes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Thu Jun 8 22:34:15 2017 From: Mark.Bush at siriuscom.com (Mark Bush) Date: Thu, 8 Jun 2017 21:34:15 +0000 Subject: [gpfsug-discuss] LROC/HAWC for CES nodes? Message-ID: <288751C9-7CB6-48E6-968E-938A4E56E786@siriuscom.com> I?m looking to improve performance of the SMB stack. My workload unfortunately has smallish files but in total it will still be large amount. I?m wondering if LROC/HAWC would be one way to speed things up. Is there a precedent for using this with protocol nodes in a cluster? Anyone else thinking/doing this? Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From TROPPENS at de.ibm.com Fri Jun 9 08:44:57 2017 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Fri, 9 Jun 2017 09:44:57 +0200 Subject: [gpfsug-discuss] ISC 2017 - Agenda User Meeting In-Reply-To: References: Message-ID: There is an update of the agenda for the User Meeting at ISC. We have added a Pawsey Site Report by Chris Schlipalius. Monday June 19, 2016 - 12:00-14:30 - Conference Room Konstant 12:00-12:10 ?[10 min] ?Opening 12:10-12:25 ?[15 min] ?Spectrum Scale Support for Docker - Olaf Weiser (IBM) 12:25-13:05 ?[40 min] ?IBM Spectrum LSF family update - Bill McMillan (IBM) 13:05-13:25 ?[20 min] ?Driving Operational Efficiencies with the IBM Spectrum LSF & Ellexus Mistral - Dr. Rosemary Francis (Ellexus) 13:25-13:40 [15 min] Pawsey Site Report - Chris Schlipalius (Pawsey) 13:40-13:55 ?[15 min] ?IBM Elastic Storage Server (ESS) Update - John Sing (IBM) 13:55-14:20 ?[25 min] ?IBM Spectrum Scale Enhancements for CORAL - Sven Oehme (IBM) 14:20-14:30 ?[10 min] ?Question & Answers -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Ulf Troppens" To: gpfsug-discuss at spectrumscale.org Cc: Fabienne Wegener Date: 07.06.2017 10:30 Subject: [gpfsug-discuss] ISC 2017 - Agenda User Meeting Sent by: gpfsug-discuss-bounces at spectrumscale.org Greetings: IBM is happy to announce the agenda for the joint IBM Spectrum Scale and IBM Spectrum LSF User Meeting at ISC. As with other user meetings, the agenda includes user stories, updates on IBM Spectrum Scale and IBM Spectrum LSF, and access to IBM experts and your peers. Please join us! To attend, please email Fabienne.Wegener at de.ibm.com so we can have an accurate count of attendees. Monday June 17, 2016 - 12:00-14:30 - Conference Room Konstant 12:00-12:10 [10 min] Opening 12:10-12:30 [20 min] Spectrum Scale Support for Docker - Olaf Weiser (IBM) 12:30-13:10 [40 min] IBM Spectrum LSF family update - Bill McMillan (IBM) 13:10-13:30 [20 min] Driving Operational Efficiencies with the IBM Spectrum LSF & Ellexus Mistral - Dr. Rosemary Francis (Ellexus) 13:30-13:50 [20 min] IBM Elastic Storage Server (ESS) Update - John Sing (IBM) 13:50-14:20 [30 min] IBM Spectrum Scale Enhancements for CORAL - Sven Oehme (IBM) 14:20-14:30 [10 min] Question & Answers Looking forward to seeing you there! -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Fri Jun 9 09:38:01 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 9 Jun 2017 08:38:01 +0000 Subject: [gpfsug-discuss] LROC/HAWC for CES nodes? In-Reply-To: <288751C9-7CB6-48E6-968E-938A4E56E786@siriuscom.com> References: <288751C9-7CB6-48E6-968E-938A4E56E786@siriuscom.com> Message-ID: I?m wary of spending a lot of money on LROC devices when I don?t know what return I will get.. that said I think the main bottleneck for any SMB installation is samba itself, not the disks, so I remain largely unconvinced that LROC will help much. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mark Bush Sent: 08 June 2017 22:34 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] LROC/HAWC for CES nodes? I?m looking to improve performance of the SMB stack. My workload unfortunately has smallish files but in total it will still be large amount. I?m wondering if LROC/HAWC would be one way to speed things up. Is there a precedent for using this with protocol nodes in a cluster? Anyone else thinking/doing this? Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Fri Jun 9 10:31:45 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 9 Jun 2017 09:31:45 +0000 Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS In-Reply-To: References: Message-ID: Thanks Mark, didn?t mean to wait so long to reply. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Marc A Kaplan Sent: 02 June 2017 17:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] TSM/SP compatibility with GPFS Upgrading from GPFS 4.2.x to GPFS 4.2.y should not "break" TSM. If it does, someone goofed, that would be a bug. (My opinion) Think of it this way. TSM is an application that uses the OS and the FileSystem(s). TSM can't verify it will work with all future versions of OS and Filesystems, and the releases can't be in lock step. Having said that, 4.2.3 has been "out" for a while, so if there were a TSM incompatibility, someone would have likely hit it or will before July... Trust but verify... From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 06/02/2017 11:51 AM Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi all, Where should I start looking for a compatibility matrix between TSM and GPFS? Specifically, we are currently running TSM 7.1.6-2 and GPFS 4.2.1-2 with the intent to upgrade to GPFS 4.2.3-latest in early July. I?ve spent 30 minutes looking over various documents and the best I can find is this: http://www-01.ibm.com/support/docview.wss?uid=swg21248771 ..which talks about TSM in a Space Management context and would suggest that we need to upgrade to Spectrum Protect i.e. 8.1 and that GPFS 4.2.2.x is the maximum supported version? Cheers Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.tower at outlook.com Sat Jun 10 11:55:54 2017 From: frank.tower at outlook.com (Frank Tower) Date: Sat, 10 Jun 2017 10:55:54 +0000 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found Message-ID: Hi everybody, I don't get why one of our compute node cannot start GPFS over IB. I have the following error: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). [I] VERBS RDMA parse verbsPorts mlx4_0/1 [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found [I] VERBS RDMA library libibverbs.so unloaded. [E] VERBS RDMA failed to start, no valid verbsPorts defined. I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. I have 2 infinibands card, both have an IP and working well. [root at rdx110 ~]# ibstat -l mlx4_0 mlx4_1 [root at rdx110 ~]# I tried configuration with both card, and no one work with GPFS. I also tried with mlx4_0/1, but same problem. Someone already have the issue ? Kind Regards, Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Sat Jun 10 13:05:04 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Sat, 10 Jun 2017 08:05:04 -0400 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found In-Reply-To: References: Message-ID: Out of curiosity could you send us the output of "ibv_devinfo -v"? -Aaron Sent from my iPhone > On Jun 10, 2017, at 06:55, Frank Tower wrote: > > Hi everybody, > > > I don't get why one of our compute node cannot start GPFS over IB. > > > I have the following error: > > > [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes > [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. > [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). > > [I] VERBS RDMA parse verbsPorts mlx4_0/1 > > [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found > > [I] VERBS RDMA library libibverbs.so unloaded. > > [E] VERBS RDMA failed to start, no valid verbsPorts defined. > > > > I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. > > > I have 2 infinibands card, both have an IP and working well. > > > [root at rdx110 ~]# ibstat -l > > mlx4_0 > > mlx4_1 > > [root at rdx110 ~]# > > I tried configuration with both card, and no one work with GPFS. > > > > I also tried with mlx4_0/1, but same problem. > > > > Someone already have the issue ? > > > Kind Regards, > > Frank > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Mon Jun 12 20:41:17 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Mon, 12 Jun 2017 15:41:17 -0400 Subject: [gpfsug-discuss] 'mmces address move' weirdness? Message-ID: <18719.1497296477@turing-police.cc.vt.edu> So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Mon Jun 12 21:01:44 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 12 Jun 2017 20:01:44 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: <18719.1497296477@turing-police.cc.vt.edu> References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of valdis.kletnieks at vt.edu Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Jun 12 21:06:09 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Mon, 12 Jun 2017 20:06:09 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkr at lbl.gov Mon Jun 12 21:17:08 2017 From: kkr at lbl.gov (Kristy Kallback-Rose) Date: Mon, 12 Jun 2017 16:17:08 -0400 Subject: [gpfsug-discuss] Meaning of API Stats Category Message-ID: Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? Category 1, GPFSFileSystemAPI: This metrics gives the following information for each file system (application view). For example: myMachine|GPFSFilesystemAPI|myCluster|myFilesystem|gpfs_fis_bytes_read . gpfs_fis_bytes_read Number of bytes read. gpfs_fis_bytes_written Number of bytes written. gpfs_fis_close_calls Number of close calls. gpfs_fis_disks Number of disks in the file system. gpfs_fis_inodes_written Number of inode updates to disk. gpfs_fis_open_calls Number of open calls. gpfs_fis_read_calls Number of read calls. gpfs_fis_readdir_calls Number of readdir calls. gpfs_fis_write_calls Number of write calls. Category 2, GPFSNodeAPI: This metrics gives the following information for a particular node from its application point of view. For example: myMachine|GPFSNodeAPI|gpfs_is_bytes_read . gpfs_is_bytes_read Number of bytes read. gpfs_is_bytes_written Number of bytes written. gpfs_is_close_calls Number of close calls. gpfs_is_inodes_written Number of inode updates to disk. gpfs_is_open_calls Number of open calls. gpfs_is_readDir_calls Number of readdir calls. gpfs_is_read_calls Number of read calls. gpfs_is_write_calls Number of write calls. Thanks, Kristy -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 12 21:42:47 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 12 Jun 2017 20:42:47 +0000 Subject: [gpfsug-discuss] Meaning of API Stats Category Message-ID: Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Mon Jun 12 22:01:36 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 12 Jun 2017 17:01:36 -0400 Subject: [gpfsug-discuss] Meaning of API Stats Category In-Reply-To: References: Message-ID: Hello Kristy, The GPFSFileSystemAPI and GPFSNodeAPI sensor metrics are from the point of view of "applications" in the sense that they provide stats about I/O requests made to files in GPFS file systems from user level applications using POSIX interfaces like open(), close(), read(), write(), etc. This is in contrast to similarly named sensors without the "API" suffix, like GPFSFilesystem and GPFSNode. Those sensors provide stats about I/O requests made by the GPFS code to NSDs (disks) making up GPFS file systems. The relationship between application I/O and disk I/O might or might not be obvious. Consider some examples. An application that starts sequentially reading a file might, at least initially, cause more disk I/O than expected because GPFS has decided to prefetch data. An application write() might not immediately cause a the writing of disk blocks due to the operation of the pagepool. Ultimately, application write()s might cause twice as much data written to disk due to the replication factor of the file system. Application I/O concerns itself with user data; disk I/O might have to occur to handle the user data and associated file system metadata (like inodes and indirect blocks). The difference between GPFSFileSystemAPI and GPFSNodeAPI: GPFSFileSystemAPI reports stats for application I/O per filesystem per node; GPFSNodeAPI reports application I/O stats per node. Similarly, GPFSFilesystem reports stats for disk I/O per filesystem per node; GPFSNode reports disk I/O stats per node. I hope this helps. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/12/2017 04:43 PM Subject: Re: [gpfsug-discuss] Meaning of API Stats Category Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 12 23:50:44 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 12 Jun 2017 22:50:44 +0000 Subject: [gpfsug-discuss] Meaning of API Stats Category Message-ID: <163FC574-4191-4C20-A4C7-E66DB1868BF3@nuance.com> Can you tell me how LROC plays into this? I?m trying to understand if the difference between gpfs_ns_bytes_read and gpfs_is_bytes_read on a cluster-wide basis reflects the amount of data that is recalled from pagepool+LROC (assuming the majority of the nodes have LROC. Any insight on LROC stats would helpful as well. [cid:image001.png at 01D2E3A4.63CEE1D0] Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of IBM Spectrum Scale Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 4:01 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Meaning of API Stats Category Hello Kristy, The GPFSFileSystemAPI and GPFSNodeAPI sensor metrics are from the point of view of "applications" in the sense that they provide stats about I/O requests made to files in GPFS file systems from user level applications using POSIX interfaces like open(), close(), read(), write(), etc. This is in contrast to similarly named sensors without the "API" suffix, like GPFSFilesystem and GPFSNode. Those sensors provide stats about I/O requests made by the GPFS code to NSDs (disks) making up GPFS file systems. The relationship between application I/O and disk I/O might or might not be obvious. Consider some examples. An application that starts sequentially reading a file might, at least initially, cause more disk I/O than expected because GPFS has decided to prefetch data. An application write() might not immediately cause a the writing of disk blocks due to the operation of the pagepool. Ultimately, application write()s might cause twice as much data written to disk due to the replication factor of the file system. Application I/O concerns itself with user data; disk I/O might have to occur to handle the user data and associated file system metadata (like inodes and indirect blocks). The difference between GPFSFileSystemAPI and GPFSNodeAPI: GPFSFileSystemAPI reports stats for application I/O per filesystem per node; GPFSNodeAPI reports application I/O stats per node. Similarly, GPFSFilesystem reports stats for disk I/O per filesystem per node; GPFSNode reports disk I/O stats per node. I hope this helps. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/12/2017 04:43 PM Subject: Re: [gpfsug-discuss] Meaning of API Stats Category Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 124065 bytes Desc: image001.png URL: From valdis.kletnieks at vt.edu Tue Jun 13 05:21:26 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 13 Jun 2017 00:21:26 -0400 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: <15827.1497327686@turing-police.cc.vt.edu> On Mon, 12 Jun 2017 20:06:09 -0000, "Simon Thompson (IT Research Support)" said: > mmces node suspend -N > > Is what you want. This will move the address and stop it being assigned one, > otherwise the rebalance will occur. Yeah, I figured that part out. What I couldn't wrap my brain around was what the purpose of 'mmces address move' is if mmsysmon is going to just put it back... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From janfrode at tanso.net Tue Jun 13 05:42:21 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Tue, 13 Jun 2017 04:42:21 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: <15827.1497327686@turing-police.cc.vt.edu> References: <18719.1497296477@turing-police.cc.vt.edu> <15827.1497327686@turing-police.cc.vt.edu> Message-ID: Switch to node affinity policy, and it will stick to where you move it. "mmces address policy node-affinity". -jf tir. 13. jun. 2017 kl. 06.21 skrev : > On Mon, 12 Jun 2017 20:06:09 -0000, "Simon Thompson (IT Research Support)" > said: > > > mmces node suspend -N > > > > Is what you want. This will move the address and stop it being assigned > one, > > otherwise the rebalance will occur. > > Yeah, I figured that part out. What I couldn't wrap my brain around was > what the purpose of 'mmces address move' is if mmsysmon is going to just > put it back... > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Jun 13 09:08:52 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 13 Jun 2017 08:08:52 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: Yes, suspending the node would do it, but in the case where you want to remove a node from service but keep it running for testing it's not ideal. I think you can set the IP address balancing policy to none which might do what we want. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Jun 13 09:12:13 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 13 Jun 2017 08:12:13 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> <15827.1497327686@turing-police.cc.vt.edu> Message-ID: Or this ? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: 13 June 2017 05:42 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Switch to node affinity policy, and it will stick to where you move it. "mmces address policy node-affinity". -jf tir. 13. jun. 2017 kl. 06.21 skrev >: On Mon, 12 Jun 2017 20:06:09 -0000, "Simon Thompson (IT Research Support)" said: > mmces node suspend -N > > Is what you want. This will move the address and stop it being assigned one, > otherwise the rebalance will occur. Yeah, I figured that part out. What I couldn't wrap my brain around was what the purpose of 'mmces address move' is if mmsysmon is going to just put it back... _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Tue Jun 13 09:28:25 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Tue, 13 Jun 2017 08:28:25 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: Suspending the node doesn't stop the services though, we've done a bunch of testing by connecting to the "real" IP on the box we wanted to test and that works fine. OK, so you end up connecting to shares like \\192.168.1.20\sharename, but its perfectly fine for testing purposes. In our experience, suspending the node has been fine for this as it moves the IP to a "working" node and keeps user service running whilst we test. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 13 June 2017 at 09:08 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Yes, suspending the node would do it, but in the case where you want to remove a node from service but keep it running for testing it?s not ideal. I think you can set the IP address balancing policy to none which might do what we want. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From:gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Jun 13 09:30:18 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 13 Jun 2017 08:30:18 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: Oh? Nice to know - thanks - will try that method next. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 13 June 2017 09:28 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Suspending the node doesn't stop the services though, we've done a bunch of testing by connecting to the "real" IP on the box we wanted to test and that works fine. OK, so you end up connecting to shares like \\192.168.1.20\sharename, but its perfectly fine for testing purposes. In our experience, suspending the node has been fine for this as it moves the IP to a "working" node and keeps user service running whilst we test. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 13 June 2017 at 09:08 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Yes, suspending the node would do it, but in the case where you want to remove a node from service but keep it running for testing it's not ideal. I think you can set the IP address balancing policy to none which might do what we want. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From:gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Tue Jun 13 14:36:30 2017 From: john.hearns at asml.com (John Hearns) Date: Tue, 13 Jun 2017 13:36:30 +0000 Subject: [gpfsug-discuss] Infiniband Quality of Service settings? Message-ID: I am investigating setting up Quality of Service parameters on an Infiniband fabric. The specific goal is to reduce the bandwidth which certain servers can use, ie if there are untested or development codes running on these servers in our cluster then they cannot adversely affect production users. I hope I do not show too much of my ignorance here. Perhaps out of date, but I find that Lustre does have a facility for setting the port range and hence associating with an ULP in Infiniband http://www.spinics.net/lists/linux-rdma/msg02150.html https://community.mellanox.com/thread/3660 (There. I said the L word. Is a quick soaping to the mouth needed?) Can anyone comment what the Infiniband Service ID for GPFS traffic is please? If the answer is blindingly obvious and is displayed by a Bat signal in the clouds above every datacenter containing GPFS then I am suitably apologetic. If it is buried in a footnote in a Redbook then a bit less apologetic. If you are familiar with Appendix A of the IBTA Architecture Release then it is truly a joy. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Tue Jun 13 15:10:43 2017 From: john.hearns at asml.com (John Hearns) Date: Tue, 13 Jun 2017 14:10:43 +0000 Subject: [gpfsug-discuss] Infiniband Quality of Service settings? In-Reply-To: References: Message-ID: Having the bad manners to answer my own question: Example If you define a service level of 2 for GPFS in the InfiniBand subnet manager set verbsRdmaQpRtrSl to 2. mmchconfig verbsRdmaQpRtrSl=2 I guess though that I still need the service ID to set the Service Level in qos-policy.conf From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of John Hearns Sent: Tuesday, June 13, 2017 3:37 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Infiniband Quality of Service settings? I am investigating setting up Quality of Service parameters on an Infiniband fabric. The specific goal is to reduce the bandwidth which certain servers can use, ie if there are untested or development codes running on these servers in our cluster then they cannot adversely affect production users. I hope I do not show too much of my ignorance here. Perhaps out of date, but I find that Lustre does have a facility for setting the port range and hence associating with an ULP in Infiniband http://www.spinics.net/lists/linux-rdma/msg02150.html https://community.mellanox.com/thread/3660 (There. I said the L word. Is a quick soaping to the mouth needed?) Can anyone comment what the Infiniband Service ID for GPFS traffic is please? If the answer is blindingly obvious and is displayed by a Bat signal in the clouds above every datacenter containing GPFS then I am suitably apologetic. If it is buried in a footnote in a Redbook then a bit less apologetic. If you are familiar with Appendix A of the IBTA Architecture Release then it is truly a joy. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From SAnderson at convergeone.com Tue Jun 13 17:31:31 2017 From: SAnderson at convergeone.com (Shaun Anderson) Date: Tue, 13 Jun 2017 16:31:31 +0000 Subject: [gpfsug-discuss] Difference between mmcesnfscrexport and 'mmnfs export add' commands. Message-ID: <2990f67cded849e8b82a4c5d2ac50d5c@NACR502.nacr.com> ?I see both of these, but only the mmnfs command is documented. Is one a wrapper of the other? SHAUN ANDERSON STORAGE ARCHITECT O 208.577.2112 M 214.263.7014 NOTICE: This email message and any attachments here to may contain confidential information. Any unauthorized review, use, disclosure, or distribution of such information is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy the original message and all copies of it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Wed Jun 14 01:50:24 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 13 Jun 2017 20:50:24 -0400 Subject: [gpfsug-discuss] Meaning of API Stats Category In-Reply-To: <163FC574-4191-4C20-A4C7-E66DB1868BF3@nuance.com> References: <163FC574-4191-4C20-A4C7-E66DB1868BF3@nuance.com> Message-ID: Hello Bob, Right. Within some observation interval, bytes read by an application will be reflected in gpfs_is_bytes_read, regardless of how the byte values were obtained (by reading from "disk", fetching from pagepool, or fetching from LROC). gpfs_ns_bytes_read is only going to reflect bytes read from "disk" within that observation interval. "mmdiag --lroc" provides some LROC stats. There is also a GPFSLROC sensor; it does not appear to be documented at this point, so I hope I haven't spoken out of turn. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Cc: "scale at us.ibm.com" Date: 06/12/2017 06:50 PM Subject: Re: Meaning of API Stats Category Can you tell me how LROC plays into this? I?m trying to understand if the difference between gpfs_ns_bytes_read and gpfs_is_bytes_read on a cluster-wide basis reflects the amount of data that is recalled from pagepool+LROC (assuming the majority of the nodes have LROC. Any insight on LROC stats would helpful as well. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of IBM Spectrum Scale Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 4:01 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Meaning of API Stats Category Hello Kristy, The GPFSFileSystemAPI and GPFSNodeAPI sensor metrics are from the point of view of "applications" in the sense that they provide stats about I/O requests made to files in GPFS file systems from user level applications using POSIX interfaces like open(), close(), read(), write(), etc. This is in contrast to similarly named sensors without the "API" suffix, like GPFSFilesystem and GPFSNode. Those sensors provide stats about I/O requests made by the GPFS code to NSDs (disks) making up GPFS file systems. The relationship between application I/O and disk I/O might or might not be obvious. Consider some examples. An application that starts sequentially reading a file might, at least initially, cause more disk I/O than expected because GPFS has decided to prefetch data. An application write() might not immediately cause a the writing of disk blocks due to the operation of the pagepool. Ultimately, application write()s might cause twice as much data written to disk due to the replication factor of the file system. Application I/O concerns itself with user data; disk I/O might have to occur to handle the user data and associated file system metadata (like inodes and indirect blocks). The difference between GPFSFileSystemAPI and GPFSNodeAPI: GPFSFileSystemAPI reports stats for application I/O per filesystem per node; GPFSNodeAPI reports application I/O stats per node. Similarly, GPFSFilesystem reports stats for disk I/O per filesystem per node; GPFSNode reports disk I/O stats per node. I hope this helps. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/12/2017 04:43 PM Subject: Re: [gpfsug-discuss] Meaning of API Stats Category Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 124065 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jun 14 17:11:33 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 14 Jun 2017 16:11:33 +0000 Subject: [gpfsug-discuss] 4.2.3.x and sub-block size Message-ID: Hi All, Back at SC16 I was told that GPFS 4.2.3.x would remove the ?a sub-block is 1/32nd of the block size? restriction. However, I have installed GPFS 4.2.3.1 on my test cluster and in the man page for mmcrfs I still see: 2. The GPFS block size determines: * The minimum disk space allocation unit. The minimum amount of space that file data can occupy is a sub?block. A sub?block is 1/32 of the block size. So has the restriction been removed? If not, is there an update on which version of GPFS will remove it? If so, can the documentation be updated to reflect the change and how to take advantage of it? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kums at us.ibm.com Wed Jun 14 18:15:27 2017 From: kums at us.ibm.com (Kumaran Rajaram) Date: Wed, 14 Jun 2017 13:15:27 -0400 Subject: [gpfsug-discuss] 4.2.3.x and sub-block size In-Reply-To: References: Message-ID: Hi, >>Back at SC16 I was told that GPFS 4.2.3.x would remove the ?a sub-block is 1/32nd of the block size? restriction. However, I have installed GPFS 4.2.3.1 on my test cluster and in the man page for mmcrfs I still see: >>So has the restriction been removed? If not, is there an update on which version of GPFS will remove it? If so, can the documentation be updated to reflect the change and how to take advantage of it? Thanks? Based on the current plan, this ?a sub-block is 1/32nd of the block size? restriction will be removed in the upcoming GPFS version 4.2.4 (Please NOTE: Support for >32 subblocks per block may subject to be delayed based on internal qualification/validation efforts). Regards, -Kums From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/14/2017 12:12 PM Subject: [gpfsug-discuss] 4.2.3.x and sub-block size Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, Back at SC16 I was told that GPFS 4.2.3.x would remove the ?a sub-block is 1/32nd of the block size? restriction. However, I have installed GPFS 4.2.3.1 on my test cluster and in the man page for mmcrfs I still see: 2. The GPFS block size determines: * The minimum disk space allocation unit. The minimum amount of space that file data can occupy is a sub?block. A sub?block is 1/32 of the block size. So has the restriction been removed? If not, is there an update on which version of GPFS will remove it? If so, can the documentation be updated to reflect the change and how to take advantage of it? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Schlipalius at pawsey.org.au Thu Jun 15 03:30:27 2017 From: Chris.Schlipalius at pawsey.org.au (Chris Schlipalius) Date: Thu, 15 Jun 2017 10:30:27 +0800 Subject: [gpfsug-discuss] Perth Australia - Spectrum Scale User Group event in August 2017 announced - Pawsey Supercomputing Centre Message-ID: Hi please find the eventbrite link (text as http/s links are usually stripped). www.eventbrite.com/e/spectrum-scale-user-group-perth-australia-gpfsugaus-au gust-2017-tickets-35227460282 Please register and let me know if you are keen to present. I have a special group booking offer on accomodation for attendees, well below usually rack rack. I will announce this Usergroup meeting on spectrumscle.org shortly. This event is on the same week and at the same location as HPC Advisory Council also being held in Perth Australia. (Call for papers is now out - I can supply the HPC AC invite separately if you wish to email me directly). If you want to know more in person and you are at ISC2017 next week I will be at the Spectrum Scale Usergroup that Ulf announced or you can catch me on the Pawsey Supercomputing Centre booth. Regards, Chris Schlipalius Senior Storage Infrastructure Specialist/Team Leader Pawsey Supercomputing Centre 12 Burvill Court Kensington WA 6151 Australia Tel +61 8 6436 8815 Email chris.schlipalius at pawsey.org.au Web www.pawsey.org.au From Kevin.Buterbaugh at Vanderbilt.Edu Thu Jun 15 21:00:47 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Thu, 15 Jun 2017 20:00:47 +0000 Subject: [gpfsug-discuss] SAN problem ... multipathd ... mmunlinkfileset ... ??? Message-ID: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> Hi All, I?ve got some very weird problems going on here (and I do have a PMR open with IBM). On Monday I attempted to unlink a fileset, something that I?ve done many times with no issues. This time, however, it hung up the filesystem. I was able to clear things up by shutting down GPFS on the filesystem manager for that filesystem and restarting it. The very next morning we awoke to problems with GPFS. I noticed in my messages file on all my NSD servers I had messages like: Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Write Protect is off Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Asking for cache data failed Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Assuming drive cache: write through Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Attached SCSI disk Jun 12 22:03:32 nsd32 multipathd: sdab: add path (uevent) Jun 12 22:03:32 nsd32 multipathd: sdab: failed to get path uid Jun 12 22:03:32 nsd32 multipathd: uevent trigger error Jun 12 22:03:42 nsd32 kernel: rport-0:0-4: blocked FC remote port time out: removing target and saving binding Since we use an FC SAN and Linux multi-pathing I was expecting some sort of problem with the switches. Now on the switches I see messages like: [114][Thu Jun 15 19:02:05.411 UTC 2017][I][8600.0020][Port][Port: 9][SYNC_LOSS] [115][Thu Jun 15 19:03:49.988 UTC 2017][I][8600.001F][Port][Port: 9][SYNC_ACQ] Which (while not in this example) do correlate time-wise with the multi path messages on the servers. So it?s not a GPFS problem and I shouldn?t be bugging this list about this EXCEPT? These issues only started on Monday after I ran the mmunlinkfileset command. That?s right ? NO such errors prior to then. And literally NOTHING changed on Monday with my SAN environment (nothing had changed there for months actually). Nothing added to nor removed from the SAN. No changes until today when, in an attempt to solve this issue, I updated the switch firmware on all switches one at a time. I also yum updated to the latest RHEL 7 version of the multipathd packages. I?ve been Googling and haven?t found anything useful on those SYNC_LOSS messages on the QLogic SANbox 5800 switches. Anybody out there happen to have any knowledge of them and what could be causing them? Oh, I?m investigating this now ? but it?s not all ports that are throwing the errors. And the ports that are seem to be random and don?t have one specific type of hardware plugged in ? i.e. some ports have NSD servers plugged in, others have storage arrays. I understand that it makes no sense that mmunlinkfileset hanging would cause problems with my SAN ? but I also don?t believe in coincidences! I?m running GPFS 4.2.2.3. Any help / suggestions apprecaiated! ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Thu Jun 15 21:50:10 2017 From: ewahl at osc.edu (Edward Wahl) Date: Thu, 15 Jun 2017 16:50:10 -0400 Subject: [gpfsug-discuss] SAN problem ... multipathd ... mmunlinkfileset ... ??? In-Reply-To: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> References: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> Message-ID: <20170615165010.6241c6d3@osc.edu> On Thu, 15 Jun 2017 20:00:47 +0000 "Buterbaugh, Kevin L" wrote: > Hi All, > > I?ve got some very weird problems going on here (and I do have a PMR open > with IBM). On Monday I attempted to unlink a fileset, something that I?ve > done many times with no issues. This time, however, it hung up the > filesystem. I was able to clear things up by shutting down GPFS on the > filesystem manager for that filesystem and restarting it. > > The very next morning we awoke to problems with GPFS. I noticed in my > messages file on all my NSD servers I had messages like: > > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Write Protect is off > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Asking for cache data failed > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Assuming drive cache: write > through Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline > device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Attached SCSI disk > Jun 12 22:03:32 nsd32 multipathd: sdab: add path (uevent) > Jun 12 22:03:32 nsd32 multipathd: sdab: failed to get path uid > Jun 12 22:03:32 nsd32 multipathd: uevent trigger error > Jun 12 22:03:42 nsd32 kernel: rport-0:0-4: blocked FC remote port time out: > removing target and saving binding > > Since we use an FC SAN and Linux multi-pathing I was expecting some sort of > problem with the switches. Now on the switches I see messages like: > > [114][Thu Jun 15 19:02:05.411 UTC 2017][I][8600.0020][Port][Port: > 9][SYNC_LOSS] [115][Thu Jun 15 19:03:49.988 UTC > 2017][I][8600.001F][Port][Port: 9][SYNC_ACQ] > > Which (while not in this example) do correlate time-wise with the multi path > messages on the servers. So it?s not a GPFS problem and I shouldn?t be > bugging this list about this EXCEPT? > > These issues only started on Monday after I ran the mmunlinkfileset command. > That?s right ? NO such errors prior to then. And literally NOTHING changed > on Monday with my SAN environment (nothing had changed there for months > actually). Nothing added to nor removed from the SAN. No changes until > today when, in an attempt to solve this issue, I updated the switch firmware > on all switches one at a time. I also yum updated to the latest RHEL 7 > version of the multipathd packages. > > I?ve been Googling and haven?t found anything useful on those SYNC_LOSS > messages on the QLogic SANbox 5800 switches. Anybody out there happen to > have any knowledge of them and what could be causing them? Oh, I?m > investigating this now ? but it?s not all ports that are throwing the > errors. And the ports that are seem to be random and don?t have one specific > type of hardware plugged in ? i.e. some ports have NSD servers plugged in, > others have storage arrays. I have a half dozen of the Sanbox 5802 switches, but no GPFS devices going through them any longer. Used to though. We do see that exact same messages when the FC interface on a device goes bad (SFP, HCA, etc) or someone moving cables. This happens when the device cannot properly join the loop with it's login. I've NEVER seen them randomly though. Nor has this been a bad cable type error. I don't recall why, but I froze our Sanbox's at : V7.4.0.16.0 I'm sure I have notes on it somewhere. I've got one right now, in fact, with a bad ancient LTO4 drive. [8124][Thu Jun 15 12:46:00.190 EDT 2017][I][8600.001F][Port][Port: 4][SYNC_ACQ] [8125][Thu Jun 15 12:49:20.920 EDT 2017][I][8600.0020][Port][Port: 4][SYNC_LOSS] Sounds like the sanbox itself is having an issue perhaps? "Show alarm" clean on the sanbox? Array has a bad HCA? 'Show port 9' errors not crazy? All power supplies working? > I understand that it makes no sense that mmunlinkfileset hanging would cause > problems with my SAN ? but I also don?t believe in coincidences! > > I?m running GPFS 4.2.2.3. Any help / suggestions apprecaiated! This does seem like QUITE the coincidence. Increased traffic on the device triggered a failure? (The fear of all RAID users!) Multipath is working properly though? Sounds like mmlsdisk would have shown devices not in 'ready'. We mysteriously lost a MD disk during a recent downtime and it caused an MMFSCK to not run properly until we figured it out. 4.2.2.3 as well. MD Replication is NOT helpful in that case. Ed > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu - > (615)875-9633 > > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From Kevin.Buterbaugh at Vanderbilt.Edu Thu Jun 15 22:14:33 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Thu, 15 Jun 2017 21:14:33 +0000 Subject: [gpfsug-discuss] SAN problem ... multipathd ... mmunlinkfileset ... ??? In-Reply-To: <20170615165010.6241c6d3@osc.edu> References: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> <20170615165010.6241c6d3@osc.edu> Message-ID: <91D4DDD0-BF4C-4F73-B369-A91C032B0FCD@vanderbilt.edu> Hi Ed, others, I have spent the intervening time since sending my original e-mail taking the logs from the SAN switches and putting them into text files where they can be sorted and grep?d ? and something potentially interesting has come to light ? While there are a number of ports on all switches that have one or two SYNC_LOSS errors on them, on two of the switches port 9 has dozens of SYNC_LOSS errors (looking at the raw logs with other messages interspersed that wasn?t obvious). Turns out that one particular dual-controller storage array is plugged into those ports and - in a stroke of good luck which usually manages to avoid me - that particular storage array is no longer in use! It, and a few others still in use, are older and about to be life-cycled. Since it?s no longer in use, I have unplugged it from the SAN and am monitoring to see if my problems now go away. Yes, correlation is not causation. And sometimes coincidences do happen. I?ll monitor to see if this is one of those occasions. Thanks? Kevin > On Jun 15, 2017, at 3:50 PM, Edward Wahl wrote: > > On Thu, 15 Jun 2017 20:00:47 +0000 > "Buterbaugh, Kevin L" wrote: > >> Hi All, >> >> I?ve got some very weird problems going on here (and I do have a PMR open >> with IBM). On Monday I attempted to unlink a fileset, something that I?ve >> done many times with no issues. This time, however, it hung up the >> filesystem. I was able to clear things up by shutting down GPFS on the >> filesystem manager for that filesystem and restarting it. >> >> The very next morning we awoke to problems with GPFS. I noticed in my >> messages file on all my NSD servers I had messages like: >> >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Write Protect is off >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Asking for cache data failed >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Assuming drive cache: write >> through Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline >> device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Attached SCSI disk >> Jun 12 22:03:32 nsd32 multipathd: sdab: add path (uevent) >> Jun 12 22:03:32 nsd32 multipathd: sdab: failed to get path uid >> Jun 12 22:03:32 nsd32 multipathd: uevent trigger error >> Jun 12 22:03:42 nsd32 kernel: rport-0:0-4: blocked FC remote port time out: >> removing target and saving binding >> >> Since we use an FC SAN and Linux multi-pathing I was expecting some sort of >> problem with the switches. Now on the switches I see messages like: >> >> [114][Thu Jun 15 19:02:05.411 UTC 2017][I][8600.0020][Port][Port: >> 9][SYNC_LOSS] [115][Thu Jun 15 19:03:49.988 UTC >> 2017][I][8600.001F][Port][Port: 9][SYNC_ACQ] >> >> Which (while not in this example) do correlate time-wise with the multi path >> messages on the servers. So it?s not a GPFS problem and I shouldn?t be >> bugging this list about this EXCEPT? >> >> These issues only started on Monday after I ran the mmunlinkfileset command. >> That?s right ? NO such errors prior to then. And literally NOTHING changed >> on Monday with my SAN environment (nothing had changed there for months >> actually). Nothing added to nor removed from the SAN. No changes until >> today when, in an attempt to solve this issue, I updated the switch firmware >> on all switches one at a time. I also yum updated to the latest RHEL 7 >> version of the multipathd packages. >> >> I?ve been Googling and haven?t found anything useful on those SYNC_LOSS >> messages on the QLogic SANbox 5800 switches. Anybody out there happen to >> have any knowledge of them and what could be causing them? Oh, I?m >> investigating this now ? but it?s not all ports that are throwing the >> errors. And the ports that are seem to be random and don?t have one specific >> type of hardware plugged in ? i.e. some ports have NSD servers plugged in, >> others have storage arrays. > > I have a half dozen of the Sanbox 5802 switches, but no GPFS devices going > through them any longer. Used to though. We do see that exact same messages > when the FC interface on a device goes bad (SFP, HCA, etc) or someone moving > cables. This happens when the device cannot properly join the loop with it's > login. I've NEVER seen them randomly though. Nor has this been a bad cable type > error. I don't recall why, but I froze our Sanbox's at : V7.4.0.16.0 I'm sure > I have notes on it somewhere. > > I've got one right now, in fact, with a bad ancient LTO4 drive. > [8124][Thu Jun 15 12:46:00.190 EDT 2017][I][8600.001F][Port][Port: 4][SYNC_ACQ] > [8125][Thu Jun 15 12:49:20.920 EDT 2017][I][8600.0020][Port][Port: 4][SYNC_LOSS] > > > Sounds like the sanbox itself is having an issue perhaps? "Show alarm" clean on > the sanbox? Array has a bad HCA? 'Show port 9' errors not crazy? All power > supplies working? > > > >> I understand that it makes no sense that mmunlinkfileset hanging would cause >> problems with my SAN ? but I also don?t believe in coincidences! >> >> I?m running GPFS 4.2.2.3. Any help / suggestions apprecaiated! > > This does seem like QUITE the coincidence. Increased traffic on the > device triggered a failure? (The fear of all RAID users!) Multipath is working > properly though? Sounds like mmlsdisk would have shown devices not in 'ready'. > We mysteriously lost a MD disk during a recent downtime and it caused an MMFSCK > to not run properly until we figured it out. 4.2.2.3 as well. MD Replication > is NOT helpful in that case. > > Ed > > > >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and Education >> Kevin.Buterbaugh at vanderbilt.edu - >> (615)875-9633 >> >> >> > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 From frank.tower at outlook.com Sun Jun 18 05:57:57 2017 From: frank.tower at outlook.com (Frank Tower) Date: Sun, 18 Jun 2017 04:57:57 +0000 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found In-Reply-To: References: , Message-ID: Hi, You were right, ibv_devinfo -v doesn't return something if both card are connected. I didn't checked ibv_* tools, I supposed once IP stack and ibstat OK, the rest should work. I'm stupid ? Anyway, once I disconnect one card, ibv_devinfo show me input but with both cards, I don't have any input except "device not found". And what is weird here, it's that it work only when one card are connected, no matter the card (both are similar: model, firmware, revision, company)... Really strange, I will dig more about the issue. Stupid and bad workaround: connected a dual port Infiniband. But production system doesn't wait.. Thank for your help, Frank ________________________________ From: Aaron Knister Sent: Saturday, June 10, 2017 2:05 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Out of curiosity could you send us the output of "ibv_devinfo -v"? -Aaron Sent from my iPhone On Jun 10, 2017, at 06:55, Frank Tower > wrote: Hi everybody, I don't get why one of our compute node cannot start GPFS over IB. I have the following error: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). [I] VERBS RDMA parse verbsPorts mlx4_0/1 [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found [I] VERBS RDMA library libibverbs.so unloaded. [E] VERBS RDMA failed to start, no valid verbsPorts defined. I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. I have 2 infinibands card, both have an IP and working well. [root at rdx110 ~]# ibstat -l mlx4_0 mlx4_1 [root at rdx110 ~]# I tried configuration with both card, and no one work with GPFS. I also tried with mlx4_0/1, but same problem. Someone already have the issue ? Kind Regards, Frank _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.tower at outlook.com Sun Jun 18 06:06:30 2017 From: frank.tower at outlook.com (Frank Tower) Date: Sun, 18 Jun 2017 05:06:30 +0000 Subject: [gpfsug-discuss] Protocol node: active directory authentication Message-ID: Hi, We finally received protocols node following the recommendations some here provided and the help of the wiki. Now we would like to use kerberized NFS, we dig into spectrumscale documentations and wiki but we would like to know if anyone is using such configuration ? Do you also have any performance issue (vs NFSv4/NFSv3 with sec=sys) ? We will also use Microsoft Active Directory and we are willing to populate all our users with UID/GID, summer is coming, we will have some spare time ? Someone is using kerberized NFSv4 with Microsoft Active Directory ? Thank by advance for your feedback. Kind Regards, Frank. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcatana at gmail.com Sun Jun 18 16:30:55 2017 From: jcatana at gmail.com (Josh Catana) Date: Sun, 18 Jun 2017 11:30:55 -0400 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found In-Reply-To: References: Message-ID: Are any cards VPI that can do both eth and ib? I remember reading in documentation that that there is a bus order to having mixed media with mellanox cards. There is a module setting during init where you can set eth ib or auto detect. If the card is on auto it might be coming up eth and making the driver flake out because it's in the wrong order. Responding from my phone so I can't really look it up myself right now about what the proper order is, but maybe this might be some help troubleshooting. On Jun 18, 2017 12:58 AM, "Frank Tower" wrote: > Hi, > > > You were right, ibv_devinfo -v doesn't return something if both card are > connected. I didn't checked ibv_* tools, I supposed once IP stack and > ibstat OK, the rest should work. I'm stupid ? > > > Anyway, once I disconnect one card, ibv_devinfo show me input but with > both cards, I don't have any input except "device not found". > > And what is weird here, it's that it work only when one card are > connected, no matter the card (both are similar: model, firmware, revision, > company)... Really strange, I will dig more about the issue. > > > Stupid and bad workaround: connected a dual port Infiniband. But > production system doesn't wait.. > > > Thank for your help, > Frank > > ------------------------------ > *From:* Aaron Knister > *Sent:* Saturday, June 10, 2017 2:05 PM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found > > Out of curiosity could you send us the output of "ibv_devinfo -v"? > > -Aaron > > Sent from my iPhone > > On Jun 10, 2017, at 06:55, Frank Tower wrote: > > Hi everybody, > > > I don't get why one of our compute node cannot start GPFS over IB. > > > I have the following error: > > > [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no > verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes > > [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and > initialized. > > [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match > (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). > > [I] VERBS RDMA parse verbsPorts mlx4_0/1 > > [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device > mlx4_0 not found > > [I] VERBS RDMA library libibverbs.so unloaded. > > [E] VERBS RDMA failed to start, no valid verbsPorts defined. > > > > I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. > > > I have 2 infinibands card, both have an IP and working well. > > > [root at rdx110 ~]# ibstat -l > > mlx4_0 > > mlx4_1 > > [root at rdx110 ~]# > > > I tried configuration with both card, and no one work with GPFS. > > > I also tried with mlx4_0/1, but same problem. > > > Someone already have the issue ? > > > Kind Regards, > > Frank > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Sun Jun 18 18:53:28 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Sun, 18 Jun 2017 17:53:28 +0000 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found Message-ID: There used to be issues with the CX-3 cards and specific ports for if you wanted to use IB and Eth, but that went away in later firmwares, as did a whole load of bits with it being slow to detect media type, so see if you are running an up to date Mellanox firmware (assuming it's a VPI card). On CX-4 there is no auto detect media, but default is IB unless you changed it. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of jcatana at gmail.com [jcatana at gmail.com] Sent: 18 June 2017 16:30 To: gpfsug main discussion list Subject: ?spam? Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Are any cards VPI that can do both eth and ib? I remember reading in documentation that that there is a bus order to having mixed media with mellanox cards. There is a module setting during init where you can set eth ib or auto detect. If the card is on auto it might be coming up eth and making the driver flake out because it's in the wrong order. Responding from my phone so I can't really look it up myself right now about what the proper order is, but maybe this might be some help troubleshooting. On Jun 18, 2017 12:58 AM, "Frank Tower" > wrote: Hi, You were right, ibv_devinfo -v doesn't return something if both card are connected. I didn't checked ibv_* tools, I supposed once IP stack and ibstat OK, the rest should work. I'm stupid ? Anyway, once I disconnect one card, ibv_devinfo show me input but with both cards, I don't have any input except "device not found". And what is weird here, it's that it work only when one card are connected, no matter the card (both are similar: model, firmware, revision, company)... Really strange, I will dig more about the issue. Stupid and bad workaround: connected a dual port Infiniband. But production system doesn't wait.. Thank for your help, Frank ________________________________ From: Aaron Knister > Sent: Saturday, June 10, 2017 2:05 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Out of curiosity could you send us the output of "ibv_devinfo -v"? -Aaron Sent from my iPhone On Jun 10, 2017, at 06:55, Frank Tower > wrote: Hi everybody, I don't get why one of our compute node cannot start GPFS over IB. I have the following error: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). [I] VERBS RDMA parse verbsPorts mlx4_0/1 [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found [I] VERBS RDMA library libibverbs.so unloaded. [E] VERBS RDMA failed to start, no valid verbsPorts defined. I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. I have 2 infinibands card, both have an IP and working well. [root at rdx110 ~]# ibstat -l mlx4_0 mlx4_1 [root at rdx110 ~]# I tried configuration with both card, and no one work with GPFS. I also tried with mlx4_0/1, but same problem. Someone already have the issue ? Kind Regards, Frank _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From eric.wonderley at vt.edu Tue Jun 20 17:03:40 2017 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Tue, 20 Jun 2017 12:03:40 -0400 Subject: [gpfsug-discuss] gui related connection fail in gpfs logs Message-ID: These type messages repeat often in our logs: 017-06-20_09:25:13.676-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_09:25:24.292-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_10:00:25.935-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 Is there any way to tell if it is a misconfiguration or communications issue? -------------- next part -------------- An HTML attachment was scrubbed... URL: From NSCHULD at de.ibm.com Wed Jun 21 08:12:36 2017 From: NSCHULD at de.ibm.com (Norbert Schuld) Date: Wed, 21 Jun 2017 09:12:36 +0200 Subject: [gpfsug-discuss] gui related connection fail in gpfs logs In-Reply-To: References: Message-ID: This happens if a the mmhealth system of a node can not forward an event to the GUI - typically on some other node. Resons could be: - Gui is not running - Firewall on used port 80 for older versions of spectrum scale or 443 for newer Mit freundlichen Gr??en / Kind regards Norbert Schuld Dr. IBM Systems Group M925: IBM Spectrum Scale Norbert Software Development Schuld IBM Deutschland R&D GmbH Phone: +49-160 70 70 335 Am Weiher 24 E-Mail: nschuld at de.ibm.com 65451 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina Koederitz /Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "J. Eric Wonderley" To: gpfsug main discussion list Date: 20/06/2017 18:04 Subject: [gpfsug-discuss] gui related connection fail in gpfs logs Sent by: gpfsug-discuss-bounces at spectrumscale.org These type messages repeat often in our logs: 017-06-20_09:25:13.676-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_09:25:24.292-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_10:00:25.935-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 Is there any way to tell if it is a misconfiguration or communications issue? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 16690161.gif Type: image/gif Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From ewahl at osc.edu Thu Jun 22 20:37:12 2017 From: ewahl at osc.edu (Edward Wahl) Date: Thu, 22 Jun 2017 15:37:12 -0400 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: <20170622153712.052d312c@osc.edu> Is there a command to show existing node Address Policy? Or are we left with grep "affinity" on /var/mmfs/gen/mmsdrfs? Ed On Tue, 13 Jun 2017 08:30:18 +0000 "Sobey, Richard A" wrote: > Oh? Nice to know - thanks - will try that method next. > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson > (IT Research Support) Sent: 13 June 2017 09:28 To: gpfsug main discussion > list Subject: Re: [gpfsug-discuss] 'mmces > address move' weirdness? > > Suspending the node doesn't stop the services though, we've done a bunch of > testing by connecting to the "real" IP on the box we wanted to test and that > works fine. > > OK, so you end up connecting to shares like > \\192.168.1.20\sharename, but its perfectly > fine for testing purposes. > > In our experience, suspending the node has been fine for this as it moves the > IP to a "working" node and keeps user service running whilst we test. > > Simon > > From: > > > on behalf of "Sobey, Richard A" > > Reply-To: > "gpfsug-discuss at spectrumscale.org" > > > Date: Tuesday, 13 June 2017 at 09:08 To: > "gpfsug-discuss at spectrumscale.org" > > > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? > > Yes, suspending the node would do it, but in the case where you want to > remove a node from service but keep it running for testing it's not ideal. > > I think you can set the IP address balancing policy to none which might do > what we want. From: > gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson > (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion > list > > > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? > > mmces node suspend -N > > Is what you want. This will move the address and stop it being assigned one, > otherwise the rebalance will occur. I think you can change the way it > balances, but the default is to distribute. > > Simon > > From: > > > on behalf of "Sobey, Richard A" > > Reply-To: > "gpfsug-discuss at spectrumscale.org" > > > Date: Monday, 12 June 2017 at 21:01 To: > "gpfsug-discuss at spectrumscale.org" > > > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? > > > I think it's intended but I don't know why. The AUTH service became unhealthy > on one of our CES nodes (SMB only) and we moved its float address elsewhere. > CES decided to move it back again moments later despite the node not being > fit. > > > > Sorry that doesn't really help but at least you're not alone! > > ________________________________ > From:gpfsug-discuss-bounces at spectrumscale.org > > > on behalf of valdis.kletnieks at vt.edu > > Sent: 12 June 2017 > 20:41 To: > gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] 'mmces address move' weirdness? > > So here's our address setup: > > mmces address list > > Address Node Group Attribute > ------------------------------------------------------------------------- > 172.28.45.72 arproto1.ar.nis.isb.internal isb none > 172.28.45.73 arproto2.ar.nis.isb.internal isb none > 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none > 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none > > Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to > move the address over to its pair so I can look around without impacting > users. However, seems like something insists on moving it right back 60 > seconds later... > > Question 1: Is this expected behavior? > Question 2: If it is, what use is 'mmces address move' if it just gets > undone a few seconds later... > > (running on arproto2.ar.nis.vtc.internal): > > ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 > --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr > show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 > 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global > secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 > Mon Jun 12 15:34:42 EDT 2017 > Mon Jun 12 15:34:43 EDT 2017 > (skipped) > Mon Jun 12 15:35:44 EDT 2017 > Mon Jun 12 15:35:45 EDT 2017 > inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 > Mon Jun 12 15:35:46 EDT 2017 > inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 > Mon Jun 12 15:35:47 EDT 2017 > inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 > ^C -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From laurence at qsplace.co.uk Thu Jun 22 23:05:49 2017 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Thu, 22 Jun 2017 23:05:49 +0100 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: <20170622153712.052d312c@osc.edu> References: <18719.1497296477@turing-police.cc.vt.edu> <20170622153712.052d312c@osc.edu> Message-ID: <6B843B3B-4C07-459D-B905-10B16E3590A0@qsplace.co.uk> "mmlscluster --ces" will show the address distribution policy. -- Lauz On 22 June 2017 20:37:12 BST, Edward Wahl wrote: > >Is there a command to show existing node Address Policy? >Or are we left with grep "affinity" on /var/mmfs/gen/mmsdrfs? > >Ed > > >On Tue, 13 Jun 2017 08:30:18 +0000 >"Sobey, Richard A" wrote: > >> Oh? Nice to know - thanks - will try that method next. >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon >Thompson >> (IT Research Support) Sent: 13 June 2017 09:28 To: gpfsug main >discussion >> list Subject: Re: [gpfsug-discuss] >'mmces >> address move' weirdness? >> >> Suspending the node doesn't stop the services though, we've done a >bunch of >> testing by connecting to the "real" IP on the box we wanted to test >and that >> works fine. >> >> OK, so you end up connecting to shares like >> \\192.168.1.20\sharename, but its >perfectly >> fine for testing purposes. >> >> In our experience, suspending the node has been fine for this as it >moves the >> IP to a "working" node and keeps user service running whilst we test. >> >> Simon >> >> From: >> >> >> on behalf of "Sobey, Richard A" >> > Reply-To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Date: Tuesday, 13 June 2017 at 09:08 To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? >> >> Yes, suspending the node would do it, but in the case where you want >to >> remove a node from service but keep it running for testing it's not >ideal. >> >> I think you can set the IP address balancing policy to none which >might do >> what we want. From: >> >gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon >Thompson >> (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main >discussion >> list >> >> >> Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? >> >> mmces node suspend -N >> >> Is what you want. This will move the address and stop it being >assigned one, >> otherwise the rebalance will occur. I think you can change the way it >> balances, but the default is to distribute. >> >> Simon >> >> From: >> >> >> on behalf of "Sobey, Richard A" >> > Reply-To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Date: Monday, 12 June 2017 at 21:01 To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? >> >> >> I think it's intended but I don't know why. The AUTH service became >unhealthy >> on one of our CES nodes (SMB only) and we moved its float address >elsewhere. >> CES decided to move it back again moments later despite the node not >being >> fit. >> >> >> >> Sorry that doesn't really help but at least you're not alone! >> >> ________________________________ >> >From:gpfsug-discuss-bounces at spectrumscale.org >> >> >> on behalf of valdis.kletnieks at vt.edu >> > Sent: 12 >June 2017 >> 20:41 To: >> >gpfsug-discuss at spectrumscale.org >> Subject: [gpfsug-discuss] 'mmces address move' weirdness? >> >> So here's our address setup: >> >> mmces address list >> >> Address Node Group >Attribute >> >------------------------------------------------------------------------- >> 172.28.45.72 arproto1.ar.nis.isb.internal isb none >> 172.28.45.73 arproto2.ar.nis.isb.internal isb none >> 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none >> 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none >> >> Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so >I try to >> move the address over to its pair so I can look around without >impacting >> users. However, seems like something insists on moving it right back >60 >> seconds later... >> >> Question 1: Is this expected behavior? >> Question 2: If it is, what use is 'mmces address move' if it just >gets >> undone a few seconds later... >> >> (running on arproto2.ar.nis.vtc.internal): >> >> ## (date; ip addr show | grep '\.72';mmces address move --ces-ip >172.28.46.72 >> --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; >ip addr >> show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon >Jun 12 >> 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global >> secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 >EDT 2017 >> Mon Jun 12 15:34:42 EDT 2017 >> Mon Jun 12 15:34:43 EDT 2017 >> (skipped) >> Mon Jun 12 15:35:44 EDT 2017 >> Mon Jun 12 15:35:45 EDT 2017 >> inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary >bond1:0 >> Mon Jun 12 15:35:46 EDT 2017 >> inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary >bond1:0 >> Mon Jun 12 15:35:47 EDT 2017 >> inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary >bond1:0 >> ^C > > > >-- > >Ed Wahl >Ohio Supercomputer Center >614-292-9302 >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Fri Jun 23 09:13:38 2017 From: john.hearns at asml.com (John Hearns) Date: Fri, 23 Jun 2017 08:13:38 +0000 Subject: [gpfsug-discuss] IO prioritisation / throttling? Message-ID: I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO 'runs wild' it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Fri Jun 23 09:57:46 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Fri, 23 Jun 2017 04:57:46 -0400 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: References: Message-ID: <11158C89-C712-4C79-8B1D-CAA9D3D8641F@gmail.com> I unfortunately don't have an answer other than to perhaps check out this presentation from a recent users group meeting: http://files.gpfsug.org/presentations/2017/Manchester/05_Ellexus_SSUG_Manchester.pdf I've never used the product but it might be able to do what you're asking. Sent from my iPhone > On Jun 23, 2017, at 04:13, John Hearns wrote: > > I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. > We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. > The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down > the production storage. If anyone has a setup like this I would be interested in chatting with you. > Is it feasible to create filesets which have higher/lower priority than others? > > Thankyou for any insights or feedback > John Hearns > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Fri Jun 23 12:04:23 2017 From: john.hearns at asml.com (John Hearns) Date: Fri, 23 Jun 2017 11:04:23 +0000 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: <11158C89-C712-4C79-8B1D-CAA9D3D8641F@gmail.com> References: <11158C89-C712-4C79-8B1D-CAA9D3D8641F@gmail.com> Message-ID: Aaron, thankyou. I know Rosemary and that is a good company! From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Friday, June 23, 2017 10:58 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] IO prioritisation / throttling? I unfortunately don't have an answer other than to perhaps check out this presentation from a recent users group meeting: http://files.gpfsug.org/presentations/2017/Manchester/05_Ellexus_SSUG_Manchester.pdf I've never used the product but it might be able to do what you're asking. Sent from my iPhone On Jun 23, 2017, at 04:13, John Hearns > wrote: I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kums at us.ibm.com Fri Jun 23 14:36:34 2017 From: kums at us.ibm.com (Kumaran Rajaram) Date: Fri, 23 Jun 2017 09:36:34 -0400 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: References: Message-ID: Hi John, >>We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. >>The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down >>the production storage. You may use the Spectrum Scale Quality of Service for I/O "mmchqos" command (details in link below) to define IOPS limits for the "others" as well as the "maintenance" class for the Dev/Test file-system "pools" (for e.g., mmchqos tds_fs --enable pool=*,other=10000IOPS, maintenance=5000IOPS). This way, the Test and Dev file-system/storage-pools IOPS can be limited/controlled to specified IOPS , giving higher priority to the production GPFS file-system/storage (with production_fs pool=* other=unlimited,maintenance=unlimited - which is the default). https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_mmchqos.htm https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_qosio_describe.htm#qosio_describe My two cents. Regards, -Kums From: John Hearns To: gpfsug main discussion list Date: 06/23/2017 04:14 AM Subject: [gpfsug-discuss] IO prioritisation / throttling? Sent by: gpfsug-discuss-bounces at spectrumscale.org I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Fri Jun 23 15:23:19 2017 From: john.hearns at asml.com (John Hearns) Date: Fri, 23 Jun 2017 14:23:19 +0000 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: References: Message-ID: Thankyou to Kumaran and Aaaron for your help. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Kumaran Rajaram Sent: Friday, June 23, 2017 3:37 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] IO prioritisation / throttling? Hi John, >>We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. >>The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down >>the production storage. You may use the Spectrum Scale Quality of Service for I/O "mmchqos" command (details in link below) to define IOPS limits for the "others" as well as the "maintenance" class for the Dev/Test file-system "pools" (for e.g., mmchqos tds_fs --enable pool=*,other=10000IOPS, maintenance=5000IOPS). This way, the Test and Dev file-system/storage-pools IOPS can be limited/controlled to specified IOPS , giving higher priority to the production GPFS file-system/storage (with production_fs pool=* other=unlimited,maintenance=unlimited - which is the default). https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_mmchqos.htm https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_qosio_describe.htm#qosio_describe My two cents. Regards, -Kums From: John Hearns > To: gpfsug main discussion list > Date: 06/23/2017 04:14 AM Subject: [gpfsug-discuss] IO prioritisation / throttling? Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 23 17:06:51 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 23 Jun 2017 16:06:51 +0000 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy Message-ID: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> Hi All, I haven?t been able to find this explicitly documented, so I?m just wanting to confirm that the behavior that I?m expecting is what GPFS is going to do in this scenario? I have a filesystem with data replication set to two. I?m creating a capacity type pool for it right now which will be used to migrate old files to. I only want to use replication of one on the capacity pool. My policy file has two rules, one to move files with an atime > 30 days to the capacity pool, to which I?ve included ?REPLICATE(1)?. The other rule is to move files from the capacity pool back to the system pool if the atime < 14 days. Since data replication is set to two, I am thinking that I do not need to explicitly have a ?REPLICATE(2)? as part of that rule ? is that correct? I.e., I?m wanting to make sure that a file moved to the capacity pool which therefore has its? replication set to one doesn?t keep that same setting even if moved back out of the capacity pool. Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 23 17:58:23 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 23 Jun 2017 12:58:23 -0400 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy In-Reply-To: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> References: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> Message-ID: I believe that is correct. If not, let us know! To recap... when running mmapplypolicy with rules like: ... MIGRATE ... REPLICATE(x) ... will change the replication factor to x, for each file selected by this rule and chosen for execution. ... MIGRATE ... /* no REPLICATE keyword */ will not mess with the replication factor From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/23/2017 12:07 PM Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I haven?t been able to find this explicitly documented, so I?m just wanting to confirm that the behavior that I?m expecting is what GPFS is going to do in this scenario? I have a filesystem with data replication set to two. I?m creating a capacity type pool for it right now which will be used to migrate old files to. I only want to use replication of one on the capacity pool. My policy file has two rules, one to move files with an atime > 30 days to the capacity pool, to which I?ve included ?REPLICATE(1)?. The other rule is to move files from the capacity pool back to the system pool if the atime < 14 days. Since data replication is set to two, I am thinking that I do not need to explicitly have a ?REPLICATE(2)? as part of that rule ? is that correct? I.e., I?m wanting to make sure that a file moved to the capacity pool which therefore has its? replication set to one doesn?t keep that same setting even if moved back out of the capacity pool. Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Fri Jun 23 18:55:15 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Fri, 23 Jun 2017 13:55:15 -0400 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy In-Reply-To: References: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> Message-ID: <82717215-1C59-4B12-BAF6-09908044688D@ulmer.org> > On Jun 23, 2017, at 12:58 PM, Marc A Kaplan > wrote: > > I believe that is correct. If not, let us know! > > To recap... when running mmapplypolicy with rules like: > > ... MIGRATE ... REPLICATE(x) ... > > will change the replication factor to x, for each file selected by this rule and chosen for execution. > > ... MIGRATE ... /* no REPLICATE keyword */ > > will not mess with the replication factor > > I think I detect an impedance mismatch... By "not mess with the replication factor" do you mean that after the move: the file will have the default replication factor for the file system the file will retain a replication factor previously set on the file You told Kevin that he was correct and I think he meant the first one, but I read what you said as the second one. Liberty, -- Stephen -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 23 20:28:28 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 23 Jun 2017 15:28:28 -0400 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy In-Reply-To: <82717215-1C59-4B12-BAF6-09908044688D@ulmer.org> References: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> <82717215-1C59-4B12-BAF6-09908044688D@ulmer.org> Message-ID: Sorry for any confusion. MIGRATing a file does NOT change the replication factor, unless you explicitly use the keyword REPLICATE. The default replication factor, as set/displayed by mm[ch|ls]fs -r only applies at file creation time, unless overriden by a policy SET POOL ... REPLICATE(x) rule. From: Stephen Ulmer To: gpfsug main discussion list Date: 06/23/2017 01:55 PM Subject: Re: [gpfsug-discuss] Replication settings when running mmapplypolicy Sent by: gpfsug-discuss-bounces at spectrumscale.org On Jun 23, 2017, at 12:58 PM, Marc A Kaplan wrote: I believe that is correct. If not, let us know! To recap... when running mmapplypolicy with rules like: ... MIGRATE ... REPLICATE(x) ... will change the replication factor to x, for each file selected by this rule and chosen for execution. ... MIGRATE ... /* no REPLICATE keyword */ will not mess with the replication factor I think I detect an impedance mismatch... By "not mess with the replication factor" do you mean that after the move: the file will have the default replication factor for the file system the file will retain a replication factor previously set on the file You told Kevin that he was correct and I think he meant the first one, but I read what you said as the second one. Liberty, -- Stephen _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From ncapit at atos.net Mon Jun 26 08:49:28 2017 From: ncapit at atos.net (CAPIT, NICOLAS) Date: Mon, 26 Jun 2017 07:49:28 +0000 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads Message-ID: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Hello, I don't know if this behavior/bug was already reported on this ML, so in doubt. Context: - SpectrumScale 4.2.2-3 - client node with 64 cores - OS: RHEL7.3 When a MPI job with 64 processes is launched on the node with 64 cores then the FS freezed (only the output log file of the MPI job is put on the GPFS; so it may be related to the 64 processes writing in a same file???). strace -p 3105 # mmfsd pid stucked Process 3105 attached wait4(-1, # stucked at this point strace ls /gpfs stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC # stucked at this point I have no problem with the other nodes of 28 cores. The GPFS command mmgetstate is working and I am able to use mmshutdown to recover the node. If I put workerThreads=72 on the 64 core node then I am not able to reproduce the freeze and I get the right behavior. Is this a known bug with a number of cores > workerThreads? Best regards, [--] Nicolas Capit -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jun 27 00:57:57 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 26 Jun 2017 19:57:57 -0400 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Message-ID: <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> That's a fascinating bug. When the node is locked up what does "mmdiag --waiters" show from the node in question? I suspect there's more low-level diagnostic data that's helpful for the gurus at IBM but I'm just curious what the waiters look like. -Aaron On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > Hello, > > I don't know if this behavior/bug was already reported on this ML, so in > doubt. > > Context: > > - SpectrumScale 4.2.2-3 > - client node with 64 cores > - OS: RHEL7.3 > > When a MPI job with 64 processes is launched on the node with 64 cores > then the FS freezed (only the output log file of the MPI job is put on > the GPFS; so it may be related to the 64 processes writing in a same > file???). > > strace -p 3105 # mmfsd pid stucked > Process 3105 attached > wait4(-1, # stucked at this point > > strace ls /gpfs > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > # stucked at this point > > I have no problem with the other nodes of 28 cores. > The GPFS command mmgetstate is working and I am able to use mmshutdown > to recover the node. > > > If I put workerThreads=72 on the 64 core node then I am not able to > reproduce the freeze and I get the right behavior. > > Is this a known bug with a number of cores > workerThreads? > > Best regards, > -- > *Nicolas Capit* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From ncapit at atos.net Tue Jun 27 07:59:19 2017 From: ncapit at atos.net (CAPIT, NICOLAS) Date: Tue, 27 Jun 2017 06:59:19 +0000 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net>, <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> Message-ID: <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Hello, When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters"). In the GPFS log file "/var/mmfs/gen/mmfslog" there is nothing and nothing in the dmesg output or system log. The "mmgetstate" command says that the node is "active". The only thing is the freeze of the FS. Best regards, Nicolas Capit ________________________________________ De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de Aaron Knister [aaron.s.knister at nasa.gov] Envoy? : mardi 27 juin 2017 01:57 ? : gpfsug-discuss at spectrumscale.org Objet : Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads That's a fascinating bug. When the node is locked up what does "mmdiag --waiters" show from the node in question? I suspect there's more low-level diagnostic data that's helpful for the gurus at IBM but I'm just curious what the waiters look like. -Aaron On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > Hello, > > I don't know if this behavior/bug was already reported on this ML, so in > doubt. > > Context: > > - SpectrumScale 4.2.2-3 > - client node with 64 cores > - OS: RHEL7.3 > > When a MPI job with 64 processes is launched on the node with 64 cores > then the FS freezed (only the output log file of the MPI job is put on > the GPFS; so it may be related to the 64 processes writing in a same > file???). > > strace -p 3105 # mmfsd pid stucked > Process 3105 attached > wait4(-1, # stucked at this point > > strace ls /gpfs > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > # stucked at this point > > I have no problem with the other nodes of 28 cores. > The GPFS command mmgetstate is working and I am able to use mmshutdown > to recover the node. > > > If I put workerThreads=72 on the 64 core node then I am not able to > reproduce the freeze and I get the right behavior. > > Is this a known bug with a number of cores > workerThreads? > > Best regards, > -- > *Nicolas Capit* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From stuartb at 4gh.net Tue Jun 27 22:34:34 2017 From: stuartb at 4gh.net (Stuart Barkley) Date: Tue, 27 Jun 2017 17:34:34 -0400 (EDT) Subject: [gpfsug-discuss] express edition vs standard edition Message-ID: Does anyone know what controls whether GPFS (4.1.1) thinks it is Express Edition versus Standard Edition? While rebuilding an old cluster from scratch we are getting the message: mmcrfs: Storage pools are not available in the GPFS Express Edition. The Problem Determination Guide says to "Install the GPFS Standard Edition on all nodes in the cluster" which we think we have done. The cluster is just 3 servers and no clients at this point. We have verified that our purchased license is for Standard Version, but have not been able to figure out what controls the GPFS view of this. mmlslicense tells us that we have Express Edition installed. mmchlicense sets server vs client license information, but does not seem to be able to control the edition. Our normal install process is to install gpfs.base-4.1.1-0.x86_64.rpm first and then install gpfs.base-4.1.1-15.x86_64.update.rpm followed by the other needed 4.1.1-15 rpms. I thought maybe we had the wrong gpfs.base and we have re-downloaded Standard Edition RPM files from IBM in case we had the wrong version. However, reinstalling and recreating the cluster does not seem to have addressed this issue. We must be doing something stupid during our install, but I'm pretty sure we used only Standard Edition rpms for our latest attempt. Thanks, Stuart Barkley -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From knop at us.ibm.com Tue Jun 27 22:54:35 2017 From: knop at us.ibm.com (Felipe Knop) Date: Tue, 27 Jun 2017 17:54:35 -0400 Subject: [gpfsug-discuss] express edition vs standard edition In-Reply-To: References: Message-ID: Stuart, I believe you will need to install the gpfs.ext RPMs , otherwise the daemons and commands will think only the Express edition is installed. Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 From: Stuart Barkley To: gpfsug-discuss at spectrumscale.org Date: 06/27/2017 05:35 PM Subject: [gpfsug-discuss] express edition vs standard edition Sent by: gpfsug-discuss-bounces at spectrumscale.org Does anyone know what controls whether GPFS (4.1.1) thinks it is Express Edition versus Standard Edition? While rebuilding an old cluster from scratch we are getting the message: mmcrfs: Storage pools are not available in the GPFS Express Edition. The Problem Determination Guide says to "Install the GPFS Standard Edition on all nodes in the cluster" which we think we have done. The cluster is just 3 servers and no clients at this point. We have verified that our purchased license is for Standard Version, but have not been able to figure out what controls the GPFS view of this. mmlslicense tells us that we have Express Edition installed. mmchlicense sets server vs client license information, but does not seem to be able to control the edition. Our normal install process is to install gpfs.base-4.1.1-0.x86_64.rpm first and then install gpfs.base-4.1.1-15.x86_64.update.rpm followed by the other needed 4.1.1-15 rpms. I thought maybe we had the wrong gpfs.base and we have re-downloaded Standard Edition RPM files from IBM in case we had the wrong version. However, reinstalling and recreating the cluster does not seem to have addressed this issue. We must be doing something stupid during our install, but I'm pretty sure we used only Standard Edition rpms for our latest attempt. Thanks, Stuart Barkley -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuartb at 4gh.net Tue Jun 27 23:48:53 2017 From: stuartb at 4gh.net (Stuart Barkley) Date: Tue, 27 Jun 2017 18:48:53 -0400 (EDT) Subject: [gpfsug-discuss] express edition vs standard edition In-Reply-To: References: Message-ID: On Tue, 27 Jun 2017 at 17:54 -0000, Felipe Knop wrote: > I believe you will need to install the gpfs.ext RPMs , otherwise the > daemons and commands will think only the Express edition is > installed. Yes. This appears to have been my problem. Thanks, Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From scale at us.ibm.com Fri Jun 30 07:57:49 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 30 Jun 2017 14:57:49 +0800 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net>, <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Message-ID: I'm not aware this kind of defects, seems it should not. but lack of data, we don't know what happened. I suggest you can open a PMR for your issue. Thanks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "CAPIT, NICOLAS" To: gpfsug main discussion list Date: 06/27/2017 02:59 PM Subject: Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters"). In the GPFS log file "/var/mmfs/gen/mmfslog" there is nothing and nothing in the dmesg output or system log. The "mmgetstate" command says that the node is "active". The only thing is the freeze of the FS. Best regards, Nicolas Capit ________________________________________ De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de Aaron Knister [aaron.s.knister at nasa.gov] Envoy? : mardi 27 juin 2017 01:57 ? : gpfsug-discuss at spectrumscale.org Objet : Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads That's a fascinating bug. When the node is locked up what does "mmdiag --waiters" show from the node in question? I suspect there's more low-level diagnostic data that's helpful for the gurus at IBM but I'm just curious what the waiters look like. -Aaron On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > Hello, > > I don't know if this behavior/bug was already reported on this ML, so in > doubt. > > Context: > > - SpectrumScale 4.2.2-3 > - client node with 64 cores > - OS: RHEL7.3 > > When a MPI job with 64 processes is launched on the node with 64 cores > then the FS freezed (only the output log file of the MPI job is put on > the GPFS; so it may be related to the 64 processes writing in a same > file???). > > strace -p 3105 # mmfsd pid stucked > Process 3105 attached > wait4(-1, # stucked at this point > > strace ls /gpfs > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > # stucked at this point > > I have no problem with the other nodes of 28 cores. > The GPFS command mmgetstate is working and I am able to use mmshutdown > to recover the node. > > > If I put workerThreads=72 on the 64 core node then I am not able to > reproduce the freeze and I get the right behavior. > > Is this a known bug with a number of cores > workerThreads? > > Best regards, > -- > *Nicolas Capit* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From heiner.billich at psi.ch Fri Jun 30 11:07:10 2017 From: heiner.billich at psi.ch (Billich Heinrich Rainer (PSI)) Date: Fri, 30 Jun 2017 10:07:10 +0000 Subject: [gpfsug-discuss] AFM - how to update directories with deleted files during prefetch Message-ID: <76D2B41B-37A6-410B-9ECE-F5FA4C7FF1EE@psi.ch> Hello I have a short question about AFM prefetch and some more remarks regarding AFM and it?s use for data migration. I understand that many of you have done this for very large amounts of data and number of files. I would welcome an input, comments or remarks. Sorry if this is a bit too long for a mailing list. Short: How can I tell an AFM cache to update a directory when I do prefetch? I know about ?find .? or ?ls ?lsR? but this really is no option for us as it takes too long. Mostly I want to update the directories to make AFM cache aware of file deletions on home. On home I can use a policy run to find all directories which changed since the last update and pass them to prefetch on AFM cache. I know that I can find some workaround based on the directory list, like an ?ls ?lsa? just for those directories, but this doesn?t sound very efficient. And depending on cache effects and timeout settings it may work or not (o.k. ? it will work most time). We do regular file deletions and will accumulated millions of deleted files on cache over time if we don?t update the directories to make AFM cache aware of the deletion. Background: We will use AFM to migrate data on filesets to another cluster. We have to do this several times in the next few months, hence I want to get a reliable and easy to use procedure. The old system is home, the new system is a read-only AFM cache. We want to use ?mmafmctl prefetch? to move the data. Home will be in use while we run the migration. Once almost all data is moved we do a (short) break for a last sync and make the read-only AFM cache a ?normal? fileset. During the break I want to use policy runs and prefetch only and no time consuming ?ls ?lsr? or ?find .? I don?t want to use this metadata intensive posix operation during operation, either. More general: AFM can be used for data migration. But I don?t see how to use it efficiently. How to do incremental transfers, how to ensure that the we really have identical copies before we switch and how to keep the switch time short , i.e. the time when both old and new aren?t accessible for clients, Wish ? maybe an RFE? I can use policy runs to collect all changed items on home since the last update. I wish that I can pass this list to afm prefetch to do all updates on AFM cache, too. Same as backup tools use the list to do incremental backups. And a tool to create policy lists of home and cache and to compare the lists would be nice, too. As you do this during the break/switch it should be fast and reliable and leave no doubts. Kind regards, Heiner From vpuvvada at in.ibm.com Fri Jun 30 13:35:18 2017 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Fri, 30 Jun 2017 18:05:18 +0530 Subject: [gpfsug-discuss] AFM - how to update directories with deleted files during prefetch In-Reply-To: <76D2B41B-37A6-410B-9ECE-F5FA4C7FF1EE@psi.ch> References: <76D2B41B-37A6-410B-9ECE-F5FA4C7FF1EE@psi.ch> Message-ID: What is the version of GPFS ? >Mostly I want to update the directories to make AFM cache aware of file deletions on home. On home I can use a policy run to find all directories which changed since the last >update and pass them to prefetch on AFM cache. AFM prefetch has undocumented option to delete files from cache without doing lookup from home. It supports all types of list files. Find all deleted file from home and prefetch at cache to delete them. mmafmctl device prefetch -j fileset --list-file --delete >Wish ? maybe an RFE? >I can use policy runs to collect all changed items on home since the last update. I wish that I can pass this list to afm prefetch to do all updates on AFM cache, too. Same >as backup tools use the list to do incremental backups. This feature support was already added but undocumented today. This feature will be externalized in future releases. ~Venkat (vpuvvada at in.ibm.com) From: "Billich Heinrich Rainer (PSI)" To: "gpfsug-discuss at spectrumscale.org" Date: 06/30/2017 03:37 PM Subject: [gpfsug-discuss] AFM - how to update directories with deleted files during prefetch Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello I have a short question about AFM prefetch and some more remarks regarding AFM and it?s use for data migration. I understand that many of you have done this for very large amounts of data and number of files. I would welcome an input, comments or remarks. Sorry if this is a bit too long for a mailing list. Short: How can I tell an AFM cache to update a directory when I do prefetch? I know about ?find .? or ?ls ?lsR? but this really is no option for us as it takes too long. Mostly I want to update the directories to make AFM cache aware of file deletions on home. On home I can use a policy run to find all directories which changed since the last update and pass them to prefetch on AFM cache. I know that I can find some workaround based on the directory list, like an ?ls ?lsa? just for those directories, but this doesn?t sound very efficient. And depending on cache effects and timeout settings it may work or not (o.k. ? it will work most time). We do regular file deletions and will accumulated millions of deleted files on cache over time if we don?t update the directories to make AFM cache aware of the deletion. Background: We will use AFM to migrate data on filesets to another cluster. We have to do this several times in the next few months, hence I want to get a reliable and easy to use procedure. The old system is home, the new system is a read-only AFM cache. We want to use ?mmafmctl prefetch? to move the data. Home will be in use while we run the migration. Once almost all data is moved we do a (short) break for a last sync and make the read-only AFM cache a ?normal? fileset. During the break I want to use policy runs and prefetch only and no time consuming ?ls ?lsr? or ?find .? I don?t want to use this metadata intensive posix operation during operation, either. More general: AFM can be used for data migration. But I don?t see how to use it efficiently. How to do incremental transfers, how to ensure that the we really have identical copies before we switch and how to keep the switch time short , i.e. the time when both old and new aren?t accessible for clients, Wish ? maybe an RFE? I can use policy runs to collect all changed items on home since the last update. I wish that I can pass this list to afm prefetch to do all updates on AFM cache, too. Same as backup tools use the list to do incremental backups. And a tool to create policy lists of home and cache and to compare the lists would be nice, too. As you do this during the break/switch it should be fast and reliable and leave no doubts. Kind regards, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From hpc-luke at uconn.edu Fri Jun 30 16:20:27 2017 From: hpc-luke at uconn.edu (hpc-luke at uconn.edu) Date: Fri, 30 Jun 2017 11:20:27 -0400 Subject: [gpfsug-discuss] Mass UID migration suggestions Message-ID: <59566c3b.kqOSy9e2kg80XZ/k%hpc-luke@uconn.edu> Hello, We're trying to change most of our users uids, is there a clean way to migrate all of one users files with say `mmapplypolicy`? We have to change the owner of around 273539588 files, and my estimates for runtime are around 6 days. What we've been doing is indexing all of the files and splitting them up by owner which takes around an hour, and then we were locking the user out while we chown their files. I made it multi threaded as it weirdly gave a 10% speedup despite my expectation that multi threading access from a single node would not give any speedup. Generally I'm looking for advice on how to make the chowning faster. Would spreading the chowning processes over multiple nodes improve performance? Should I not stat the files before running lchown on them, since lchown checks the file before changing it? I saw mention of inodescan(), in an old gpfsug email, which speeds up disk read access, by not guaranteeing that the data is up to date. We have a maintenance day coming up where all users will be locked out, so the file handles(?) from GPFS's perspective will not be able to go stale. Is there a function with similar constraints to inodescan that I can use to speed up this process? Thank you for your time, Luke Storrs-HPC University of Connecticut From aaron.knister at gmail.com Fri Jun 30 16:47:40 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Fri, 30 Jun 2017 11:47:40 -0400 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Message-ID: Nicolas, By chance do you have a skylake or kabylake based CPU? Sent from my iPhone > On Jun 30, 2017, at 02:57, IBM Spectrum Scale wrote: > > I'm not aware this kind of defects, seems it should not. but lack of data, we don't know what happened. I suggest you can open a PMR for your issue. Thanks. > > Regards, The Spectrum Scale (GPFS) team > > ------------------------------------------------------------------------------------------------------------------ > If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. > > If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. > > The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. > > "CAPIT, NICOLAS" ---06/27/2017 02:59:59 PM---Hello, When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters") > > From: "CAPIT, NICOLAS" > To: gpfsug main discussion list > Date: 06/27/2017 02:59 PM > Subject: Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > Hello, > > When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters"). > In the GPFS log file "/var/mmfs/gen/mmfslog" there is nothing and nothing in the dmesg output or system log. > The "mmgetstate" command says that the node is "active". > The only thing is the freeze of the FS. > > Best regards, > Nicolas Capit > ________________________________________ > De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de Aaron Knister [aaron.s.knister at nasa.gov] > Envoy? : mardi 27 juin 2017 01:57 > ? : gpfsug-discuss at spectrumscale.org > Objet : Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads > > That's a fascinating bug. When the node is locked up what does "mmdiag > --waiters" show from the node in question? I suspect there's more > low-level diagnostic data that's helpful for the gurus at IBM but I'm > just curious what the waiters look like. > > -Aaron > > On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > > Hello, > > > > I don't know if this behavior/bug was already reported on this ML, so in > > doubt. > > > > Context: > > > > - SpectrumScale 4.2.2-3 > > - client node with 64 cores > > - OS: RHEL7.3 > > > > When a MPI job with 64 processes is launched on the node with 64 cores > > then the FS freezed (only the output log file of the MPI job is put on > > the GPFS; so it may be related to the 64 processes writing in a same > > file???). > > > > strace -p 3105 # mmfsd pid stucked > > Process 3105 attached > > wait4(-1, # stucked at this point > > > > strace ls /gpfs > > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > > # stucked at this point > > > > I have no problem with the other nodes of 28 cores. > > The GPFS command mmgetstate is working and I am able to use mmshutdown > > to recover the node. > > > > > > If I put workerThreads=72 on the 64 core node then I am not able to > > reproduce the freeze and I get the right behavior. > > > > Is this a known bug with a number of cores > workerThreads? > > > > Best regards, > > -- > > *Nicolas Capit* > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Jun 30 18:14:07 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 30 Jun 2017 13:14:07 -0400 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: <487469581.449569.1498832342497.JavaMail.webinst@w30112> References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: I'm curious to know why this doesn't affect GSS/ESS? Is it a feature of the additional check-summing done on those platforms? -------- Forwarded Message -------- Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) Date: Fri, 30 Jun 2017 14:19:02 +0000 From: IBM My Notifications To: aaron.s.knister at nasa.gov My Notifications for Storage - 30 Jun 2017 Dear Subscriber (aaron.s.knister at nasa.gov), Here are your updates from IBM My Notifications. Your support Notifications display in English by default. Machine translation based on your IBM profile language setting is added if you specify this option in My defaults within My Notifications. (Note: Not all languages are available at this time, and the English version always takes precedence over the machine translated version.) ------------------------------------------------------------------------------ 1. IBM Spectrum Scale - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error - URL: http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM Spectrum Scale versions where the NSD server is enabled to use RDMA for file IO and the storage used in your GPFS cluster accessed via NSD servers (not fully SAN accessible) includes anything other than IBM Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under these conditions, when the RDMA-enabled network adapter fails, the issue may result in undetected data corruption for file write or read operations. ------------------------------------------------------------------------------ Manage your My Notifications subscriptions, or send questions and comments. - Subscribe or Unsubscribe - https://www.ibm.com/support/mynotifications - Feedback - https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html - Follow us on Twitter - https://twitter.com/IBMStorageSupt To ensure proper delivery please add mynotify at stg.events.ihost.com to your address book. You received this email because you are subscribed to IBM My Notifications as: aaron.s.knister at nasa.gov Please do not reply to this message as it is generated by an automated service machine. (C) International Business Machines Corporation 2017. All rights reserved. From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 30 18:25:56 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 30 Jun 2017 17:25:56 +0000 Subject: [gpfsug-discuss] Mass UID migration suggestions In-Reply-To: <59566c3b.kqOSy9e2kg80XZ/k%hpc-luke@uconn.edu> References: <59566c3b.kqOSy9e2kg80XZ/k%hpc-luke@uconn.edu> Message-ID: <1901859B-7620-42E2-9064-14930AC50EE3@vanderbilt.edu> Hi Luke, I?ve got an off the wall suggestion for you, which may or may not work depending on whether or not you have any UID conflicts with old and new UIDs ? this won?t actually speed things up but it will eliminate the ?downtime? for your users. And the big caveat is that there can?t be any UID conflicts ? i.e. someone?s new UID can?t be someone else?s old UID. Given that ? what if you set an ACL to allow access to both their old and new UIDs ? then change their UID to the new UID ? then chown the files to the new UID and remove the ACL? More work for you, but no downtime for them. We actually may need to do something similar as we will need to change Windows-assigned UID?s based on SIDs to ?correct? UIDs at some point in the future on one of our storage systems. If someone has a better way to solve your problem, I hope they?ll post it to the list, as it may help us as well. HTHAL. Thanks? Kevin On Jun 30, 2017, at 10:20 AM, hpc-luke at uconn.edu wrote: Hello, We're trying to change most of our users uids, is there a clean way to migrate all of one users files with say `mmapplypolicy`? We have to change the owner of around 273539588 files, and my estimates for runtime are around 6 days. What we've been doing is indexing all of the files and splitting them up by owner which takes around an hour, and then we were locking the user out while we chown their files. I made it multi threaded as it weirdly gave a 10% speedup despite my expectation that multi threading access from a single node would not give any speedup. Generally I'm looking for advice on how to make the chowning faster. Would spreading the chowning processes over multiple nodes improve performance? Should I not stat the files before running lchown on them, since lchown checks the file before changing it? I saw mention of inodescan(), in an old gpfsug email, which speeds up disk read access, by not guaranteeing that the data is up to date. We have a maintenance day coming up where all users will be locked out, so the file handles(?) from GPFS's perspective will not be able to go stale. Is there a function with similar constraints to inodescan that I can use to speed up this process? Thank you for your time, Luke Storrs-HPC University of Connecticut _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Fri Jun 30 18:37:30 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Fri, 30 Jun 2017 19:37:30 +0200 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Fri Jun 30 18:41:43 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Fri, 30 Jun 2017 13:41:43 -0400 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: Thanks Olaf, that's good to know (and is kind of what I suspected). I've requested a number of times this capability for those of us who can't use or aren't using GNR and the answer is effectively "no". This response is curious to me because I'm sure IBM doesn't believe that data integrity is only important and of value to customers who purchase their hardware *and* software. -Aaron On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser wrote: > yes.. in case of GNR (GPFS native raid) .. we do end-to-end check-summing > ... client --> server --> downToDisk > GNR writes down a chksum to disk (to all pdisks /all "raid" segments ) so > that dropped writes can be detected as well as miss-done writes (bit > flips..) > > > > From: Aaron Knister > To: gpfsug main discussion list > Date: 06/30/2017 07:15 PM > Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): > RDMA-enabled network adapter failure on the NSD server may result in file > IO error (2017.06.30) > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > I'm curious to know why this doesn't affect GSS/ESS? Is it a feature of > the additional check-summing done on those platforms? > > > -------- Forwarded Message -------- > Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled > network adapter > failure on the NSD server may result in file IO error (2017.06.30) > Date: Fri, 30 Jun 2017 14:19:02 +0000 > From: IBM My Notifications > > To: aaron.s.knister at nasa.gov > > > > > My Notifications for Storage - 30 Jun 2017 > > Dear Subscriber (aaron.s.knister at nasa.gov), > > Here are your updates from IBM My Notifications. > > Your support Notifications display in English by default. Machine > translation based on your IBM profile > language setting is added if you specify this option in My defaults > within My Notifications. > (Note: Not all languages are available at this time, and the English > version always takes precedence > over the machine translated version.) > > ------------------------------------------------------------ > ------------------ > 1. IBM Spectrum Scale > > - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure > on the NSD server may result in file IO error > - URL: > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233& > myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_- > OCSTXKQY-OCSWJ00-_-E > - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM > Spectrum Scale versions where the NSD server is enabled to use RDMA for > file IO and the storage used in your GPFS cluster accessed via NSD > servers (not fully SAN accessible) includes anything other than IBM > Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under these > conditions, when the RDMA-enabled network adapter fails, the issue may > result in undetected data corruption for file write or read operations. > > ------------------------------------------------------------ > ------------------ > Manage your My Notifications subscriptions, or send questions and comments. > - Subscribe or Unsubscribe - https://www.ibm.com/support/mynotifications > - Feedback - > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotif > ications.html > > - Follow us on Twitter - https://twitter.com/IBMStorageSupt > > > > To ensure proper delivery please add mynotify at stg.events.ihost.com to > your address book. > You received this email because you are subscribed to IBM My > Notifications as: > aaron.s.knister at nasa.gov > > Please do not reply to this message as it is generated by an automated > service machine. > > (C) International Business Machines Corporation 2017. All rights reserved. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Jun 30 18:53:16 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 30 Jun 2017 13:53:16 -0400 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: <2689cf86-eca2-dab6-c6aa-7fc54d923e55@nasa.gov> In fact the answer was quite literally "no": https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523 (the RFE was declined and the answer was that the "function is already available in GNR environments"). Regarding GNR, see this RFE request https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090 requesting the use of GNR outside of an ESS/GSS environment. It's interesting to note this is the highest voted Public RFE for GPFS that I can see, at least. It too was declined. -Aaron On 6/30/17 1:41 PM, Aaron Knister wrote: > Thanks Olaf, that's good to know (and is kind of what I suspected). I've > requested a number of times this capability for those of us who can't > use or aren't using GNR and the answer is effectively "no". This > response is curious to me because I'm sure IBM doesn't believe that data > integrity is only important and of value to customers who purchase their > hardware *and* software. > > -Aaron > > On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser > wrote: > > yes.. in case of GNR (GPFS native raid) .. we do end-to-end > check-summing ... client --> server --> downToDisk > GNR writes down a chksum to disk (to all pdisks /all "raid" segments > ) so that dropped writes can be detected as well as miss-done > writes (bit flips..) > > > > From: Aaron Knister > > To: gpfsug main discussion list > > Date: 06/30/2017 07:15 PM > Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): > RDMA-enabled network adapter failure on the NSD server may result in > file IO error (2017.06.30) > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------ > > > > I'm curious to know why this doesn't affect GSS/ESS? Is it a feature of > the additional check-summing done on those platforms? > > > -------- Forwarded Message -------- > Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network > adapter > failure on the NSD server may result in file IO error (2017.06.30) > Date: Fri, 30 Jun 2017 14:19:02 +0000 > From: IBM My Notifications > > > To: aaron.s.knister at nasa.gov > > > > > My Notifications for Storage - 30 Jun 2017 > > Dear Subscriber (aaron.s.knister at nasa.gov > ), > > Here are your updates from IBM My Notifications. > > Your support Notifications display in English by default. Machine > translation based on your IBM profile > language setting is added if you specify this option in My defaults > within My Notifications. > (Note: Not all languages are available at this time, and the English > version always takes precedence > over the machine translated version.) > > ------------------------------------------------------------------------------ > 1. IBM Spectrum Scale > > - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter > failure > on the NSD server may result in file IO error > - URL: > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E > > - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM > Spectrum Scale versions where the NSD server is enabled to use RDMA for > file IO and the storage used in your GPFS cluster accessed via NSD > servers (not fully SAN accessible) includes anything other than IBM > Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under these > conditions, when the RDMA-enabled network adapter fails, the issue may > result in undetected data corruption for file write or read operations. > > ------------------------------------------------------------------------------ > Manage your My Notifications subscriptions, or send questions and > comments. > - Subscribe or Unsubscribe - > https://www.ibm.com/support/mynotifications > > - Feedback - > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html > > > - Follow us on Twitter - https://twitter.com/IBMStorageSupt > > > > > To ensure proper delivery please add mynotify at stg.events.ihost.com > to > your address book. > You received this email because you are subscribed to IBM My > Notifications as: > aaron.s.knister at nasa.gov > > Please do not reply to this message as it is generated by an automated > service machine. > > (C) International Business Machines Corporation 2017. All rights > reserved. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From oehmes at gmail.com Fri Jun 30 19:25:28 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 30 Jun 2017 18:25:28 +0000 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: <2689cf86-eca2-dab6-c6aa-7fc54d923e55@nasa.gov> References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> <2689cf86-eca2-dab6-c6aa-7fc54d923e55@nasa.gov> Message-ID: end-to-end data integrity is very important and the reason it hasn't been done in Scale is not because its not important, its because its very hard to do without impacting performance in a very dramatic way. imagine your raid controller blocksize is 1mb and your filesystem blocksize is 1MB . if your application does a 1 MB write this ends up being a perfect full block , full track de-stage to your raid layer and everything works fine and fast. as soon as you add checksum support you need to add data somehow into this, means your 1MB is no longer 1 MB but 1 MB+checksum. to store this additional data you have multiple options, inline , outside the data block or some combination ,the net is either you need to do more physical i/o's to different places to get both the data and the corresponding checksum or your per block on disc structure becomes bigger than than what your application reads/or writes, both put massive burden on the Storage layer as e.g. a 1 MB write will now, even the blocks are all aligned from the application down to the raid layer, cause a read/modify/write on the raid layer as the data is bigger than the physical track size. so to get end-to-end checksum in Scale outside of ESS the best way is to get GNR as SW to run on generic HW, this is what people should vote for as RFE if they need that functionality. beside end-to-end checksums you get read/write cache and acceleration , fast rebuild and many other goodies as a added bonus. Sven On Fri, Jun 30, 2017 at 10:53 AM Aaron Knister wrote: > In fact the answer was quite literally "no": > > https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523 > (the RFE was declined and the answer was that the "function is already > available in GNR environments"). > > Regarding GNR, see this RFE request > https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090 > requesting the use of GNR outside of an ESS/GSS environment. It's > interesting to note this is the highest voted Public RFE for GPFS that I > can see, at least. It too was declined. > > -Aaron > > On 6/30/17 1:41 PM, Aaron Knister wrote: > > Thanks Olaf, that's good to know (and is kind of what I suspected). I've > > requested a number of times this capability for those of us who can't > > use or aren't using GNR and the answer is effectively "no". This > > response is curious to me because I'm sure IBM doesn't believe that data > > integrity is only important and of value to customers who purchase their > > hardware *and* software. > > > > -Aaron > > > > On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser > > wrote: > > > > yes.. in case of GNR (GPFS native raid) .. we do end-to-end > > check-summing ... client --> server --> downToDisk > > GNR writes down a chksum to disk (to all pdisks /all "raid" segments > > ) so that dropped writes can be detected as well as miss-done > > writes (bit flips..) > > > > > > > > From: Aaron Knister > > > > To: gpfsug main discussion list > > > > Date: 06/30/2017 07:15 PM > > Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): > > RDMA-enabled network adapter failure on the NSD server may result in > > file IO error (2017.06.30) > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > ------------------------------------------------------------------------ > > > > > > > > I'm curious to know why this doesn't affect GSS/ESS? Is it a feature > of > > the additional check-summing done on those platforms? > > > > > > -------- Forwarded Message -------- > > Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network > > adapter > > failure on the NSD server may result in file IO error (2017.06.30) > > Date: Fri, 30 Jun 2017 14:19:02 +0000 > > From: IBM My Notifications > > >> > > To: aaron.s.knister at nasa.gov > > > > > > > > > > My Notifications for Storage - 30 Jun 2017 > > > > Dear Subscriber (aaron.s.knister at nasa.gov > > ), > > > > Here are your updates from IBM My Notifications. > > > > Your support Notifications display in English by default. Machine > > translation based on your IBM profile > > language setting is added if you specify this option in My defaults > > within My Notifications. > > (Note: Not all languages are available at this time, and the English > > version always takes precedence > > over the machine translated version.) > > > > > ------------------------------------------------------------------------------ > > 1. IBM Spectrum Scale > > > > - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter > > failure > > on the NSD server may result in file IO error > > - URL: > > > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E > > < > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E > > > > - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM > > Spectrum Scale versions where the NSD server is enabled to use RDMA > for > > file IO and the storage used in your GPFS cluster accessed via NSD > > servers (not fully SAN accessible) includes anything other than IBM > > Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under > these > > conditions, when the RDMA-enabled network adapter fails, the issue > may > > result in undetected data corruption for file write or read > operations. > > > > > ------------------------------------------------------------------------------ > > Manage your My Notifications subscriptions, or send questions and > > comments. > > - Subscribe or Unsubscribe - > > https://www.ibm.com/support/mynotifications > > > > - Feedback - > > > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html > > < > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html > > > > > > - Follow us on Twitter - https://twitter.com/IBMStorageSupt > > > > > > > > > > To ensure proper delivery please add mynotify at stg.events.ihost.com > > to > > your address book. > > You received this email because you are subscribed to IBM My > > Notifications as: > > aaron.s.knister at nasa.gov > > > > Please do not reply to this message as it is generated by an > automated > > service machine. > > > > (C) International Business Machines Corporation 2017. All rights > > reserved. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alandhae at gmx.de Thu Jun 1 10:35:37 2017 From: alandhae at gmx.de (=?ISO-8859-15?Q?Andreas_Landh=E4u=DFer?=) Date: Thu, 1 Jun 2017 11:35:37 +0200 (CEST) Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup Message-ID: Hello all out there, customer wants to receive periodical reports on the filesystem heat (relatively age) of files. We already switched on fileheat using mmchconfig. mmchconfig fileheatlosspercent=10,fileHeatPeriodMinutes=1440 for the reports, I think I need to know the file usage in a given time period. Are there any how-to for obtaining this information, examples for ILM policies to be used as a start? any help will be highly appreciated. Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de From olaf.weiser at de.ibm.com Thu Jun 1 12:15:57 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 1 Jun 2017 13:15:57 +0200 Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Thu Jun 1 14:40:30 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 1 Jun 2017 09:40:30 -0400 Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup In-Reply-To: References: Message-ID: To generate a list of files and their file heat values... define([TS],esyscmd(date +%Y-%m-%d-%H-%M | tr -d '\n')) RULE 'x1' EXTERNAL LIST 'heat.TS' EXEC '' RULE 'x2' LIST 'heat.TS' SHOW(FILE_HEAT) WEIGHT(FILE_HEAT) /* use a WHERE clause to select or exclude files */ mmapplypolicy /gpfs-path-of-interest -I defer -P policy-rules-shown-above -f /path-for-result ... other good options are -N nodes -g /shared-temp To do it periodically use crontab. I defined the TS macro so each time you run it, you get a different filename. WEIGHT clause will cause "hottest" files to be listed firstmost. Notice that FILE_HEAT "shows" as a floating point number. --marc From: "Olaf Weiser" To: Andreas Landh?u?er , gpfsug main discussion list Date: 06/01/2017 07:17 AM Subject: Re: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Andreas, one could use the WEIGHT statement ... a simple policy for e.g. rule ?repack? MIGRATE FROM POOL ?xxxxxx? TO POOL ?xxxx? WEIGHT(FILE_HEAT) and then the -I prepare to see, what would be done by policy.. or you use the LIST function .. or .. and so on .. From: Andreas Landh?u?er To: gpfsug-discuss at spectrumscale.org Date: 06/01/2017 11:36 AM Subject: [gpfsug-discuss] gpfs filesystem heat reporting, howto setup Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all out there, customer wants to receive periodical reports on the filesystem heat (relatively age) of files. We already switched on fileheat using mmchconfig. mmchconfig fileheatlosspercent=10,fileHeatPeriodMinutes=1440 for the reports, I think I need to know the file usage in a given time period. Are there any how-to for obtaining this information, examples for ILM policies to be used as a start? any help will be highly appreciated. Best regards Andreas -- Andreas Landh?u?er +49 151 12133027 (mobile) alandhae at gmx.de_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Fri Jun 2 04:10:44 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Fri, 2 Jun 2017 03:10:44 +0000 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - Space Management (GPFS HSM) Message-ID: An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Fri Jun 2 10:28:36 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 02 Jun 2017 05:28:36 -0400 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - Space Management (GPFS HSM) In-Reply-To: References: Message-ID: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca> We have that situation. Users don't need to login to NSD's What you need is to add at least one gpfs client to the cluster (or multi-cluster), mount the DMAPI enabled file system, and use that node as a gateway for end-users. They can access the contents on the mount point with their own underprivileged accounts. Whether or not on a schedule, the moment an application or linux command (such as cp, cat, vi, etc) accesses a stub, the file will be staged. Jaime Quoting "Andrew Beattie" : > Quick question, Does anyone have a Scale / GPFS environment (HPC) > where users need the ability to recall data sets after they have been > stubbed, but only System Administrators are permitted to log onto the > NSD servers for security purposes. And if so how do you provide the > ability for the users to schedule their data set recalls? Regards, > Andrew Beattie Software Defined Storage - IT Specialist Phone: > 614-2133-7927 E-mail: abeattie at au1.ibm.com[1] > > > Links: > ------ > [1] mailto:abeattie at au1.ibm.com > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From abeattie at au1.ibm.com Fri Jun 2 10:48:11 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Fri, 2 Jun 2017 09:48:11 +0000 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) In-Reply-To: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca> References: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca>, Message-ID: An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Fri Jun 2 16:12:41 2017 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 02 Jun 2017 11:12:41 -0400 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) In-Reply-To: References: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca>, Message-ID: <20170602111241.56882fx2qr2yz2ax@support.scinet.utoronto.ca> It has been a while since I used HSM with GPFS via TSM, but as far as I can remember, unprivileged users can run dsmmigrate and dsmrecall. Based on the instructions on the link, dsmrecall may now leverage the Recommended Access Order (RAO) available on enterprise drives, however root would have to be the one to invoke that feature. In that case we may have to develop a middleware/wrapper for dsmrecall that will run as root and act on behalf of the user when optimization is requested. Someone here more familiar with the latest version of TSM-HSM may be able to give us some hints on how people are doing this in practice. Jaime Quoting "Andrew Beattie" : > Thanks Jaime, How do you get around Optimised recalls? from what I > can see the optimised recall process needs a root level account to > retrieve a list of files > https://www.ibm.com/support/knowledgecenter/SSSR2R_7.1.1/com.ibm.itsm.hsmul.doc/c_recall_optimized_tape.html[1] > Regards, Andrew Beattie Software Defined Storage - IT Specialist > Phone: 614-2133-7927 E-mail: abeattie at au1.ibm.com[2] ----- > Original message ----- > From: "Jaime Pinto" > To: "gpfsug main discussion list" , > "Andrew Beattie" > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - > Space Management (GPFS HSM) > Date: Fri, Jun 2, 2017 7:28 PM > We have that situation. > Users don't need to login to NSD's > > What you need is to add at least one gpfs client to the cluster (or > multi-cluster), mount the DMAPI enabled file system, and use that > node > as a gateway for end-users. They can access the contents on the mount > > point with their own underprivileged accounts. > > Whether or not on a schedule, the moment an application or linux > command (such as cp, cat, vi, etc) accesses a stub, the file will be > > staged. > > Jaime > > Quoting "Andrew Beattie" : > >> Quick question, Does anyone have a Scale / GPFS environment (HPC) >> where users need the ability to recall data sets after they have > been >> stubbed, but only System Administrators are permitted to log onto > the >> NSD servers for security purposes. And if so how do you provide > the >> ability for the users to schedule their data set recalls? > Regards, >> Andrew Beattie Software Defined Storage - IT Specialist Phone: >> 614-2133-7927 E-mail: abeattie at au1.ibm.com[1] >> >> >> Links: >> ------ >> [1] mailto:abeattie at au1.ibm.com[3] >> > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials[4] > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From r.sobey at imperial.ac.uk Fri Jun 2 16:51:12 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 2 Jun 2017 15:51:12 +0000 Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS Message-ID: Hi all, Where should I start looking for a compatibility matrix between TSM and GPFS? Specifically, we are currently running TSM 7.1.6-2 and GPFS 4.2.1-2 with the intent to upgrade to GPFS 4.2.3-latest in early July. I've spent 30 minutes looking over various documents and the best I can find is this: http://www-01.ibm.com/support/docview.wss?uid=swg21248771 ..which talks about TSM in a Space Management context and would suggest that we need to upgrade to Spectrum Protect i.e. 8.1 and that GPFS 4.2.2.x is the maximum supported version... Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 2 17:40:11 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 2 Jun 2017 12:40:11 -0400 Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS In-Reply-To: References: Message-ID: Upgrading from GPFS 4.2.x to GPFS 4.2.y should not "break" TSM. If it does, someone goofed, that would be a bug. (My opinion) Think of it this way. TSM is an application that uses the OS and the FileSystem(s). TSM can't verify it will work with all future versions of OS and Filesystems, and the releases can't be in lock step. Having said that, 4.2.3 has been "out" for a while, so if there were a TSM incompatibility, someone would have likely hit it or will before July... Trust but verify... From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 06/02/2017 11:51 AM Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi all, Where should I start looking for a compatibility matrix between TSM and GPFS? Specifically, we are currently running TSM 7.1.6-2 and GPFS 4.2.1-2 with the intent to upgrade to GPFS 4.2.3-latest in early July. I?ve spent 30 minutes looking over various documents and the best I can find is this: http://www-01.ibm.com/support/docview.wss?uid=swg21248771 ..which talks about TSM in a Space Management context and would suggest that we need to upgrade to Spectrum Protect i.e. 8.1 and that GPFS 4.2.2.x is the maximum supported version? Cheers Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 08:51:10 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 08:51:10 +0100 Subject: [gpfsug-discuss] NSD access routes Message-ID: Morning all, Just a quick one about NSD access and read only disks. Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? Cheers, ----------------------------- Dave Goodbourn Head of Systems Milk Visual Effects Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Mon Jun 5 08:52:39 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Mon, 5 Jun 2017 07:52:39 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: Message-ID: Hi Have you look at LROC instead? Might fit in simpler way to what your are describing. -- Cheers > On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: > > Morning all, > > Just a quick one about NSD access and read only disks. > > Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. > > Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? > > Cheers, > ----------------------------- > Dave Goodbourn > > Head of Systems > Milk Visual Effects > Tel: +44 (0)20 3697 8448 > Mob: +44 (0)7917 411 069 Ellei edell?? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 09:02:16 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 09:02:16 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: Yeah, that was my back up plan but would be more costly in the cloud. Read only is a limitation of most cloud providers not something that I "want". Just trying to move a network bottleneck. Cheers, ----------------------------- Dave Goodbourn Head of Systems Milk Visual Effects Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 > On 5 Jun 2017, at 08:52, Luis Bolinches wrote: > > Hi > > Have you look at LROC instead? Might fit in simpler way to what your are describing. > > -- > Cheers > >> On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: >> >> Morning all, >> >> Just a quick one about NSD access and read only disks. >> >> Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. >> >> Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? >> >> Cheers, >> ----------------------------- >> Dave Goodbourn >> >> Head of Systems >> Milk Visual Effects >> Tel: +44 (0)20 3697 8448 >> Mob: +44 (0)7917 411 069 > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 13:19:47 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 13:19:47 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: OK scrap my first question, I can't do what I wanted to do anyway! I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. Cheers, ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 08:52, Luis Bolinches wrote: > Hi > > Have you look at LROC instead? Might fit in simpler way to what your are > describing. > > -- > Cheers > > On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: > > Morning all, > > Just a quick one about NSD access and read only disks. > > Can you have 2 NSD servers, one with read/write access to a disk and one > with just read only access to the same disk? I know you can write to a disk > over the network via another NSD server but can you mount the disk in read > only mode to increase the read performance? This is all virtual/cloud based. > > Is GPFS clever enough (or can it be configured) to know to read from the > locally attached read only disk but write back via another NSD server over > the GPFS network? > > Cheers, > ----------------------------- > Dave Goodbourn > > Head of Systems > Milk Visual Effects > Tel: +44 (0)20 3697 8448 <+44%2020%203697%208448> > Mob: +44 (0)7917 411 069 <+44%207917%20411069> > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Jun 5 13:24:27 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Mon, 5 Jun 2017 12:24:27 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: mmdiag --lroc ? From: > on behalf of "dave at milk-vfx.com" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 5 June 2017 at 13:19 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] NSD access routes OK scrap my first question, I can't do what I wanted to do anyway! I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. Cheers, ---------------------------------------------------- Dave Goodbourn Head of Systems MILK VISUAL EFFECTS [http://www.milk-vfx.com/src/milk_email_logo.jpg] 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 On 5 June 2017 at 08:52, Luis Bolinches > wrote: Hi Have you look at LROC instead? Might fit in simpler way to what your are describing. -- Cheers On 5 Jun 2017, at 10.51, Dave Goodbourn > wrote: Morning all, Just a quick one about NSD access and read only disks. Can you have 2 NSD servers, one with read/write access to a disk and one with just read only access to the same disk? I know you can write to a disk over the network via another NSD server but can you mount the disk in read only mode to increase the read performance? This is all virtual/cloud based. Is GPFS clever enough (or can it be configured) to know to read from the locally attached read only disk but write back via another NSD server over the GPFS network? Cheers, ----------------------------- Dave Goodbourn Head of Systems Milk Visual Effects Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 5 13:48:48 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 5 Jun 2017 12:48:48 +0000 Subject: [gpfsug-discuss] NSD access routes Message-ID: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Hi Dave I?ve done a large-scale (600 node) LROC deployment here - feel free to reach out if you have questions. mmdiag --lroc is about all there is but it does give you a pretty good idea how the cache is performing but you can?t tell which files are cached. Also, watch out that the LROC cached will steal pagepool memory (1% of the LROC cache size) Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Dave Goodbourn Reply-To: gpfsug main discussion list Date: Monday, June 5, 2017 at 7:19 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] NSD access routes I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 14:10:21 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 14:10:21 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: Message-ID: Ah yep, thanks a lot. ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 13:24, Simon Thompson (IT Research Support) < S.J.Thompson at bham.ac.uk> wrote: > mmdiag --lroc > > ? > > > From: on behalf of " > dave at milk-vfx.com" > Reply-To: "gpfsug-discuss at spectrumscale.org" < > gpfsug-discuss at spectrumscale.org> > Date: Monday, 5 June 2017 at 13:19 > To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] NSD access routes > > OK scrap my first question, I can't do what I wanted to do anyway! > > I'm testing out the LROC idea. All seems to be working well, but, is there > anyway to monitor what's cached? How full it might be? The performance etc?? > > I can see some stats in mmfsadm dump lroc but that's about it. > > Cheers, > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 08:52, Luis Bolinches wrote: > >> Hi >> >> Have you look at LROC instead? Might fit in simpler way to what your are >> describing. >> >> -- >> Cheers >> >> On 5 Jun 2017, at 10.51, Dave Goodbourn wrote: >> >> Morning all, >> >> Just a quick one about NSD access and read only disks. >> >> Can you have 2 NSD servers, one with read/write access to a disk and one >> with just read only access to the same disk? I know you can write to a disk >> over the network via another NSD server but can you mount the disk in read >> only mode to increase the read performance? This is all virtual/cloud based. >> >> Is GPFS clever enough (or can it be configured) to know to read from the >> locally attached read only disk but write back via another NSD server over >> the GPFS network? >> >> Cheers, >> ----------------------------- >> Dave Goodbourn >> >> Head of Systems >> Milk Visual Effects >> Tel: +44 (0)20 3697 8448 <+44%2020%203697%208448> >> Mob: +44 (0)7917 411 069 <+44%207917%20411069> >> >> >> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >> Oy IBM Finland Ab >> PL 265, 00101 Helsinki, Finland >> Business ID, Y-tunnus: 0195876-3 >> Registered in Finland >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 14:49:55 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 14:49:55 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: Thanks Bob, That pagepool comment has just answered my next question! But it doesn't seem to be working. Here's my mmdiag output: === mmdiag: lroc === LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' status Running Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 Max capacity: 1151997 MB, currently in use: 0 MB Statistics from: Mon Jun 5 13:40:50 2017 Total objects stored 0 (0 MB) recalled 0 (0 MB) objects failed to store 0 failed to recall 0 failed to inval 0 objects queried 0 (0 MB) not found 0 = 0.00 % objects invalidated 0 (0 MB) Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Inode objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Directory objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Data objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 agent inserts=0, reads=0 response times (usec): insert min/max/avg=0/0/0 read min/max/avg=0/0/0 ssd writeIOs=0, writePages=0 readIOs=0, readPages=0 response times (usec): write min/max/avg=0/0/0 read min/max/avg=0/0/0 I've restarted GPFS on that node just in case but that didn't seem to help. I have LROC on a node that DOESN'T have direct access to an NSD so will hopefully cache files that get requested over NFS. How often are these stats updated? The Statistics line doesn't seem to update when running the command again. Dave, ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 13:48, Oesterlin, Robert wrote: > Hi Dave > > > > I?ve done a large-scale (600 node) LROC deployment here - feel free to > reach out if you have questions. > > > > mmdiag --lroc is about all there is but it does give you a pretty good > idea how the cache is performing but you can?t tell which files are cached. > Also, watch out that the LROC cached will steal pagepool memory (1% of the > LROC cache size) > > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > > > > > *From: * on behalf of Dave > Goodbourn > *Reply-To: *gpfsug main discussion list > *Date: *Monday, June 5, 2017 at 7:19 AM > *To: *gpfsug main discussion list > *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes > > > > I'm testing out the LROC idea. All seems to be working well, but, is there > anyway to monitor what's cached? How full it might be? The performance etc?? > > > > I can see some stats in mmfsadm dump lroc but that's about it. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 14:55:22 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 14:55:22 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: OK slightly ignore that last email. It's still not updating the output but I realise the Stats from line is when they started so probably won't update! :( Still nothing seems to being cached though. ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 14:49, Dave Goodbourn wrote: > Thanks Bob, > > That pagepool comment has just answered my next question! > > But it doesn't seem to be working. Here's my mmdiag output: > > === mmdiag: lroc === > LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' > status Running > Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 > Max capacity: 1151997 MB, currently in use: 0 MB > Statistics from: Mon Jun 5 13:40:50 2017 > > Total objects stored 0 (0 MB) recalled 0 (0 MB) > objects failed to store 0 failed to recall 0 failed to inval 0 > objects queried 0 (0 MB) not found 0 = 0.00 % > objects invalidated 0 (0 MB) > > Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Inode objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Directory objects failed to store 0 failed to recall 0 failed to > query 0 failed to inval 0 > > Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Data objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > agent inserts=0, reads=0 > response times (usec): > insert min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > ssd writeIOs=0, writePages=0 > readIOs=0, readPages=0 > response times (usec): > write min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > > I've restarted GPFS on that node just in case but that didn't seem to > help. I have LROC on a node that DOESN'T have direct access to an NSD so > will hopefully cache files that get requested over NFS. > > How often are these stats updated? The Statistics line doesn't seem to > update when running the command again. > > Dave, > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 13:48, Oesterlin, Robert > wrote: > >> Hi Dave >> >> >> >> I?ve done a large-scale (600 node) LROC deployment here - feel free to >> reach out if you have questions. >> >> >> >> mmdiag --lroc is about all there is but it does give you a pretty good >> idea how the cache is performing but you can?t tell which files are cached. >> Also, watch out that the LROC cached will steal pagepool memory (1% of the >> LROC cache size) >> >> >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> >> >> >> >> *From: * on behalf of Dave >> Goodbourn >> *Reply-To: *gpfsug main discussion list > > >> *Date: *Monday, June 5, 2017 at 7:19 AM >> *To: *gpfsug main discussion list >> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >> >> >> >> I'm testing out the LROC idea. All seems to be working well, but, is >> there anyway to monitor what's cached? How full it might be? The >> performance etc?? >> >> >> >> I can see some stats in mmfsadm dump lroc but that's about it. >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Jun 5 14:59:07 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Mon, 5 Jun 2017 13:59:07 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> , Message-ID: We've seen exactly this behaviour. Removing and readding the lroc nsd device worked for us. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of dave at milk-vfx.com [dave at milk-vfx.com] Sent: 05 June 2017 14:55 To: Oesterlin, Robert Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] NSD access routes OK slightly ignore that last email. It's still not updating the output but I realise the Stats from line is when they started so probably won't update! :( Still nothing seems to being cached though. ---------------------------------------------------- Dave Goodbourn Head of Systems MILK VISUAL EFFECTS [http://www.milk-vfx.com/src/milk_email_logo.jpg] 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 On 5 June 2017 at 14:49, Dave Goodbourn > wrote: Thanks Bob, That pagepool comment has just answered my next question! But it doesn't seem to be working. Here's my mmdiag output: === mmdiag: lroc === LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' status Running Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 Max capacity: 1151997 MB, currently in use: 0 MB Statistics from: Mon Jun 5 13:40:50 2017 Total objects stored 0 (0 MB) recalled 0 (0 MB) objects failed to store 0 failed to recall 0 failed to inval 0 objects queried 0 (0 MB) not found 0 = 0.00 % objects invalidated 0 (0 MB) Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Inode objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Directory objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) Data objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0 agent inserts=0, reads=0 response times (usec): insert min/max/avg=0/0/0 read min/max/avg=0/0/0 ssd writeIOs=0, writePages=0 readIOs=0, readPages=0 response times (usec): write min/max/avg=0/0/0 read min/max/avg=0/0/0 I've restarted GPFS on that node just in case but that didn't seem to help. I have LROC on a node that DOESN'T have direct access to an NSD so will hopefully cache files that get requested over NFS. How often are these stats updated? The Statistics line doesn't seem to update when running the command again. Dave, ---------------------------------------------------- Dave Goodbourn Head of Systems MILK VISUAL EFFECTS [http://www.milk-vfx.com/src/milk_email_logo.jpg] 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: +44 (0)20 3697 8448 Mob: +44 (0)7917 411 069 On 5 June 2017 at 13:48, Oesterlin, Robert > wrote: Hi Dave I?ve done a large-scale (600 node) LROC deployment here - feel free to reach out if you have questions. mmdiag --lroc is about all there is but it does give you a pretty good idea how the cache is performing but you can?t tell which files are cached. Also, watch out that the LROC cached will steal pagepool memory (1% of the LROC cache size) Bob Oesterlin Sr Principal Storage Engineer, Nuance From: > on behalf of Dave Goodbourn > Reply-To: gpfsug main discussion list > Date: Monday, June 5, 2017 at 7:19 AM To: gpfsug main discussion list > Subject: [EXTERNAL] Re: [gpfsug-discuss] NSD access routes I'm testing out the LROC idea. All seems to be working well, but, is there anyway to monitor what's cached? How full it might be? The performance etc?? I can see some stats in mmfsadm dump lroc but that's about it. From oehmes at gmail.com Mon Jun 5 14:59:44 2017 From: oehmes at gmail.com (Sven Oehme) Date: Mon, 05 Jun 2017 13:59:44 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: if you are using O_DIRECT calls they will be ignored by default for LROC, same for encrypted data. how exactly are you testing this? On Mon, Jun 5, 2017 at 6:50 AM Dave Goodbourn wrote: > Thanks Bob, > > That pagepool comment has just answered my next question! > > But it doesn't seem to be working. Here's my mmdiag output: > > === mmdiag: lroc === > LROC Device(s): > '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' > status Running > Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 > Max capacity: 1151997 MB, currently in use: 0 MB > Statistics from: Mon Jun 5 13:40:50 2017 > > Total objects stored 0 (0 MB) recalled 0 (0 MB) > objects failed to store 0 failed to recall 0 failed to inval 0 > objects queried 0 (0 MB) not found 0 = 0.00 % > objects invalidated 0 (0 MB) > > Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Inode objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Directory objects failed to store 0 failed to recall 0 failed to > query 0 failed to inval 0 > > Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % > Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) > Data objects failed to store 0 failed to recall 0 failed to query 0 > failed to inval 0 > > agent inserts=0, reads=0 > response times (usec): > insert min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > ssd writeIOs=0, writePages=0 > readIOs=0, readPages=0 > response times (usec): > write min/max/avg=0/0/0 > read min/max/avg=0/0/0 > > > I've restarted GPFS on that node just in case but that didn't seem to > help. I have LROC on a node that DOESN'T have direct access to an NSD so > will hopefully cache files that get requested over NFS. > > How often are these stats updated? The Statistics line doesn't seem to > update when running the command again. > > Dave, > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 13:48, Oesterlin, Robert > wrote: > >> Hi Dave >> >> >> >> I?ve done a large-scale (600 node) LROC deployment here - feel free to >> reach out if you have questions. >> >> >> >> mmdiag --lroc is about all there is but it does give you a pretty good >> idea how the cache is performing but you can?t tell which files are cached. >> Also, watch out that the LROC cached will steal pagepool memory (1% of the >> LROC cache size) >> >> >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> >> >> >> >> *From: * on behalf of Dave >> Goodbourn >> *Reply-To: *gpfsug main discussion list > > >> *Date: *Monday, June 5, 2017 at 7:19 AM >> *To: *gpfsug main discussion list >> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >> >> >> >> I'm testing out the LROC idea. All seems to be working well, but, is >> there anyway to monitor what's cached? How full it might be? The >> performance etc?? >> >> >> >> I can see some stats in mmfsadm dump lroc but that's about it. >> >> >> >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 15:00:45 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 15:00:45 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: OK I'm going to hang my head in the corner...RTFM...I've not filled the memory buffer pool yet so I doubt it will have anything in it yet!! :( ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 14:55, Dave Goodbourn wrote: > OK slightly ignore that last email. It's still not updating the output but > I realise the Stats from line is when they started so probably won't > update! :( > > Still nothing seems to being cached though. > > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 14:49, Dave Goodbourn wrote: > >> Thanks Bob, >> >> That pagepool comment has just answered my next question! >> >> But it doesn't seem to be working. Here's my mmdiag output: >> >> === mmdiag: lroc === >> LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF >> 0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' status Running >> Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 >> Max capacity: 1151997 MB, currently in use: 0 MB >> Statistics from: Mon Jun 5 13:40:50 2017 >> >> Total objects stored 0 (0 MB) recalled 0 (0 MB) >> objects failed to store 0 failed to recall 0 failed to inval 0 >> objects queried 0 (0 MB) not found 0 = 0.00 % >> objects invalidated 0 (0 MB) >> >> Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >> Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >> Inode objects failed to store 0 failed to recall 0 failed to query >> 0 failed to inval 0 >> >> Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >> Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >> Directory objects failed to store 0 failed to recall 0 failed to >> query 0 failed to inval 0 >> >> Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >> Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >> Data objects failed to store 0 failed to recall 0 failed to query 0 >> failed to inval 0 >> >> agent inserts=0, reads=0 >> response times (usec): >> insert min/max/avg=0/0/0 >> read min/max/avg=0/0/0 >> >> ssd writeIOs=0, writePages=0 >> readIOs=0, readPages=0 >> response times (usec): >> write min/max/avg=0/0/0 >> read min/max/avg=0/0/0 >> >> >> I've restarted GPFS on that node just in case but that didn't seem to >> help. I have LROC on a node that DOESN'T have direct access to an NSD so >> will hopefully cache files that get requested over NFS. >> >> How often are these stats updated? The Statistics line doesn't seem to >> update when running the command again. >> >> Dave, >> ---------------------------------------------------- >> *Dave Goodbourn* >> Head of Systems >> *MILK VISUAL EFFECTS* >> >> 5th floor, Threeways House, >> 40-44 Clipstone Street London, W1W 5DW >> Tel: *+44 (0)20 3697 8448* >> Mob: *+44 (0)7917 411 069* >> >> On 5 June 2017 at 13:48, Oesterlin, Robert >> wrote: >> >>> Hi Dave >>> >>> >>> >>> I?ve done a large-scale (600 node) LROC deployment here - feel free to >>> reach out if you have questions. >>> >>> >>> >>> mmdiag --lroc is about all there is but it does give you a pretty good >>> idea how the cache is performing but you can?t tell which files are cached. >>> Also, watch out that the LROC cached will steal pagepool memory (1% of the >>> LROC cache size) >>> >>> >>> >>> Bob Oesterlin >>> Sr Principal Storage Engineer, Nuance >>> >>> >>> >>> >>> >>> >>> >>> *From: * on behalf of Dave >>> Goodbourn >>> *Reply-To: *gpfsug main discussion list >> org> >>> *Date: *Monday, June 5, 2017 at 7:19 AM >>> *To: *gpfsug main discussion list >>> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >>> >>> >>> >>> I'm testing out the LROC idea. All seems to be working well, but, is >>> there anyway to monitor what's cached? How full it might be? The >>> performance etc?? >>> >>> >>> >>> I can see some stats in mmfsadm dump lroc but that's about it. >>> >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Mon Jun 5 15:03:28 2017 From: oehmes at gmail.com (Sven Oehme) Date: Mon, 05 Jun 2017 14:03:28 +0000 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: yes as long as you haven't pushed anything to it (means pagepool got under enough pressure to free up space) you won't see anything in the stats :-) sven On Mon, Jun 5, 2017 at 7:00 AM Dave Goodbourn wrote: > OK I'm going to hang my head in the corner...RTFM...I've not filled the > memory buffer pool yet so I doubt it will have anything in it yet!! :( > > ---------------------------------------------------- > *Dave Goodbourn* > Head of Systems > *MILK VISUAL EFFECTS* > > 5th floor, Threeways House, > 40-44 Clipstone Street London, W1W 5DW > Tel: *+44 (0)20 3697 8448* > Mob: *+44 (0)7917 411 069* > > On 5 June 2017 at 14:55, Dave Goodbourn wrote: > >> OK slightly ignore that last email. It's still not updating the output >> but I realise the Stats from line is when they started so probably won't >> update! :( >> >> Still nothing seems to being cached though. >> >> ---------------------------------------------------- >> *Dave Goodbourn* >> Head of Systems >> *MILK VISUAL EFFECTS* >> >> 5th floor, Threeways House, >> 40-44 Clipstone Street London, W1W 5DW >> Tel: *+44 (0)20 3697 8448* >> Mob: *+44 (0)7917 411 069* >> >> On 5 June 2017 at 14:49, Dave Goodbourn wrote: >> >>> Thanks Bob, >>> >>> That pagepool comment has just answered my next question! >>> >>> But it doesn't seem to be working. Here's my mmdiag output: >>> >>> === mmdiag: lroc === >>> LROC Device(s): >>> '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' >>> status Running >>> Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 >>> Max capacity: 1151997 MB, currently in use: 0 MB >>> Statistics from: Mon Jun 5 13:40:50 2017 >>> >>> Total objects stored 0 (0 MB) recalled 0 (0 MB) >>> objects failed to store 0 failed to recall 0 failed to inval 0 >>> objects queried 0 (0 MB) not found 0 = 0.00 % >>> objects invalidated 0 (0 MB) >>> >>> Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>> Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>> Inode objects failed to store 0 failed to recall 0 failed to query >>> 0 failed to inval 0 >>> >>> Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>> Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>> Directory objects failed to store 0 failed to recall 0 failed to >>> query 0 failed to inval 0 >>> >>> Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>> Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>> Data objects failed to store 0 failed to recall 0 failed to query >>> 0 failed to inval 0 >>> >>> agent inserts=0, reads=0 >>> response times (usec): >>> insert min/max/avg=0/0/0 >>> read min/max/avg=0/0/0 >>> >>> ssd writeIOs=0, writePages=0 >>> readIOs=0, readPages=0 >>> response times (usec): >>> write min/max/avg=0/0/0 >>> read min/max/avg=0/0/0 >>> >>> >>> I've restarted GPFS on that node just in case but that didn't seem to >>> help. I have LROC on a node that DOESN'T have direct access to an NSD so >>> will hopefully cache files that get requested over NFS. >>> >>> How often are these stats updated? The Statistics line doesn't seem to >>> update when running the command again. >>> >>> Dave, >>> ---------------------------------------------------- >>> *Dave Goodbourn* >>> Head of Systems >>> *MILK VISUAL EFFECTS* >>> >>> 5th floor, Threeways House, >>> 40-44 Clipstone Street London, W1W 5DW >>> Tel: *+44 (0)20 3697 8448* >>> Mob: *+44 (0)7917 411 069* >>> >>> On 5 June 2017 at 13:48, Oesterlin, Robert >>> wrote: >>> >>>> Hi Dave >>>> >>>> >>>> >>>> I?ve done a large-scale (600 node) LROC deployment here - feel free to >>>> reach out if you have questions. >>>> >>>> >>>> >>>> mmdiag --lroc is about all there is but it does give you a pretty good >>>> idea how the cache is performing but you can?t tell which files are cached. >>>> Also, watch out that the LROC cached will steal pagepool memory (1% of the >>>> LROC cache size) >>>> >>>> >>>> >>>> Bob Oesterlin >>>> Sr Principal Storage Engineer, Nuance >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *From: * on behalf of Dave >>>> Goodbourn >>>> *Reply-To: *gpfsug main discussion list < >>>> gpfsug-discuss at spectrumscale.org> >>>> *Date: *Monday, June 5, 2017 at 7:19 AM >>>> *To: *gpfsug main discussion list >>>> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >>>> >>>> >>>> >>>> I'm testing out the LROC idea. All seems to be working well, but, is >>>> there anyway to monitor what's cached? How full it might be? The >>>> performance etc?? >>>> >>>> >>>> >>>> I can see some stats in mmfsadm dump lroc but that's about it. >>>> >>>> >>>> >>>> >>> >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at milk-vfx.com Mon Jun 5 15:15:00 2017 From: dave at milk-vfx.com (Dave Goodbourn) Date: Mon, 5 Jun 2017 15:15:00 +0100 Subject: [gpfsug-discuss] NSD access routes In-Reply-To: References: <259A2E45-7B84-4637-A37E-B18C4271DDB2@nuance.com> Message-ID: Ha! A quick shrink of the pagepool and we're in action! Thanks all. Dave. ---------------------------------------------------- *Dave Goodbourn* Head of Systems *MILK VISUAL EFFECTS* 5th floor, Threeways House, 40-44 Clipstone Street London, W1W 5DW Tel: *+44 (0)20 3697 8448* Mob: *+44 (0)7917 411 069* On 5 June 2017 at 15:03, Sven Oehme wrote: > yes as long as you haven't pushed anything to it (means pagepool got under > enough pressure to free up space) you won't see anything in the stats :-) > > sven > > > On Mon, Jun 5, 2017 at 7:00 AM Dave Goodbourn wrote: > >> OK I'm going to hang my head in the corner...RTFM...I've not filled the >> memory buffer pool yet so I doubt it will have anything in it yet!! :( >> >> ---------------------------------------------------- >> *Dave Goodbourn* >> Head of Systems >> *MILK VISUAL EFFECTS* >> >> 5th floor, Threeways House, >> 40-44 Clipstone Street London, W1W 5DW >> Tel: *+44 (0)20 3697 8448* >> Mob: *+44 (0)7917 411 069* >> >> On 5 June 2017 at 14:55, Dave Goodbourn wrote: >> >>> OK slightly ignore that last email. It's still not updating the output >>> but I realise the Stats from line is when they started so probably won't >>> update! :( >>> >>> Still nothing seems to being cached though. >>> >>> ---------------------------------------------------- >>> *Dave Goodbourn* >>> Head of Systems >>> *MILK VISUAL EFFECTS* >>> >>> 5th floor, Threeways House, >>> 40-44 Clipstone Street London, W1W 5DW >>> Tel: *+44 (0)20 3697 8448* >>> Mob: *+44 (0)7917 411 069* >>> >>> On 5 June 2017 at 14:49, Dave Goodbourn wrote: >>> >>>> Thanks Bob, >>>> >>>> That pagepool comment has just answered my next question! >>>> >>>> But it doesn't seem to be working. Here's my mmdiag output: >>>> >>>> === mmdiag: lroc === >>>> LROC Device(s): '0AF0000259355BA8#/dev/sdb;0AF0000259355BA9#/dev/sdc;0AF0000259355BAA#/dev/sdd;' >>>> status Running >>>> Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 >>>> Max capacity: 1151997 MB, currently in use: 0 MB >>>> Statistics from: Mon Jun 5 13:40:50 2017 >>>> >>>> Total objects stored 0 (0 MB) recalled 0 (0 MB) >>>> objects failed to store 0 failed to recall 0 failed to inval 0 >>>> objects queried 0 (0 MB) not found 0 = 0.00 % >>>> objects invalidated 0 (0 MB) >>>> >>>> Inode objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>>> Inode objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>>> Inode objects failed to store 0 failed to recall 0 failed to >>>> query 0 failed to inval 0 >>>> >>>> Directory objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>>> Directory objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>>> Directory objects failed to store 0 failed to recall 0 failed to >>>> query 0 failed to inval 0 >>>> >>>> Data objects stored 0 (0 MB) recalled 0 (0 MB) = 0.00 % >>>> Data objects queried 0 (0 MB) = 0.00 % invalidated 0 (0 MB) >>>> Data objects failed to store 0 failed to recall 0 failed to query >>>> 0 failed to inval 0 >>>> >>>> agent inserts=0, reads=0 >>>> response times (usec): >>>> insert min/max/avg=0/0/0 >>>> read min/max/avg=0/0/0 >>>> >>>> ssd writeIOs=0, writePages=0 >>>> readIOs=0, readPages=0 >>>> response times (usec): >>>> write min/max/avg=0/0/0 >>>> read min/max/avg=0/0/0 >>>> >>>> >>>> I've restarted GPFS on that node just in case but that didn't seem to >>>> help. I have LROC on a node that DOESN'T have direct access to an NSD so >>>> will hopefully cache files that get requested over NFS. >>>> >>>> How often are these stats updated? The Statistics line doesn't seem to >>>> update when running the command again. >>>> >>>> Dave, >>>> ---------------------------------------------------- >>>> *Dave Goodbourn* >>>> Head of Systems >>>> *MILK VISUAL EFFECTS* >>>> >>>> 5th floor, Threeways House, >>>> 40-44 Clipstone Street London, W1W 5DW >>>> Tel: *+44 (0)20 3697 8448* >>>> Mob: *+44 (0)7917 411 069* >>>> >>>> On 5 June 2017 at 13:48, Oesterlin, Robert >>> > wrote: >>>> >>>>> Hi Dave >>>>> >>>>> >>>>> >>>>> I?ve done a large-scale (600 node) LROC deployment here - feel free to >>>>> reach out if you have questions. >>>>> >>>>> >>>>> >>>>> mmdiag --lroc is about all there is but it does give you a pretty good >>>>> idea how the cache is performing but you can?t tell which files are cached. >>>>> Also, watch out that the LROC cached will steal pagepool memory (1% of the >>>>> LROC cache size) >>>>> >>>>> >>>>> >>>>> Bob Oesterlin >>>>> Sr Principal Storage Engineer, Nuance >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *From: * on behalf of Dave >>>>> Goodbourn >>>>> *Reply-To: *gpfsug main discussion list >>>> org> >>>>> *Date: *Monday, June 5, 2017 at 7:19 AM >>>>> *To: *gpfsug main discussion list >>>>> *Subject: *[EXTERNAL] Re: [gpfsug-discuss] NSD access routes >>>>> >>>>> >>>>> >>>>> I'm testing out the LROC idea. All seems to be working well, but, is >>>>> there anyway to monitor what's cached? How full it might be? The >>>>> performance etc?? >>>>> >>>>> >>>>> >>>>> I can see some stats in mmfsadm dump lroc but that's about it. >>>>> >>>>> >>>>> >>>>> >>>> >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 5 16:54:09 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 5 Jun 2017 15:54:09 +0000 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add Message-ID: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> Our node build process re-adds a node to the cluster and then does a ?service gpfs start?, but GPFS doesn?t start. From the build log: + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' + rc=0 + chkconfig gpfs on + service gpfs start The ?service gpfs start? command hangs and never seems to return. If I look at the process tree: [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12292 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e s/\.num$// 21639 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 This is GPFS 4.2.2-1 This seems to occur only on the initial startup after build - if I try to start GPFS again, it works just fine - any ideas on what it?s sitting here waiting? Nothing in mmfslog (does not exist) Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Mon Jun 5 20:54:31 2017 From: ewahl at osc.edu (Edward Wahl) Date: Mon, 5 Jun 2017 15:54:31 -0400 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add In-Reply-To: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> References: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> Message-ID: <20170605155431.75b42322@osc.edu> Just a thought, as we noticed the EXACT opposite of this, and what I think is new behavior in either mmmount or mmfsfuncs.. Does the file system exist in your /etc/fstab (or AIX equiv) yet? Ed On Mon, 5 Jun 2017 15:54:09 +0000 "Oesterlin, Robert" wrote: > Our node build process re-adds a node to the cluster and then does a ?service > gpfs start?, but GPFS doesn?t start. From the build log: > > + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com > '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' > + rc=0 > + chkconfig gpfs on > + service gpfs start > > The ?service gpfs start? command hangs and never seems to return. > > If I look at the process tree: > > [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" > 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post > 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 > 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? > S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S > 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > 12292 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num > 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e > s/\.num$// 21639 ? S > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 > > This is GPFS 4.2.2-1 > > This seems to occur only on the initial startup after build - if I try to > start GPFS again, it works just fine - any ideas on what it?s sitting here > waiting? Nothing in mmfslog (does not exist) > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > 507-269-0413 > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From ewahl at osc.edu Mon Jun 5 20:56:55 2017 From: ewahl at osc.edu (Edward Wahl) Date: Mon, 5 Jun 2017 15:56:55 -0400 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add In-Reply-To: <20170605155431.75b42322@osc.edu> References: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> <20170605155431.75b42322@osc.edu> Message-ID: <20170605155655.3ce54084@osc.edu> On Mon, 5 Jun 2017 15:54:31 -0400 Edward Wahl wrote: > Just a thought, as we noticed the EXACT opposite of this, and what I think is > new behavior in either mmmount or .. Does the file system exist in > your /etc/fstab (or AIX equiv) yet? Apologies, I meant mmsdrfsdef, not mmfsfuncs. Ed > > Ed > > On Mon, 5 Jun 2017 15:54:09 +0000 > "Oesterlin, Robert" wrote: > > > Our node build process re-adds a node to the cluster and then does a > > ?service gpfs start?, but GPFS doesn?t start. >From the build log: > > > > + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com > > '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' > > + rc=0 > > + chkconfig gpfs on > > + service gpfs start > > > > The ?service gpfs start? command hangs and never seems to return. > > > > If I look at the process tree: > > > > [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" > > 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post > > 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 > > 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? > > S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S > > 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > > 12292 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot > > 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num > > 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e > > s/\.num$// 21639 ? S > > 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 > > > > This is GPFS 4.2.2-1 > > > > This seems to occur only on the initial startup after build - if I try to > > start GPFS again, it works just fine - any ideas on what it?s sitting here > > waiting? Nothing in mmfslog (does not exist) > > > > Bob Oesterlin > > Sr Principal Storage Engineer, Nuance > > 507-269-0413 > > > > > > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From scale at us.ibm.com Mon Jun 5 22:49:23 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 5 Jun 2017 17:49:23 -0400 Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add In-Reply-To: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> References: <1314020E-D554-47AC-81A1-371B5A526817@nuance.com> Message-ID: Looks like a bug in the code. The command hung in grep command. It has missing argument. Please open a PMR to have this fix. Instead of "service gpfs start", can you use mmstartup? You can also try to run mm list command before service gpfs start as a workaround. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/05/2017 11:54 AM Subject: [gpfsug-discuss] Odd behavior - GPSF failed to start after initial node add Sent by: gpfsug-discuss-bounces at spectrumscale.org Our node build process re-adds a node to the cluster and then does a ?service gpfs start?, but GPFS doesn?t start. From the build log: + ssh -o StrictHostKeyChecking=no nrg1-gpfs01.nrg1.us.grid.nuance.com '/usr/local/sbin/addnode.sh cnq-r02r09u27.nrg1.us.grid.nuance.com' + rc=0 + chkconfig gpfs on + service gpfs start The ?service gpfs start? command hangs and never seems to return. If I look at the process tree: [root at cnq-r02r09u27 ~]# ps ax | egrep "mm|gpfs" 11715 ? S 0:00 /bin/bash ./nrgX_gpfs_post 12191 ? Ssl 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log 128 yes no 12208 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 12271 ? S 0:00 /bin/sh /sbin/service gpfs start 12276 ? S 0:00 /bin/sh /etc/init.d/gpfs start 12278 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12292 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmautoload reboot 12293 ? S 0:00 /bin/grep -lw /var/mmfs/gen/nodeFiles/*.num 12294 ? S 0:00 /bin/sed -e s%/var/mmfs/gen/nodeFiles/....%% -e s/\.num$// 21639 ? S 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15 This is GPFS 4.2.2-1 This seems to occur only on the initial startup after build - if I try to start GPFS again, it works just fine - any ideas on what it?s sitting here waiting? Nothing in mmfslog (does not exist) Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From stijn.deweirdt at ugent.be Tue Jun 6 08:05:06 2017 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Tue, 6 Jun 2017 09:05:06 +0200 Subject: [gpfsug-discuss] gpfs waiters debugging Message-ID: hi all, we have recently been hit by quite a few cases that triggered long waiters. we are aware of the excellent slides http://files.gpfsug.org/presentations/2017/NERSC/GPFS-Troubleshooting-Apr-2017.pdf but we are wondering if and how we can cause those waiters ourself, so we can train ourself in debugging and resolving them (either on test system or in controlled environment on the production clusters). all hints welcome. stijn From Robert.Oesterlin at nuance.com Tue Jun 6 12:44:31 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 6 Jun 2017 11:44:31 +0000 Subject: [gpfsug-discuss] gpfs waiters debugging Message-ID: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> Hi Stijn You need to provide some more details on the type and duration of the waiters before the group can offer some advice. Bob Oesterlin Sr Principal Storage Engineer, Nuance On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Stijn De Weirdt" wrote: but we are wondering if and how we can cause those waiters ourself, so we can train ourself in debugging and resolving them (either on test system or in controlled environment on the production clusters). all hints welcome. stijn _______________________________________________ From stijn.deweirdt at ugent.be Tue Jun 6 13:29:43 2017 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Tue, 6 Jun 2017 14:29:43 +0200 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> Message-ID: <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> hi bob, waiters from RPC replies and/or threads waiting on mutex are most "popular". but my question is not how to resolve them, the question is how to create such a waiter so we can train ourself in grep and mmfsadm etc etc we want to recreate the waiters a few times, try out some things and either script or at least put instructions on our internal wiki what to do. the instructions in the slides are clear enough, but there are a lot of slides, and typically when this occurs offshift, you don't want to start with rereading the slides and wondering what to do next; let alone debug scripts ;) thanks, stijn On 06/06/2017 01:44 PM, Oesterlin, Robert wrote: > Hi Stijn > > You need to provide some more details on the type and duration of the waiters before the group can offer some advice. > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Stijn De Weirdt" wrote: > > > but we are wondering if and how we can cause those waiters ourself, so > we can train ourself in debugging and resolving them (either on test > system or in controlled environment on the production clusters). > > all hints welcome. > > stijn > _______________________________________________ > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From stockf at us.ibm.com Tue Jun 6 13:57:00 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 6 Jun 2017 08:57:00 -0400 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> Message-ID: Realize that generally any waiter under 1 second should be ignored. In an active GPFS system there are always waiters and the greater the use of the system likely the more waiters you will see. The point is waiters themselves are not an indication your system is having problems. As for creating them any steady level of activity against the file system should cause waiters to appear, though most should be of a short duration. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: Stijn De Weirdt To: gpfsug-discuss at spectrumscale.org Date: 06/06/2017 08:31 AM Subject: Re: [gpfsug-discuss] gpfs waiters debugging Sent by: gpfsug-discuss-bounces at spectrumscale.org hi bob, waiters from RPC replies and/or threads waiting on mutex are most "popular". but my question is not how to resolve them, the question is how to create such a waiter so we can train ourself in grep and mmfsadm etc etc we want to recreate the waiters a few times, try out some things and either script or at least put instructions on our internal wiki what to do. the instructions in the slides are clear enough, but there are a lot of slides, and typically when this occurs offshift, you don't want to start with rereading the slides and wondering what to do next; let alone debug scripts ;) thanks, stijn On 06/06/2017 01:44 PM, Oesterlin, Robert wrote: > Hi Stijn > > You need to provide some more details on the type and duration of the waiters before the group can offer some advice. > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > > On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Stijn De Weirdt" wrote: > > > but we are wondering if and how we can cause those waiters ourself, so > we can train ourself in debugging and resolving them (either on test > system or in controlled environment on the production clusters). > > all hints welcome. > > stijn > _______________________________________________ > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stijn.deweirdt at ugent.be Tue Jun 6 14:06:57 2017 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Tue, 6 Jun 2017 15:06:57 +0200 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> Message-ID: oh sure, i meant waiters that last > 300 seconds or so (something that could trigger deadlock). obviously we're not interested in debugging the short ones, it's not that gpfs doesn't work or anything ;) stijn On 06/06/2017 02:57 PM, Frederick Stock wrote: > Realize that generally any waiter under 1 second should be ignored. In an > active GPFS system there are always waiters and the greater the use of the > system likely the more waiters you will see. The point is waiters > themselves are not an indication your system is having problems. > > As for creating them any steady level of activity against the file system > should cause waiters to appear, though most should be of a short duration. > > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > stockf at us.ibm.com > > > > From: Stijn De Weirdt > To: gpfsug-discuss at spectrumscale.org > Date: 06/06/2017 08:31 AM > Subject: Re: [gpfsug-discuss] gpfs waiters debugging > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > hi bob, > > waiters from RPC replies and/or threads waiting on mutex are most > "popular". > > but my question is not how to resolve them, the question is how to > create such a waiter so we can train ourself in grep and mmfsadm etc etc > > we want to recreate the waiters a few times, try out some things and > either script or at least put instructions on our internal wiki what to > do. > > the instructions in the slides are clear enough, but there are a lot of > slides, and typically when this occurs offshift, you don't want to start > with rereading the slides and wondering what to do next; let alone debug > scripts ;) > > thanks, > > stijn > > On 06/06/2017 01:44 PM, Oesterlin, Robert wrote: >> Hi Stijn >> >> You need to provide some more details on the type and duration of the > waiters before the group can offer some advice. >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> >> >> >> On 6/6/17, 2:05 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of Stijn De Weirdt" stijn.deweirdt at ugent.be> wrote: >> >> >> but we are wondering if and how we can cause those waiters ourself, > so >> we can train ourself in debugging and resolving them (either on test >> system or in controlled environment on the production clusters). >> >> all hints welcome. >> >> stijn >> _______________________________________________ >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From valdis.kletnieks at vt.edu Tue Jun 6 17:45:51 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 06 Jun 2017 12:45:51 -0400 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> Message-ID: <6873.1496767551@turing-police.cc.vt.edu> On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt said: > oh sure, i meant waiters that last > 300 seconds or so (something that > could trigger deadlock). obviously we're not interested in debugging the > short ones, it's not that gpfs doesn't work or anything ;) At least at one time, a lot of the mm(whatever) administrative commands would leave one dangling waiter for the duration of the command - which could be a while if the command was mmdeldisk or mmrestripefs. I admit not having specifically checked for gpfs 4.2, but it was true for 3.2 through 4.1.... And my addition to the collective debugging knowledge: A bash one-liner to dump all the waiters across a cluster, sorted by wait time. Note that our clusters tend to be 5-8 servers, this may be painful for those of you who have 400+ node clusters. :) ##!/bin/bash for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh $i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done | sort -n -r -k 3 -t' ' We've found it useful - if you have 1 waiter on one node that's 1278 seconds old, and 3 other nodes have waiters that are 1275 seconds old, it's a good chance the other 3 nodes waiters are waiting on the first node's waiter to resolve itself.... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From stockf at us.ibm.com Tue Jun 6 17:54:06 2017 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 6 Jun 2017 12:54:06 -0400 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: <6873.1496767551@turing-police.cc.vt.edu> References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com><3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> <6873.1496767551@turing-police.cc.vt.edu> Message-ID: On recent releases you can accomplish the same with the command, "mmlsnode -N waiters -L". Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list Date: 06/06/2017 12:46 PM Subject: Re: [gpfsug-discuss] gpfs waiters debugging Sent by: gpfsug-discuss-bounces at spectrumscale.org On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt said: > oh sure, i meant waiters that last > 300 seconds or so (something that > could trigger deadlock). obviously we're not interested in debugging the > short ones, it's not that gpfs doesn't work or anything ;) At least at one time, a lot of the mm(whatever) administrative commands would leave one dangling waiter for the duration of the command - which could be a while if the command was mmdeldisk or mmrestripefs. I admit not having specifically checked for gpfs 4.2, but it was true for 3.2 through 4.1.... And my addition to the collective debugging knowledge: A bash one-liner to dump all the waiters across a cluster, sorted by wait time. Note that our clusters tend to be 5-8 servers, this may be painful for those of you who have 400+ node clusters. :) ##!/bin/bash for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh $i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done | sort -n -r -k 3 -t' ' We've found it useful - if you have 1 waiter on one node that's 1278 seconds old, and 3 other nodes have waiters that are 1275 seconds old, it's a good chance the other 3 nodes waiters are waiting on the first node's waiter to resolve itself.... [attachment "attltepl.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Tue Jun 6 19:05:15 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 6 Jun 2017 14:05:15 -0400 Subject: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) In-Reply-To: <20170602111241.56882fx2qr2yz2ax@support.scinet.utoronto.ca> References: <20170602052836.11563o7dj205wptw@support.scinet.utoronto.ca>, <20170602111241.56882fx2qr2yz2ax@support.scinet.utoronto.ca> Message-ID: Hi, Just as Jaime has explained, any GPFS node in the cluster, can induce a recall (as he called "staged") by access to file data. It is not optimized by tape order, and a dynamic file access of any pattern, such as "find" or "cat *" will surely result in an inefficient processing of the data recall if all data lives in physical tape. But if migrated data lives on spinning disk on the TSM server, there is no harm in such a recall pattern because recalls from a disk pool incur no significant overhead or delay for tape loading and positioning. Unprivileged users may not run "dsmcrecall" because creating a DMAPI session as the dsmrecall program must do, requires admin user privilege on that node. You may be able to wrap dsmrecall in a set-uid wrapper if you want to permit users to run that, but of course that comes with the danger that a recall storm could monopolize resources on your cluster. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Jaime Pinto" To: "Andrew Beattie" Cc: gpfsug-discuss at spectrumscale.org Date: 06/02/2017 11:13 AM Subject: Re: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - SpaceManagement (GPFS HSM) Sent by: gpfsug-discuss-bounces at spectrumscale.org It has been a while since I used HSM with GPFS via TSM, but as far as I can remember, unprivileged users can run dsmmigrate and dsmrecall. Based on the instructions on the link, dsmrecall may now leverage the Recommended Access Order (RAO) available on enterprise drives, however root would have to be the one to invoke that feature. In that case we may have to develop a middleware/wrapper for dsmrecall that will run as root and act on behalf of the user when optimization is requested. Someone here more familiar with the latest version of TSM-HSM may be able to give us some hints on how people are doing this in practice. Jaime Quoting "Andrew Beattie" : > Thanks Jaime, How do you get around Optimised recalls? from what I > can see the optimised recall process needs a root level account to > retrieve a list of files > https://www.ibm.com/support/knowledgecenter/SSSR2R_7.1.1/com.ibm.itsm.hsmul.doc/c_recall_optimized_tape.html [1] > Regards, Andrew Beattie Software Defined Storage - IT Specialist > Phone: 614-2133-7927 E-mail: abeattie at au1.ibm.com[2] ----- > Original message ----- > From: "Jaime Pinto" > To: "gpfsug main discussion list" , > "Andrew Beattie" > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale - Spectrum Protect - > Space Management (GPFS HSM) > Date: Fri, Jun 2, 2017 7:28 PM > We have that situation. > Users don't need to login to NSD's > > What you need is to add at least one gpfs client to the cluster (or > multi-cluster), mount the DMAPI enabled file system, and use that > node > as a gateway for end-users. They can access the contents on the mount > > point with their own underprivileged accounts. > > Whether or not on a schedule, the moment an application or linux > command (such as cp, cat, vi, etc) accesses a stub, the file will be > > staged. > > Jaime > > Quoting "Andrew Beattie" : > >> Quick question, Does anyone have a Scale / GPFS environment (HPC) >> where users need the ability to recall data sets after they have > been >> stubbed, but only System Administrators are permitted to log onto > the >> NSD servers for security purposes. And if so how do you provide > the >> ability for the users to schedule their data set recalls? > Regards, >> Andrew Beattie Software Defined Storage - IT Specialist Phone: >> 614-2133-7927 E-mail: abeattie at au1.ibm.com[1] >> >> >> Links: >> ------ >> [1] mailto:abeattie at au1.ibm.com[3] >> > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials[4] > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jun 6 19:15:22 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 6 Jun 2017 18:15:22 +0000 Subject: [gpfsug-discuss] gpfs waiters debugging In-Reply-To: References: <3152E93F-DC76-456F-BBAC-E203D06E597E@nuance.com> <3cbb9375-86c9-3f2e-ec3a-bd4cea1455d8@ugent.be> <6873.1496767551@turing-police.cc.vt.edu> Message-ID: All, mmlsnode -N waiters is great ? I also appreciate the ?-s? option to it. Very helpful when you know the problem started say, slightly more than half an hour ago and you therefore don?t care about sub-1800 second waiters? Kevin On Jun 6, 2017, at 11:54 AM, Frederick Stock > wrote: On recent releases you can accomplish the same with the command, "mmlsnode -N waiters -L". Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list > Date: 06/06/2017 12:46 PM Subject: Re: [gpfsug-discuss] gpfs waiters debugging Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt said: > oh sure, i meant waiters that last > 300 seconds or so (something that > could trigger deadlock). obviously we're not interested in debugging the > short ones, it's not that gpfs doesn't work or anything ;) At least at one time, a lot of the mm(whatever) administrative commands would leave one dangling waiter for the duration of the command - which could be a while if the command was mmdeldisk or mmrestripefs. I admit not having specifically checked for gpfs 4.2, but it was true for 3.2 through 4.1.... And my addition to the collective debugging knowledge: A bash one-liner to dump all the waiters across a cluster, sorted by wait time. Note that our clusters tend to be 5-8 servers, this may be painful for those of you who have 400+ node clusters. :) ##!/bin/bash for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh $i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done | sort -n -r -k 3 -t' ' We've found it useful - if you have 1 waiter on one node that's 1278 seconds old, and 3 other nodes have waiters that are 1275 seconds old, it's a good chance the other 3 nodes waiters are waiting on the first node's waiter to resolve itself.... [attachment "attltepl.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Tue Jun 6 21:31:01 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 06 Jun 2017 16:31:01 -0400 Subject: [gpfsug-discuss] mmapplypolicy and ltfsee - identifying progress... Message-ID: <25944.1496781061@turing-police.cc.vt.edu> So I'm trying to get a handle on where exactly an mmapplypolicy that's doing a premigrate is in its progress. I've already determined that 'ltfsee info jobs' will only report where in the current batch it is, but that still leaves me unable to tell the difference between [I] 2017-06-05 at 17:31:47.995 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 10000 files dispatched. and [I] 2017-06-06 at 02:44:48.236 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 225000 files dispatched. Is there any better way to figure out where it is than writing the cron job to launch it as mmapplypolicy | tee /tmp/something and then go scraping the messages? (And yes, I know not all chunks of 1,000 files are created equal. Sometimes it's 1,000 C source files that total to less than a megabyte, other times it's 1,000 streaming video files that total to over a terabye - but even knowing it's 194,000 into 243,348 files is better than what I have now...) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jun 6 22:20:31 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 6 Jun 2017 21:20:31 +0000 Subject: [gpfsug-discuss] mmapplypolicy and ltfsee - identifying progress... In-Reply-To: <25944.1496781061@turing-police.cc.vt.edu> References: <25944.1496781061@turing-police.cc.vt.edu> Message-ID: <5987990A-39F6-47A5-981D-A34A3054E4D8@vanderbilt.edu> Hi Valdis, I?m not sure this is ?better?, but what I typically do is have mmapplypolicy running from a shell script launched by a cron job and redirecting output to a file in /tmp. Once the mmapplypolicy finishes the SysAdmin?s get the tmp file e-mailed to them and then it gets deleted. Of course, while the mmapplypolicy is running you can ?tail -f /tmp/mmapplypolicy.log? or grep it or whatever. HTHAL? Kevin On Jun 6, 2017, at 3:31 PM, valdis.kletnieks at vt.edu wrote: So I'm trying to get a handle on where exactly an mmapplypolicy that's doing a premigrate is in its progress. I've already determined that 'ltfsee info jobs' will only report where in the current batch it is, but that still leaves me unable to tell the difference between [I] 2017-06-05 at 17:31:47.995 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 10000 files dispatched. and [I] 2017-06-06 at 02:44:48.236 Executing file list: /gpfs/archive/config/tmp/ mmPolicy.chosnlist.97168.79FD2A24.pre. 225000 files dispatched. Is there any better way to figure out where it is than writing the cron job to launch it as mmapplypolicy | tee /tmp/something and then go scraping the messages? (And yes, I know not all chunks of 1,000 files are created equal. Sometimes it's 1,000 C source files that total to less than a megabyte, other times it's 1,000 streaming video files that total to over a terabye - but even knowing it's 194,000 into 243,348 files is better than what I have now...) ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From TROPPENS at de.ibm.com Wed Jun 7 09:30:17 2017 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Wed, 7 Jun 2017 08:30:17 +0000 Subject: [gpfsug-discuss] ISC 2017 - Agenda User Meeting Message-ID: An HTML attachment was scrubbed... URL: From jan.sundermann at kit.edu Wed Jun 7 15:04:28 2017 From: jan.sundermann at kit.edu (Sundermann, Jan Erik (SCC)) Date: Wed, 7 Jun 2017 14:04:28 +0000 Subject: [gpfsug-discuss] Upgrade with architecture change Message-ID: Hi, we are operating a small Spectrum Scale cluster with about 100 clients and 6 NSD servers. The cluster is FPO-enabled. For historical reasons the NSD servers are running on ppc64 while the clients are a mixture of ppc64le and x86_64 machines. Most machines are running Red Hat Enterprise Linux 7 but we also have few machines running AIX. At the moment we have installed Spectrum Scale version 4.1.1 but would like to do an upgrade to 4.2.3. In the course of the upgrade we would like to change the architecture of all NSD servers and reinstall them with ppc64le instead of ppc64. From what I?ve learned so far it should be possible to upgrade directly from 4.1.1 to 4.2.3. Before doing the upgrade we would like to ask for some advice on the best strategy. For the NSD servers, one by one, we are thinking about doing the following: 1) Disable auto recovery 2) Unmount GPFS file system 3) Suspend disks 4) Shutdown gpfs 5) Reboot and reinstall with changed architecture ppc64le 6) Install gpfs 4.2.3 7) Recover cluster config using mmsdrrestore 8) Resume and start disks 9) Reenable auto recovery Can GPFS handle the change of the NSD server?s architecture and would it be fine to operate a mixture of different architectures for the NSD servers? Thanks, Jan Erik From tarak.patel at canada.ca Wed Jun 7 16:42:45 2017 From: tarak.patel at canada.ca (Patel, Tarak (SSC/SPC)) Date: Wed, 7 Jun 2017 15:42:45 +0000 Subject: [gpfsug-discuss] Remote cluster gpfs communication on IP different then one for Daemon or Admin node name. Message-ID: <50fd0dc6cf47485c8728fc09b7ae0263@PEVDACDEXC009.birch.int.bell.ca> Hi all, We've been experiencing issues with remote cluster node expelling CES nodes causing remote filesystems to unmount. The issue is related gpfs communication using Ethernet IP rather than IP defined on IB which is used for Daemon node name and Admin node name. So remote cluster is aware of IPs that are not defined in GPFS configuration as Admin/Daemon node name. The CES nodes are configure to have IB as well as Ethernet (for client interactive and NFS access). We've double checked /etc/hosts and DNS and all looks to be in order since the CES IPoIB IP is present in /etc/hosts of remote cluster. I'm unsure where cluster manager for remote cluster is getting the Ethernet IP if there is no mention of it in GPFS configuration. The CES nodes were added later therefore they are not listed as Contact Nodes in 'mmremotecluster show' output. The CES nodes use IP defined on IB for GPFS configuration and we also have Ethernet which has the default route defined. In order to ensure that all IB communication passes via IPoIB, we've even defined a static route so that all GPFS communication will use IPoIB (since we are dealing with a different fabric). 'mmfsadm dump tscomm' reports multiple IPs for CES nodes which includes the Ethernet and also the IPoIB. I'm unsure if there is a way to drop some connections on GPFS (cluster wide) after stopping a specific CES node and ensure that only IB is listed. I realize that one option would be to define subnet parameter for remote cluster which will require a downtime (solution to be explored at later date). Hope that someone can explain how or why remote cluster is picking IPs not used in GPFS config for remote nodes and how to ensure those IPs are not used in future. Thank you, Tarak -- Tarak Patel Chef d'?quipe, Integration HPC, Solution de calcul E-Science Service partag? Canada / Gouvernment du Canada tarak.patel at canada.ca 1-514-421-7299 Team Lead, HPC Integration, E-Science Computing Solution Shared Services Canada, Government of Canada tarak.patel at canada.ca 1-514-421-7299 -------------- next part -------------- An HTML attachment was scrubbed... URL: From chekh at stanford.edu Wed Jun 7 23:12:56 2017 From: chekh at stanford.edu (Alex Chekholko) Date: Wed, 7 Jun 2017 15:12:56 -0700 Subject: [gpfsug-discuss] Upgrade with architecture change In-Reply-To: References: Message-ID: Hi Jan, I don't have hands-on experience with FPO or ppc64 but your procedure sounds OK to me. How do you currently handle just shutting down an NSD node for maintenance? I guess you'd have the same process except skip 5,6,7 How do you currently handle OS rebuild on NSD node? Maybe try that first without the architecture change. But I don't see why it would matter so long as you don't touch the GPFS disks. Regards, Alex On 06/07/2017 07:04 AM, Sundermann, Jan Erik (SCC) wrote: > Hi, > > we are operating a small Spectrum Scale cluster with about 100 clients and 6 NSD servers. The cluster is FPO-enabled. For historical reasons the NSD servers are running on ppc64 while the clients are a mixture of ppc64le and x86_64 machines. Most machines are running Red Hat Enterprise Linux 7 but we also have few machines running AIX. > > At the moment we have installed Spectrum Scale version 4.1.1 but would like to do an upgrade to 4.2.3. In the course of the upgrade we would like to change the architecture of all NSD servers and reinstall them with ppc64le instead of ppc64. > > From what I?ve learned so far it should be possible to upgrade directly from 4.1.1 to 4.2.3. Before doing the upgrade we would like to ask for some advice on the best strategy. > > For the NSD servers, one by one, we are thinking about doing the following: > > 1) Disable auto recovery > 2) Unmount GPFS file system > 3) Suspend disks > 4) Shutdown gpfs > 5) Reboot and reinstall with changed architecture ppc64le > 6) Install gpfs 4.2.3 > 7) Recover cluster config using mmsdrrestore > 8) Resume and start disks > 9) Reenable auto recovery > > Can GPFS handle the change of the NSD server?s architecture and would it be fine to operate a mixture of different architectures for the NSD servers? > > > Thanks, > Jan Erik From Philipp.Rehs at uni-duesseldorf.de Thu Jun 8 10:35:57 2017 From: Philipp.Rehs at uni-duesseldorf.de (Philipp Helo Rehs) Date: Thu, 8 Jun 2017 11:35:57 +0200 Subject: [gpfsug-discuss] GPFS for aarch64? Message-ID: <5848d2c0-d526-3d81-a469-6b7a10b9bf3a@uni-duesseldorf.de> Hello, we got a Cavium ThunderX-based Server and would like to use GPFS on it. Are the any package for gpfs on aarch64? Kind regards Philipp Rehs --------------------------- Zentrum f?r Informations- und Medientechnologie Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern Heinrich-Heine-Universit?t D?sseldorf Universit?tsstr. 1 Raum 25.41.00.51 40225 D?sseldorf / Germany Tel: +49-211-81-15557 From abeattie at au1.ibm.com Thu Jun 8 10:45:38 2017 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Thu, 8 Jun 2017 09:45:38 +0000 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: <5848d2c0-d526-3d81-a469-6b7a10b9bf3a@uni-duesseldorf.de> References: <5848d2c0-d526-3d81-a469-6b7a10b9bf3a@uni-duesseldorf.de> Message-ID: An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Thu Jun 8 10:54:15 2017 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 8 Jun 2017 09:54:15 +0000 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: Message-ID: And Linux on Z/VM If interested feel free to open a RFE -- Cheers > On 8 Jun 2017, at 12.46, Andrew Beattie wrote: > > Philipp, > > Not to my knowledge, > > AIX > Linux on x86 ( RHEL / SUSE / Ubuntu) > Linux on Power (RHEL / SUSE) > WIndows > > are the current supported platforms > Andrew Beattie > Software Defined Storage - IT Specialist > Phone: 614-2133-7927 > E-mail: abeattie at au1.ibm.com > > > ----- Original message ----- > From: Philipp Helo Rehs > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [gpfsug-discuss] GPFS for aarch64? > Date: Thu, Jun 8, 2017 7:36 PM > > Hello, > > we got a Cavium ThunderX-based Server and would like to use GPFS on it. > > Are the any package for gpfs on aarch64? > > > Kind regards > > Philipp Rehs > > --------------------------- > > Zentrum f?r Informations- und Medientechnologie > Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern > > Heinrich-Heine-Universit?t D?sseldorf > Universit?tsstr. 1 > Raum 25.41.00.51 > 40225 D?sseldorf / Germany > Tel: +49-211-81-15557 > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From Philipp.Rehs at uni-duesseldorf.de Thu Jun 8 11:40:23 2017 From: Philipp.Rehs at uni-duesseldorf.de (Philipp Helo Rehs) Date: Thu, 8 Jun 2017 12:40:23 +0200 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: References: Message-ID: <9f47c897-74ff-9473-2ab3-343e4ce69d15@uni-duesseldorf.de> Thanks for the Information. I created an RFE: https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=106218 Kind regards, Philipp Rehs > Message: 6 > Date: Thu, 8 Jun 2017 09:54:15 +0000 > From: "Luis Bolinches" > To: "gpfsug main discussion list" > Subject: Re: [gpfsug-discuss] GPFS for aarch64? > Message-ID: > > > Content-Type: text/plain; charset="utf-8" > > And Linux on Z/VM > > If interested feel free to open a RFE > > -- > Cheers > >> On 8 Jun 2017, at 12.46, Andrew Beattie wrote: >> >> Philipp, >> >> Not to my knowledge, >> >> AIX >> Linux on x86 ( RHEL / SUSE / Ubuntu) >> Linux on Power (RHEL / SUSE) >> WIndows >> >> are the current supported platforms >> Andrew Beattie >> Software Defined Storage - IT Specialist >> Phone: 614-2133-7927 >> E-mail: abeattie at au1.ibm.com >> >> >> ----- Original message ----- >> From: Philipp Helo Rehs >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: >> Subject: [gpfsug-discuss] GPFS for aarch64? >> Date: Thu, Jun 8, 2017 7:36 PM >> >> Hello, >> >> we got a Cavium ThunderX-based Server and would like to use GPFS on it. >> >> Are the any package for gpfs on aarch64? >> >> >> Kind regards >> >> Philipp Rehs >> >> --------------------------- >> >> Zentrum f?r Informations- und Medientechnologie >> Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern >> >> Heinrich-Heine-Universit?t D?sseldorf >> Universit?tsstr. 1 >> Raum 25.41.00.51 >> 40225 D?sseldorf / Germany >> Tel: +49-211-81-15557 >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 65, Issue 17 > ********************************************** > From daniel.kidger at uk.ibm.com Thu Jun 8 11:54:04 2017 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 8 Jun 2017 10:54:04 +0000 Subject: [gpfsug-discuss] GPFS for aarch64? In-Reply-To: Message-ID: I often hear requests for Spectrum Scale on ARM. It is always for clients. In general people are happy to have their NSD servers, etc. on x86 or POWER. It is also an anomaly that for a HPC cluster, IBM supports LSF on ARM v7/v8 but not Spectrum Scale on ARM. Daniel Daniel Kidger Technical Sales Specialist, IBM UK IBM Spectrum Storage Software daniel.kidger at uk.ibm.com +44 (0)7818 522266 > On 8 Jun 2017, at 10:54, Luis Bolinches wrote: > > And Linux on Z/VM > > If interested feel free to open a RFE > > -- > Cheers > >> On 8 Jun 2017, at 12.46, Andrew Beattie wrote: >> >> Philipp, >> >> Not to my knowledge, >> >> AIX >> Linux on x86 ( RHEL / SUSE / Ubuntu) >> Linux on Power (RHEL / SUSE) >> WIndows >> >> are the current supported platforms >> Andrew Beattie >> Software Defined Storage - IT Specialist >> Phone: 614-2133-7927 >> E-mail: abeattie at au1.ibm.com >> >> >> ----- Original message ----- >> From: Philipp Helo Rehs >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: >> Subject: [gpfsug-discuss] GPFS for aarch64? >> Date: Thu, Jun 8, 2017 7:36 PM >> >> Hello, >> >> we got a Cavium ThunderX-based Server and would like to use GPFS on it. >> >> Are the any package for gpfs on aarch64? >> >> >> Kind regards >> >> Philipp Rehs >> >> --------------------------- >> >> Zentrum f?r Informations- und Medientechnologie >> Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern >> >> Heinrich-Heine-Universit?t D?sseldorf >> Universit?tsstr. 1 >> Raum 25.41.00.51 >> 40225 D?sseldorf / Germany >> Tel: +49-211-81-15557 >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From duersch at us.ibm.com Thu Jun 8 15:09:03 2017 From: duersch at us.ibm.com (Steve Duersch) Date: Thu, 8 Jun 2017 10:09:03 -0400 Subject: [gpfsug-discuss] Upgrade with architecture change In-Reply-To: References: Message-ID: We have not tested such a procedure. The only route that we have done is a complete mmdelnode/mmaddnode scenario. This would mean an mmdeldisk. It would be more time consuming since data has to move. Operating in a mixed architecture environment is not a problem. We have tested and support that. Steve Duersch Spectrum Scale 845-433-7902 IBM Poughkeepsie, New York > > Message: 1 > Date: Wed, 7 Jun 2017 14:04:28 +0000 > From: "Sundermann, Jan Erik (SCC)" > To: "gpfsug-discuss at spectrumscale.org" > > Subject: [gpfsug-discuss] Upgrade with architecture change > Message-ID: > Content-Type: text/plain; charset="utf-8" > > Hi, > > we are operating a small Spectrum Scale cluster with about 100 > clients and 6 NSD servers. The cluster is FPO-enabled. For > historical reasons the NSD servers are running on ppc64 while the > clients are a mixture of ppc64le and x86_64 machines. Most machines > are running Red Hat Enterprise Linux 7 but we also have few machines > running AIX. > > At the moment we have installed Spectrum Scale version 4.1.1 but > would like to do an upgrade to 4.2.3. In the course of the upgrade > we would like to change the architecture of all NSD servers and > reinstall them with ppc64le instead of ppc64. > > From what I?ve learned so far it should be possible to upgrade > directly from 4.1.1 to 4.2.3. Before doing the upgrade we would like > to ask for some advice on the best strategy. > > For the NSD servers, one by one, we are thinking about doing the following: > > 1) Disable auto recovery > 2) Unmount GPFS file system > 3) Suspend disks > 4) Shutdown gpfs > 5) Reboot and reinstall with changed architecture ppc64le > 6) Install gpfs 4.2.3 > 7) Recover cluster config using mmsdrrestore > 8) Resume and start disks > 9) Reenable auto recovery > > Can GPFS handle the change of the NSD server?s architecture and > would it be fine to operate a mixture of different architectures for > the NSD servers? > > > Thanks, > Jan Erik > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Thu Jun 8 17:01:07 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Thu, 8 Jun 2017 12:01:07 -0400 Subject: [gpfsug-discuss] Upgrade with architecture change In-Reply-To: References: Message-ID: If you proceed carefully, it should not be necessary to mmdeldisk and mmadddisks. Although we may not have tested your exact scenario, GPFS does support fiber channel disks attached to multiple nodes. So the same disk can be attached to multiple GPFS nodes - and those nodes can be running different OSes and different GPFS versions. (That's something we do actually test!) Since GPFS can handle that with several nodes simultaneously active -- it can also handle the case when nodes come and go... Or in your case are killed and then reborn with new software... The key is to be careful... You want to unmount the file system and not re-mount until all of the disks become available again via one or more (NSD) nodes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Thu Jun 8 22:34:15 2017 From: Mark.Bush at siriuscom.com (Mark Bush) Date: Thu, 8 Jun 2017 21:34:15 +0000 Subject: [gpfsug-discuss] LROC/HAWC for CES nodes? Message-ID: <288751C9-7CB6-48E6-968E-938A4E56E786@siriuscom.com> I?m looking to improve performance of the SMB stack. My workload unfortunately has smallish files but in total it will still be large amount. I?m wondering if LROC/HAWC would be one way to speed things up. Is there a precedent for using this with protocol nodes in a cluster? Anyone else thinking/doing this? Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From TROPPENS at de.ibm.com Fri Jun 9 08:44:57 2017 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Fri, 9 Jun 2017 09:44:57 +0200 Subject: [gpfsug-discuss] ISC 2017 - Agenda User Meeting In-Reply-To: References: Message-ID: There is an update of the agenda for the User Meeting at ISC. We have added a Pawsey Site Report by Chris Schlipalius. Monday June 19, 2016 - 12:00-14:30 - Conference Room Konstant 12:00-12:10 ?[10 min] ?Opening 12:10-12:25 ?[15 min] ?Spectrum Scale Support for Docker - Olaf Weiser (IBM) 12:25-13:05 ?[40 min] ?IBM Spectrum LSF family update - Bill McMillan (IBM) 13:05-13:25 ?[20 min] ?Driving Operational Efficiencies with the IBM Spectrum LSF & Ellexus Mistral - Dr. Rosemary Francis (Ellexus) 13:25-13:40 [15 min] Pawsey Site Report - Chris Schlipalius (Pawsey) 13:40-13:55 ?[15 min] ?IBM Elastic Storage Server (ESS) Update - John Sing (IBM) 13:55-14:20 ?[25 min] ?IBM Spectrum Scale Enhancements for CORAL - Sven Oehme (IBM) 14:20-14:30 ?[10 min] ?Question & Answers -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Ulf Troppens" To: gpfsug-discuss at spectrumscale.org Cc: Fabienne Wegener Date: 07.06.2017 10:30 Subject: [gpfsug-discuss] ISC 2017 - Agenda User Meeting Sent by: gpfsug-discuss-bounces at spectrumscale.org Greetings: IBM is happy to announce the agenda for the joint IBM Spectrum Scale and IBM Spectrum LSF User Meeting at ISC. As with other user meetings, the agenda includes user stories, updates on IBM Spectrum Scale and IBM Spectrum LSF, and access to IBM experts and your peers. Please join us! To attend, please email Fabienne.Wegener at de.ibm.com so we can have an accurate count of attendees. Monday June 17, 2016 - 12:00-14:30 - Conference Room Konstant 12:00-12:10 [10 min] Opening 12:10-12:30 [20 min] Spectrum Scale Support for Docker - Olaf Weiser (IBM) 12:30-13:10 [40 min] IBM Spectrum LSF family update - Bill McMillan (IBM) 13:10-13:30 [20 min] Driving Operational Efficiencies with the IBM Spectrum LSF & Ellexus Mistral - Dr. Rosemary Francis (Ellexus) 13:30-13:50 [20 min] IBM Elastic Storage Server (ESS) Update - John Sing (IBM) 13:50-14:20 [30 min] IBM Spectrum Scale Enhancements for CORAL - Sven Oehme (IBM) 14:20-14:30 [10 min] Question & Answers Looking forward to seeing you there! -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Fri Jun 9 09:38:01 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 9 Jun 2017 08:38:01 +0000 Subject: [gpfsug-discuss] LROC/HAWC for CES nodes? In-Reply-To: <288751C9-7CB6-48E6-968E-938A4E56E786@siriuscom.com> References: <288751C9-7CB6-48E6-968E-938A4E56E786@siriuscom.com> Message-ID: I?m wary of spending a lot of money on LROC devices when I don?t know what return I will get.. that said I think the main bottleneck for any SMB installation is samba itself, not the disks, so I remain largely unconvinced that LROC will help much. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mark Bush Sent: 08 June 2017 22:34 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] LROC/HAWC for CES nodes? I?m looking to improve performance of the SMB stack. My workload unfortunately has smallish files but in total it will still be large amount. I?m wondering if LROC/HAWC would be one way to speed things up. Is there a precedent for using this with protocol nodes in a cluster? Anyone else thinking/doing this? Thanks Mark This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Fri Jun 9 10:31:45 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Fri, 9 Jun 2017 09:31:45 +0000 Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS In-Reply-To: References: Message-ID: Thanks Mark, didn?t mean to wait so long to reply. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Marc A Kaplan Sent: 02 June 2017 17:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] TSM/SP compatibility with GPFS Upgrading from GPFS 4.2.x to GPFS 4.2.y should not "break" TSM. If it does, someone goofed, that would be a bug. (My opinion) Think of it this way. TSM is an application that uses the OS and the FileSystem(s). TSM can't verify it will work with all future versions of OS and Filesystems, and the releases can't be in lock step. Having said that, 4.2.3 has been "out" for a while, so if there were a TSM incompatibility, someone would have likely hit it or will before July... Trust but verify... From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 06/02/2017 11:51 AM Subject: [gpfsug-discuss] TSM/SP compatibility with GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi all, Where should I start looking for a compatibility matrix between TSM and GPFS? Specifically, we are currently running TSM 7.1.6-2 and GPFS 4.2.1-2 with the intent to upgrade to GPFS 4.2.3-latest in early July. I?ve spent 30 minutes looking over various documents and the best I can find is this: http://www-01.ibm.com/support/docview.wss?uid=swg21248771 ..which talks about TSM in a Space Management context and would suggest that we need to upgrade to Spectrum Protect i.e. 8.1 and that GPFS 4.2.2.x is the maximum supported version? Cheers Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.tower at outlook.com Sat Jun 10 11:55:54 2017 From: frank.tower at outlook.com (Frank Tower) Date: Sat, 10 Jun 2017 10:55:54 +0000 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found Message-ID: Hi everybody, I don't get why one of our compute node cannot start GPFS over IB. I have the following error: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). [I] VERBS RDMA parse verbsPorts mlx4_0/1 [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found [I] VERBS RDMA library libibverbs.so unloaded. [E] VERBS RDMA failed to start, no valid verbsPorts defined. I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. I have 2 infinibands card, both have an IP and working well. [root at rdx110 ~]# ibstat -l mlx4_0 mlx4_1 [root at rdx110 ~]# I tried configuration with both card, and no one work with GPFS. I also tried with mlx4_0/1, but same problem. Someone already have the issue ? Kind Regards, Frank -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Sat Jun 10 13:05:04 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Sat, 10 Jun 2017 08:05:04 -0400 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found In-Reply-To: References: Message-ID: Out of curiosity could you send us the output of "ibv_devinfo -v"? -Aaron Sent from my iPhone > On Jun 10, 2017, at 06:55, Frank Tower wrote: > > Hi everybody, > > > I don't get why one of our compute node cannot start GPFS over IB. > > > I have the following error: > > > [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes > [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. > [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). > > [I] VERBS RDMA parse verbsPorts mlx4_0/1 > > [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found > > [I] VERBS RDMA library libibverbs.so unloaded. > > [E] VERBS RDMA failed to start, no valid verbsPorts defined. > > > > I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. > > > I have 2 infinibands card, both have an IP and working well. > > > [root at rdx110 ~]# ibstat -l > > mlx4_0 > > mlx4_1 > > [root at rdx110 ~]# > > I tried configuration with both card, and no one work with GPFS. > > > > I also tried with mlx4_0/1, but same problem. > > > > Someone already have the issue ? > > > Kind Regards, > > Frank > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Mon Jun 12 20:41:17 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Mon, 12 Jun 2017 15:41:17 -0400 Subject: [gpfsug-discuss] 'mmces address move' weirdness? Message-ID: <18719.1497296477@turing-police.cc.vt.edu> So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Mon Jun 12 21:01:44 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 12 Jun 2017 20:01:44 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: <18719.1497296477@turing-police.cc.vt.edu> References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of valdis.kletnieks at vt.edu Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Jun 12 21:06:09 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Mon, 12 Jun 2017 20:06:09 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkr at lbl.gov Mon Jun 12 21:17:08 2017 From: kkr at lbl.gov (Kristy Kallback-Rose) Date: Mon, 12 Jun 2017 16:17:08 -0400 Subject: [gpfsug-discuss] Meaning of API Stats Category Message-ID: Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? Category 1, GPFSFileSystemAPI: This metrics gives the following information for each file system (application view). For example: myMachine|GPFSFilesystemAPI|myCluster|myFilesystem|gpfs_fis_bytes_read . gpfs_fis_bytes_read Number of bytes read. gpfs_fis_bytes_written Number of bytes written. gpfs_fis_close_calls Number of close calls. gpfs_fis_disks Number of disks in the file system. gpfs_fis_inodes_written Number of inode updates to disk. gpfs_fis_open_calls Number of open calls. gpfs_fis_read_calls Number of read calls. gpfs_fis_readdir_calls Number of readdir calls. gpfs_fis_write_calls Number of write calls. Category 2, GPFSNodeAPI: This metrics gives the following information for a particular node from its application point of view. For example: myMachine|GPFSNodeAPI|gpfs_is_bytes_read . gpfs_is_bytes_read Number of bytes read. gpfs_is_bytes_written Number of bytes written. gpfs_is_close_calls Number of close calls. gpfs_is_inodes_written Number of inode updates to disk. gpfs_is_open_calls Number of open calls. gpfs_is_readDir_calls Number of readdir calls. gpfs_is_read_calls Number of read calls. gpfs_is_write_calls Number of write calls. Thanks, Kristy -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 12 21:42:47 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 12 Jun 2017 20:42:47 +0000 Subject: [gpfsug-discuss] Meaning of API Stats Category Message-ID: Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Mon Jun 12 22:01:36 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 12 Jun 2017 17:01:36 -0400 Subject: [gpfsug-discuss] Meaning of API Stats Category In-Reply-To: References: Message-ID: Hello Kristy, The GPFSFileSystemAPI and GPFSNodeAPI sensor metrics are from the point of view of "applications" in the sense that they provide stats about I/O requests made to files in GPFS file systems from user level applications using POSIX interfaces like open(), close(), read(), write(), etc. This is in contrast to similarly named sensors without the "API" suffix, like GPFSFilesystem and GPFSNode. Those sensors provide stats about I/O requests made by the GPFS code to NSDs (disks) making up GPFS file systems. The relationship between application I/O and disk I/O might or might not be obvious. Consider some examples. An application that starts sequentially reading a file might, at least initially, cause more disk I/O than expected because GPFS has decided to prefetch data. An application write() might not immediately cause a the writing of disk blocks due to the operation of the pagepool. Ultimately, application write()s might cause twice as much data written to disk due to the replication factor of the file system. Application I/O concerns itself with user data; disk I/O might have to occur to handle the user data and associated file system metadata (like inodes and indirect blocks). The difference between GPFSFileSystemAPI and GPFSNodeAPI: GPFSFileSystemAPI reports stats for application I/O per filesystem per node; GPFSNodeAPI reports application I/O stats per node. Similarly, GPFSFilesystem reports stats for disk I/O per filesystem per node; GPFSNode reports disk I/O stats per node. I hope this helps. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/12/2017 04:43 PM Subject: Re: [gpfsug-discuss] Meaning of API Stats Category Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Jun 12 23:50:44 2017 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 12 Jun 2017 22:50:44 +0000 Subject: [gpfsug-discuss] Meaning of API Stats Category Message-ID: <163FC574-4191-4C20-A4C7-E66DB1868BF3@nuance.com> Can you tell me how LROC plays into this? I?m trying to understand if the difference between gpfs_ns_bytes_read and gpfs_is_bytes_read on a cluster-wide basis reflects the amount of data that is recalled from pagepool+LROC (assuming the majority of the nodes have LROC. Any insight on LROC stats would helpful as well. [cid:image001.png at 01D2E3A4.63CEE1D0] Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of IBM Spectrum Scale Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 4:01 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Meaning of API Stats Category Hello Kristy, The GPFSFileSystemAPI and GPFSNodeAPI sensor metrics are from the point of view of "applications" in the sense that they provide stats about I/O requests made to files in GPFS file systems from user level applications using POSIX interfaces like open(), close(), read(), write(), etc. This is in contrast to similarly named sensors without the "API" suffix, like GPFSFilesystem and GPFSNode. Those sensors provide stats about I/O requests made by the GPFS code to NSDs (disks) making up GPFS file systems. The relationship between application I/O and disk I/O might or might not be obvious. Consider some examples. An application that starts sequentially reading a file might, at least initially, cause more disk I/O than expected because GPFS has decided to prefetch data. An application write() might not immediately cause a the writing of disk blocks due to the operation of the pagepool. Ultimately, application write()s might cause twice as much data written to disk due to the replication factor of the file system. Application I/O concerns itself with user data; disk I/O might have to occur to handle the user data and associated file system metadata (like inodes and indirect blocks). The difference between GPFSFileSystemAPI and GPFSNodeAPI: GPFSFileSystemAPI reports stats for application I/O per filesystem per node; GPFSNodeAPI reports application I/O stats per node. Similarly, GPFSFilesystem reports stats for disk I/O per filesystem per node; GPFSNode reports disk I/O stats per node. I hope this helps. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/12/2017 04:43 PM Subject: Re: [gpfsug-discuss] Meaning of API Stats Category Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 124065 bytes Desc: image001.png URL: From valdis.kletnieks at vt.edu Tue Jun 13 05:21:26 2017 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Tue, 13 Jun 2017 00:21:26 -0400 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: <15827.1497327686@turing-police.cc.vt.edu> On Mon, 12 Jun 2017 20:06:09 -0000, "Simon Thompson (IT Research Support)" said: > mmces node suspend -N > > Is what you want. This will move the address and stop it being assigned one, > otherwise the rebalance will occur. Yeah, I figured that part out. What I couldn't wrap my brain around was what the purpose of 'mmces address move' is if mmsysmon is going to just put it back... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From janfrode at tanso.net Tue Jun 13 05:42:21 2017 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Tue, 13 Jun 2017 04:42:21 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: <15827.1497327686@turing-police.cc.vt.edu> References: <18719.1497296477@turing-police.cc.vt.edu> <15827.1497327686@turing-police.cc.vt.edu> Message-ID: Switch to node affinity policy, and it will stick to where you move it. "mmces address policy node-affinity". -jf tir. 13. jun. 2017 kl. 06.21 skrev : > On Mon, 12 Jun 2017 20:06:09 -0000, "Simon Thompson (IT Research Support)" > said: > > > mmces node suspend -N > > > > Is what you want. This will move the address and stop it being assigned > one, > > otherwise the rebalance will occur. > > Yeah, I figured that part out. What I couldn't wrap my brain around was > what the purpose of 'mmces address move' is if mmsysmon is going to just > put it back... > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Jun 13 09:08:52 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 13 Jun 2017 08:08:52 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: Yes, suspending the node would do it, but in the case where you want to remove a node from service but keep it running for testing it's not ideal. I think you can set the IP address balancing policy to none which might do what we want. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Jun 13 09:12:13 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 13 Jun 2017 08:12:13 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> <15827.1497327686@turing-police.cc.vt.edu> Message-ID: Or this ? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: 13 June 2017 05:42 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Switch to node affinity policy, and it will stick to where you move it. "mmces address policy node-affinity". -jf tir. 13. jun. 2017 kl. 06.21 skrev >: On Mon, 12 Jun 2017 20:06:09 -0000, "Simon Thompson (IT Research Support)" said: > mmces node suspend -N > > Is what you want. This will move the address and stop it being assigned one, > otherwise the rebalance will occur. Yeah, I figured that part out. What I couldn't wrap my brain around was what the purpose of 'mmces address move' is if mmsysmon is going to just put it back... _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Tue Jun 13 09:28:25 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Tue, 13 Jun 2017 08:28:25 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: Suspending the node doesn't stop the services though, we've done a bunch of testing by connecting to the "real" IP on the box we wanted to test and that works fine. OK, so you end up connecting to shares like \\192.168.1.20\sharename, but its perfectly fine for testing purposes. In our experience, suspending the node has been fine for this as it moves the IP to a "working" node and keeps user service running whilst we test. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 13 June 2017 at 09:08 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Yes, suspending the node would do it, but in the case where you want to remove a node from service but keep it running for testing it?s not ideal. I think you can set the IP address balancing policy to none which might do what we want. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From:gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Tue Jun 13 09:30:18 2017 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 13 Jun 2017 08:30:18 +0000 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: Oh? Nice to know - thanks - will try that method next. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 13 June 2017 09:28 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Suspending the node doesn't stop the services though, we've done a bunch of testing by connecting to the "real" IP on the box we wanted to test and that works fine. OK, so you end up connecting to shares like \\192.168.1.20\sharename, but its perfectly fine for testing purposes. In our experience, suspending the node has been fine for this as it moves the IP to a "working" node and keeps user service running whilst we test. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Tuesday, 13 June 2017 at 09:08 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? Yes, suspending the node would do it, but in the case where you want to remove a node from service but keep it running for testing it's not ideal. I think you can set the IP address balancing policy to none which might do what we want. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? mmces node suspend -N Is what you want. This will move the address and stop it being assigned one, otherwise the rebalance will occur. I think you can change the way it balances, but the default is to distribute. Simon From: > on behalf of "Sobey, Richard A" > Reply-To: "gpfsug-discuss at spectrumscale.org" > Date: Monday, 12 June 2017 at 21:01 To: "gpfsug-discuss at spectrumscale.org" > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? I think it's intended but I don't know why. The AUTH service became unhealthy on one of our CES nodes (SMB only) and we moved its float address elsewhere. CES decided to move it back again moments later despite the node not being fit. Sorry that doesn't really help but at least you're not alone! ________________________________ From:gpfsug-discuss-bounces at spectrumscale.org > on behalf of valdis.kletnieks at vt.edu > Sent: 12 June 2017 20:41 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] 'mmces address move' weirdness? So here's our address setup: mmces address list Address Node Group Attribute ------------------------------------------------------------------------- 172.28.45.72 arproto1.ar.nis.isb.internal isb none 172.28.45.73 arproto2.ar.nis.isb.internal isb none 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to move the address over to its pair so I can look around without impacting users. However, seems like something insists on moving it right back 60 seconds later... Question 1: Is this expected behavior? Question 2: If it is, what use is 'mmces address move' if it just gets undone a few seconds later... (running on arproto2.ar.nis.vtc.internal): ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 Mon Jun 12 15:34:42 EDT 2017 Mon Jun 12 15:34:43 EDT 2017 (skipped) Mon Jun 12 15:35:44 EDT 2017 Mon Jun 12 15:35:45 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:46 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 Mon Jun 12 15:35:47 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 ^C -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Tue Jun 13 14:36:30 2017 From: john.hearns at asml.com (John Hearns) Date: Tue, 13 Jun 2017 13:36:30 +0000 Subject: [gpfsug-discuss] Infiniband Quality of Service settings? Message-ID: I am investigating setting up Quality of Service parameters on an Infiniband fabric. The specific goal is to reduce the bandwidth which certain servers can use, ie if there are untested or development codes running on these servers in our cluster then they cannot adversely affect production users. I hope I do not show too much of my ignorance here. Perhaps out of date, but I find that Lustre does have a facility for setting the port range and hence associating with an ULP in Infiniband http://www.spinics.net/lists/linux-rdma/msg02150.html https://community.mellanox.com/thread/3660 (There. I said the L word. Is a quick soaping to the mouth needed?) Can anyone comment what the Infiniband Service ID for GPFS traffic is please? If the answer is blindingly obvious and is displayed by a Bat signal in the clouds above every datacenter containing GPFS then I am suitably apologetic. If it is buried in a footnote in a Redbook then a bit less apologetic. If you are familiar with Appendix A of the IBTA Architecture Release then it is truly a joy. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Tue Jun 13 15:10:43 2017 From: john.hearns at asml.com (John Hearns) Date: Tue, 13 Jun 2017 14:10:43 +0000 Subject: [gpfsug-discuss] Infiniband Quality of Service settings? In-Reply-To: References: Message-ID: Having the bad manners to answer my own question: Example If you define a service level of 2 for GPFS in the InfiniBand subnet manager set verbsRdmaQpRtrSl to 2. mmchconfig verbsRdmaQpRtrSl=2 I guess though that I still need the service ID to set the Service Level in qos-policy.conf From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of John Hearns Sent: Tuesday, June 13, 2017 3:37 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Infiniband Quality of Service settings? I am investigating setting up Quality of Service parameters on an Infiniband fabric. The specific goal is to reduce the bandwidth which certain servers can use, ie if there are untested or development codes running on these servers in our cluster then they cannot adversely affect production users. I hope I do not show too much of my ignorance here. Perhaps out of date, but I find that Lustre does have a facility for setting the port range and hence associating with an ULP in Infiniband http://www.spinics.net/lists/linux-rdma/msg02150.html https://community.mellanox.com/thread/3660 (There. I said the L word. Is a quick soaping to the mouth needed?) Can anyone comment what the Infiniband Service ID for GPFS traffic is please? If the answer is blindingly obvious and is displayed by a Bat signal in the clouds above every datacenter containing GPFS then I am suitably apologetic. If it is buried in a footnote in a Redbook then a bit less apologetic. If you are familiar with Appendix A of the IBTA Architecture Release then it is truly a joy. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From SAnderson at convergeone.com Tue Jun 13 17:31:31 2017 From: SAnderson at convergeone.com (Shaun Anderson) Date: Tue, 13 Jun 2017 16:31:31 +0000 Subject: [gpfsug-discuss] Difference between mmcesnfscrexport and 'mmnfs export add' commands. Message-ID: <2990f67cded849e8b82a4c5d2ac50d5c@NACR502.nacr.com> ?I see both of these, but only the mmnfs command is documented. Is one a wrapper of the other? SHAUN ANDERSON STORAGE ARCHITECT O 208.577.2112 M 214.263.7014 NOTICE: This email message and any attachments here to may contain confidential information. Any unauthorized review, use, disclosure, or distribution of such information is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy the original message and all copies of it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Wed Jun 14 01:50:24 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 13 Jun 2017 20:50:24 -0400 Subject: [gpfsug-discuss] Meaning of API Stats Category In-Reply-To: <163FC574-4191-4C20-A4C7-E66DB1868BF3@nuance.com> References: <163FC574-4191-4C20-A4C7-E66DB1868BF3@nuance.com> Message-ID: Hello Bob, Right. Within some observation interval, bytes read by an application will be reflected in gpfs_is_bytes_read, regardless of how the byte values were obtained (by reading from "disk", fetching from pagepool, or fetching from LROC). gpfs_ns_bytes_read is only going to reflect bytes read from "disk" within that observation interval. "mmdiag --lroc" provides some LROC stats. There is also a GPFSLROC sensor; it does not appear to be documented at this point, so I hope I haven't spoken out of turn. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Cc: "scale at us.ibm.com" Date: 06/12/2017 06:50 PM Subject: Re: Meaning of API Stats Category Can you tell me how LROC plays into this? I?m trying to understand if the difference between gpfs_ns_bytes_read and gpfs_is_bytes_read on a cluster-wide basis reflects the amount of data that is recalled from pagepool+LROC (assuming the majority of the nodes have LROC. Any insight on LROC stats would helpful as well. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of IBM Spectrum Scale Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 4:01 PM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Meaning of API Stats Category Hello Kristy, The GPFSFileSystemAPI and GPFSNodeAPI sensor metrics are from the point of view of "applications" in the sense that they provide stats about I/O requests made to files in GPFS file systems from user level applications using POSIX interfaces like open(), close(), read(), write(), etc. This is in contrast to similarly named sensors without the "API" suffix, like GPFSFilesystem and GPFSNode. Those sensors provide stats about I/O requests made by the GPFS code to NSDs (disks) making up GPFS file systems. The relationship between application I/O and disk I/O might or might not be obvious. Consider some examples. An application that starts sequentially reading a file might, at least initially, cause more disk I/O than expected because GPFS has decided to prefetch data. An application write() might not immediately cause a the writing of disk blocks due to the operation of the pagepool. Ultimately, application write()s might cause twice as much data written to disk due to the replication factor of the file system. Application I/O concerns itself with user data; disk I/O might have to occur to handle the user data and associated file system metadata (like inodes and indirect blocks). The difference between GPFSFileSystemAPI and GPFSNodeAPI: GPFSFileSystemAPI reports stats for application I/O per filesystem per node; GPFSNodeAPI reports application I/O stats per node. Similarly, GPFSFilesystem reports stats for disk I/O per filesystem per node; GPFSNode reports disk I/O stats per node. I hope this helps. Eric Agar Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 06/12/2017 04:43 PM Subject: Re: [gpfsug-discuss] Meaning of API Stats Category Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Kristy What I *think* the difference is: gpfs_fis: - calls to the GPFS file system interface gpfs_fs: calls from the node that actually make it to the NSD server/metadata The difference being what?s served out of the local node pagepool. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Kristy Kallback-Rose Reply-To: gpfsug main discussion list Date: Monday, June 12, 2017 at 3:17 PM To: gpfsug main discussion list Subject: [EXTERNAL] [gpfsug-discuss] Meaning of API Stats Category Hi, Can anyone provide more detail about what is meant by the following two categories of stats? The PDG has a limited description as far as I could see. I'm not sure what is meant by Application PoV. Would the Grafana bridge count as an "application"? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 124065 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jun 14 17:11:33 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 14 Jun 2017 16:11:33 +0000 Subject: [gpfsug-discuss] 4.2.3.x and sub-block size Message-ID: Hi All, Back at SC16 I was told that GPFS 4.2.3.x would remove the ?a sub-block is 1/32nd of the block size? restriction. However, I have installed GPFS 4.2.3.1 on my test cluster and in the man page for mmcrfs I still see: 2. The GPFS block size determines: * The minimum disk space allocation unit. The minimum amount of space that file data can occupy is a sub?block. A sub?block is 1/32 of the block size. So has the restriction been removed? If not, is there an update on which version of GPFS will remove it? If so, can the documentation be updated to reflect the change and how to take advantage of it? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kums at us.ibm.com Wed Jun 14 18:15:27 2017 From: kums at us.ibm.com (Kumaran Rajaram) Date: Wed, 14 Jun 2017 13:15:27 -0400 Subject: [gpfsug-discuss] 4.2.3.x and sub-block size In-Reply-To: References: Message-ID: Hi, >>Back at SC16 I was told that GPFS 4.2.3.x would remove the ?a sub-block is 1/32nd of the block size? restriction. However, I have installed GPFS 4.2.3.1 on my test cluster and in the man page for mmcrfs I still see: >>So has the restriction been removed? If not, is there an update on which version of GPFS will remove it? If so, can the documentation be updated to reflect the change and how to take advantage of it? Thanks? Based on the current plan, this ?a sub-block is 1/32nd of the block size? restriction will be removed in the upcoming GPFS version 4.2.4 (Please NOTE: Support for >32 subblocks per block may subject to be delayed based on internal qualification/validation efforts). Regards, -Kums From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/14/2017 12:12 PM Subject: [gpfsug-discuss] 4.2.3.x and sub-block size Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, Back at SC16 I was told that GPFS 4.2.3.x would remove the ?a sub-block is 1/32nd of the block size? restriction. However, I have installed GPFS 4.2.3.1 on my test cluster and in the man page for mmcrfs I still see: 2. The GPFS block size determines: * The minimum disk space allocation unit. The minimum amount of space that file data can occupy is a sub?block. A sub?block is 1/32 of the block size. So has the restriction been removed? If not, is there an update on which version of GPFS will remove it? If so, can the documentation be updated to reflect the change and how to take advantage of it? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Schlipalius at pawsey.org.au Thu Jun 15 03:30:27 2017 From: Chris.Schlipalius at pawsey.org.au (Chris Schlipalius) Date: Thu, 15 Jun 2017 10:30:27 +0800 Subject: [gpfsug-discuss] Perth Australia - Spectrum Scale User Group event in August 2017 announced - Pawsey Supercomputing Centre Message-ID: Hi please find the eventbrite link (text as http/s links are usually stripped). www.eventbrite.com/e/spectrum-scale-user-group-perth-australia-gpfsugaus-au gust-2017-tickets-35227460282 Please register and let me know if you are keen to present. I have a special group booking offer on accomodation for attendees, well below usually rack rack. I will announce this Usergroup meeting on spectrumscle.org shortly. This event is on the same week and at the same location as HPC Advisory Council also being held in Perth Australia. (Call for papers is now out - I can supply the HPC AC invite separately if you wish to email me directly). If you want to know more in person and you are at ISC2017 next week I will be at the Spectrum Scale Usergroup that Ulf announced or you can catch me on the Pawsey Supercomputing Centre booth. Regards, Chris Schlipalius Senior Storage Infrastructure Specialist/Team Leader Pawsey Supercomputing Centre 12 Burvill Court Kensington WA 6151 Australia Tel +61 8 6436 8815 Email chris.schlipalius at pawsey.org.au Web www.pawsey.org.au From Kevin.Buterbaugh at Vanderbilt.Edu Thu Jun 15 21:00:47 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Thu, 15 Jun 2017 20:00:47 +0000 Subject: [gpfsug-discuss] SAN problem ... multipathd ... mmunlinkfileset ... ??? Message-ID: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> Hi All, I?ve got some very weird problems going on here (and I do have a PMR open with IBM). On Monday I attempted to unlink a fileset, something that I?ve done many times with no issues. This time, however, it hung up the filesystem. I was able to clear things up by shutting down GPFS on the filesystem manager for that filesystem and restarting it. The very next morning we awoke to problems with GPFS. I noticed in my messages file on all my NSD servers I had messages like: Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Write Protect is off Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Asking for cache data failed Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Assuming drive cache: write through Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Attached SCSI disk Jun 12 22:03:32 nsd32 multipathd: sdab: add path (uevent) Jun 12 22:03:32 nsd32 multipathd: sdab: failed to get path uid Jun 12 22:03:32 nsd32 multipathd: uevent trigger error Jun 12 22:03:42 nsd32 kernel: rport-0:0-4: blocked FC remote port time out: removing target and saving binding Since we use an FC SAN and Linux multi-pathing I was expecting some sort of problem with the switches. Now on the switches I see messages like: [114][Thu Jun 15 19:02:05.411 UTC 2017][I][8600.0020][Port][Port: 9][SYNC_LOSS] [115][Thu Jun 15 19:03:49.988 UTC 2017][I][8600.001F][Port][Port: 9][SYNC_ACQ] Which (while not in this example) do correlate time-wise with the multi path messages on the servers. So it?s not a GPFS problem and I shouldn?t be bugging this list about this EXCEPT? These issues only started on Monday after I ran the mmunlinkfileset command. That?s right ? NO such errors prior to then. And literally NOTHING changed on Monday with my SAN environment (nothing had changed there for months actually). Nothing added to nor removed from the SAN. No changes until today when, in an attempt to solve this issue, I updated the switch firmware on all switches one at a time. I also yum updated to the latest RHEL 7 version of the multipathd packages. I?ve been Googling and haven?t found anything useful on those SYNC_LOSS messages on the QLogic SANbox 5800 switches. Anybody out there happen to have any knowledge of them and what could be causing them? Oh, I?m investigating this now ? but it?s not all ports that are throwing the errors. And the ports that are seem to be random and don?t have one specific type of hardware plugged in ? i.e. some ports have NSD servers plugged in, others have storage arrays. I understand that it makes no sense that mmunlinkfileset hanging would cause problems with my SAN ? but I also don?t believe in coincidences! I?m running GPFS 4.2.2.3. Any help / suggestions apprecaiated! ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Thu Jun 15 21:50:10 2017 From: ewahl at osc.edu (Edward Wahl) Date: Thu, 15 Jun 2017 16:50:10 -0400 Subject: [gpfsug-discuss] SAN problem ... multipathd ... mmunlinkfileset ... ??? In-Reply-To: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> References: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> Message-ID: <20170615165010.6241c6d3@osc.edu> On Thu, 15 Jun 2017 20:00:47 +0000 "Buterbaugh, Kevin L" wrote: > Hi All, > > I?ve got some very weird problems going on here (and I do have a PMR open > with IBM). On Monday I attempted to unlink a fileset, something that I?ve > done many times with no issues. This time, however, it hung up the > filesystem. I was able to clear things up by shutting down GPFS on the > filesystem manager for that filesystem and restarting it. > > The very next morning we awoke to problems with GPFS. I noticed in my > messages file on all my NSD servers I had messages like: > > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Write Protect is off > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Asking for cache data failed > Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Assuming drive cache: write > through Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline > device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Attached SCSI disk > Jun 12 22:03:32 nsd32 multipathd: sdab: add path (uevent) > Jun 12 22:03:32 nsd32 multipathd: sdab: failed to get path uid > Jun 12 22:03:32 nsd32 multipathd: uevent trigger error > Jun 12 22:03:42 nsd32 kernel: rport-0:0-4: blocked FC remote port time out: > removing target and saving binding > > Since we use an FC SAN and Linux multi-pathing I was expecting some sort of > problem with the switches. Now on the switches I see messages like: > > [114][Thu Jun 15 19:02:05.411 UTC 2017][I][8600.0020][Port][Port: > 9][SYNC_LOSS] [115][Thu Jun 15 19:03:49.988 UTC > 2017][I][8600.001F][Port][Port: 9][SYNC_ACQ] > > Which (while not in this example) do correlate time-wise with the multi path > messages on the servers. So it?s not a GPFS problem and I shouldn?t be > bugging this list about this EXCEPT? > > These issues only started on Monday after I ran the mmunlinkfileset command. > That?s right ? NO such errors prior to then. And literally NOTHING changed > on Monday with my SAN environment (nothing had changed there for months > actually). Nothing added to nor removed from the SAN. No changes until > today when, in an attempt to solve this issue, I updated the switch firmware > on all switches one at a time. I also yum updated to the latest RHEL 7 > version of the multipathd packages. > > I?ve been Googling and haven?t found anything useful on those SYNC_LOSS > messages on the QLogic SANbox 5800 switches. Anybody out there happen to > have any knowledge of them and what could be causing them? Oh, I?m > investigating this now ? but it?s not all ports that are throwing the > errors. And the ports that are seem to be random and don?t have one specific > type of hardware plugged in ? i.e. some ports have NSD servers plugged in, > others have storage arrays. I have a half dozen of the Sanbox 5802 switches, but no GPFS devices going through them any longer. Used to though. We do see that exact same messages when the FC interface on a device goes bad (SFP, HCA, etc) or someone moving cables. This happens when the device cannot properly join the loop with it's login. I've NEVER seen them randomly though. Nor has this been a bad cable type error. I don't recall why, but I froze our Sanbox's at : V7.4.0.16.0 I'm sure I have notes on it somewhere. I've got one right now, in fact, with a bad ancient LTO4 drive. [8124][Thu Jun 15 12:46:00.190 EDT 2017][I][8600.001F][Port][Port: 4][SYNC_ACQ] [8125][Thu Jun 15 12:49:20.920 EDT 2017][I][8600.0020][Port][Port: 4][SYNC_LOSS] Sounds like the sanbox itself is having an issue perhaps? "Show alarm" clean on the sanbox? Array has a bad HCA? 'Show port 9' errors not crazy? All power supplies working? > I understand that it makes no sense that mmunlinkfileset hanging would cause > problems with my SAN ? but I also don?t believe in coincidences! > > I?m running GPFS 4.2.2.3. Any help / suggestions apprecaiated! This does seem like QUITE the coincidence. Increased traffic on the device triggered a failure? (The fear of all RAID users!) Multipath is working properly though? Sounds like mmlsdisk would have shown devices not in 'ready'. We mysteriously lost a MD disk during a recent downtime and it caused an MMFSCK to not run properly until we figured it out. 4.2.2.3 as well. MD Replication is NOT helpful in that case. Ed > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu - > (615)875-9633 > > > -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From Kevin.Buterbaugh at Vanderbilt.Edu Thu Jun 15 22:14:33 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Thu, 15 Jun 2017 21:14:33 +0000 Subject: [gpfsug-discuss] SAN problem ... multipathd ... mmunlinkfileset ... ??? In-Reply-To: <20170615165010.6241c6d3@osc.edu> References: <35CB524D-E657-4006-8689-833127720023@vanderbilt.edu> <20170615165010.6241c6d3@osc.edu> Message-ID: <91D4DDD0-BF4C-4F73-B369-A91C032B0FCD@vanderbilt.edu> Hi Ed, others, I have spent the intervening time since sending my original e-mail taking the logs from the SAN switches and putting them into text files where they can be sorted and grep?d ? and something potentially interesting has come to light ? While there are a number of ports on all switches that have one or two SYNC_LOSS errors on them, on two of the switches port 9 has dozens of SYNC_LOSS errors (looking at the raw logs with other messages interspersed that wasn?t obvious). Turns out that one particular dual-controller storage array is plugged into those ports and - in a stroke of good luck which usually manages to avoid me - that particular storage array is no longer in use! It, and a few others still in use, are older and about to be life-cycled. Since it?s no longer in use, I have unplugged it from the SAN and am monitoring to see if my problems now go away. Yes, correlation is not causation. And sometimes coincidences do happen. I?ll monitor to see if this is one of those occasions. Thanks? Kevin > On Jun 15, 2017, at 3:50 PM, Edward Wahl wrote: > > On Thu, 15 Jun 2017 20:00:47 +0000 > "Buterbaugh, Kevin L" wrote: > >> Hi All, >> >> I?ve got some very weird problems going on here (and I do have a PMR open >> with IBM). On Monday I attempted to unlink a fileset, something that I?ve >> done many times with no issues. This time, however, it hung up the >> filesystem. I was able to clear things up by shutting down GPFS on the >> filesystem manager for that filesystem and restarting it. >> >> The very next morning we awoke to problems with GPFS. I noticed in my >> messages file on all my NSD servers I had messages like: >> >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Write Protect is off >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline device >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Asking for cache data failed >> Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Assuming drive cache: write >> through Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: rejecting I/O to offline >> device Jun 12 22:03:32 nsd32 kernel: sd 0:0:4:2: [sdab] Attached SCSI disk >> Jun 12 22:03:32 nsd32 multipathd: sdab: add path (uevent) >> Jun 12 22:03:32 nsd32 multipathd: sdab: failed to get path uid >> Jun 12 22:03:32 nsd32 multipathd: uevent trigger error >> Jun 12 22:03:42 nsd32 kernel: rport-0:0-4: blocked FC remote port time out: >> removing target and saving binding >> >> Since we use an FC SAN and Linux multi-pathing I was expecting some sort of >> problem with the switches. Now on the switches I see messages like: >> >> [114][Thu Jun 15 19:02:05.411 UTC 2017][I][8600.0020][Port][Port: >> 9][SYNC_LOSS] [115][Thu Jun 15 19:03:49.988 UTC >> 2017][I][8600.001F][Port][Port: 9][SYNC_ACQ] >> >> Which (while not in this example) do correlate time-wise with the multi path >> messages on the servers. So it?s not a GPFS problem and I shouldn?t be >> bugging this list about this EXCEPT? >> >> These issues only started on Monday after I ran the mmunlinkfileset command. >> That?s right ? NO such errors prior to then. And literally NOTHING changed >> on Monday with my SAN environment (nothing had changed there for months >> actually). Nothing added to nor removed from the SAN. No changes until >> today when, in an attempt to solve this issue, I updated the switch firmware >> on all switches one at a time. I also yum updated to the latest RHEL 7 >> version of the multipathd packages. >> >> I?ve been Googling and haven?t found anything useful on those SYNC_LOSS >> messages on the QLogic SANbox 5800 switches. Anybody out there happen to >> have any knowledge of them and what could be causing them? Oh, I?m >> investigating this now ? but it?s not all ports that are throwing the >> errors. And the ports that are seem to be random and don?t have one specific >> type of hardware plugged in ? i.e. some ports have NSD servers plugged in, >> others have storage arrays. > > I have a half dozen of the Sanbox 5802 switches, but no GPFS devices going > through them any longer. Used to though. We do see that exact same messages > when the FC interface on a device goes bad (SFP, HCA, etc) or someone moving > cables. This happens when the device cannot properly join the loop with it's > login. I've NEVER seen them randomly though. Nor has this been a bad cable type > error. I don't recall why, but I froze our Sanbox's at : V7.4.0.16.0 I'm sure > I have notes on it somewhere. > > I've got one right now, in fact, with a bad ancient LTO4 drive. > [8124][Thu Jun 15 12:46:00.190 EDT 2017][I][8600.001F][Port][Port: 4][SYNC_ACQ] > [8125][Thu Jun 15 12:49:20.920 EDT 2017][I][8600.0020][Port][Port: 4][SYNC_LOSS] > > > Sounds like the sanbox itself is having an issue perhaps? "Show alarm" clean on > the sanbox? Array has a bad HCA? 'Show port 9' errors not crazy? All power > supplies working? > > > >> I understand that it makes no sense that mmunlinkfileset hanging would cause >> problems with my SAN ? but I also don?t believe in coincidences! >> >> I?m running GPFS 4.2.2.3. Any help / suggestions apprecaiated! > > This does seem like QUITE the coincidence. Increased traffic on the > device triggered a failure? (The fear of all RAID users!) Multipath is working > properly though? Sounds like mmlsdisk would have shown devices not in 'ready'. > We mysteriously lost a MD disk during a recent downtime and it caused an MMFSCK > to not run properly until we figured it out. 4.2.2.3 as well. MD Replication > is NOT helpful in that case. > > Ed > > > >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and Education >> Kevin.Buterbaugh at vanderbilt.edu - >> (615)875-9633 >> >> >> > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 From frank.tower at outlook.com Sun Jun 18 05:57:57 2017 From: frank.tower at outlook.com (Frank Tower) Date: Sun, 18 Jun 2017 04:57:57 +0000 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found In-Reply-To: References: , Message-ID: Hi, You were right, ibv_devinfo -v doesn't return something if both card are connected. I didn't checked ibv_* tools, I supposed once IP stack and ibstat OK, the rest should work. I'm stupid ? Anyway, once I disconnect one card, ibv_devinfo show me input but with both cards, I don't have any input except "device not found". And what is weird here, it's that it work only when one card are connected, no matter the card (both are similar: model, firmware, revision, company)... Really strange, I will dig more about the issue. Stupid and bad workaround: connected a dual port Infiniband. But production system doesn't wait.. Thank for your help, Frank ________________________________ From: Aaron Knister Sent: Saturday, June 10, 2017 2:05 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Out of curiosity could you send us the output of "ibv_devinfo -v"? -Aaron Sent from my iPhone On Jun 10, 2017, at 06:55, Frank Tower > wrote: Hi everybody, I don't get why one of our compute node cannot start GPFS over IB. I have the following error: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). [I] VERBS RDMA parse verbsPorts mlx4_0/1 [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found [I] VERBS RDMA library libibverbs.so unloaded. [E] VERBS RDMA failed to start, no valid verbsPorts defined. I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. I have 2 infinibands card, both have an IP and working well. [root at rdx110 ~]# ibstat -l mlx4_0 mlx4_1 [root at rdx110 ~]# I tried configuration with both card, and no one work with GPFS. I also tried with mlx4_0/1, but same problem. Someone already have the issue ? Kind Regards, Frank _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.tower at outlook.com Sun Jun 18 06:06:30 2017 From: frank.tower at outlook.com (Frank Tower) Date: Sun, 18 Jun 2017 05:06:30 +0000 Subject: [gpfsug-discuss] Protocol node: active directory authentication Message-ID: Hi, We finally received protocols node following the recommendations some here provided and the help of the wiki. Now we would like to use kerberized NFS, we dig into spectrumscale documentations and wiki but we would like to know if anyone is using such configuration ? Do you also have any performance issue (vs NFSv4/NFSv3 with sec=sys) ? We will also use Microsoft Active Directory and we are willing to populate all our users with UID/GID, summer is coming, we will have some spare time ? Someone is using kerberized NFSv4 with Microsoft Active Directory ? Thank by advance for your feedback. Kind Regards, Frank. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcatana at gmail.com Sun Jun 18 16:30:55 2017 From: jcatana at gmail.com (Josh Catana) Date: Sun, 18 Jun 2017 11:30:55 -0400 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found In-Reply-To: References: Message-ID: Are any cards VPI that can do both eth and ib? I remember reading in documentation that that there is a bus order to having mixed media with mellanox cards. There is a module setting during init where you can set eth ib or auto detect. If the card is on auto it might be coming up eth and making the driver flake out because it's in the wrong order. Responding from my phone so I can't really look it up myself right now about what the proper order is, but maybe this might be some help troubleshooting. On Jun 18, 2017 12:58 AM, "Frank Tower" wrote: > Hi, > > > You were right, ibv_devinfo -v doesn't return something if both card are > connected. I didn't checked ibv_* tools, I supposed once IP stack and > ibstat OK, the rest should work. I'm stupid ? > > > Anyway, once I disconnect one card, ibv_devinfo show me input but with > both cards, I don't have any input except "device not found". > > And what is weird here, it's that it work only when one card are > connected, no matter the card (both are similar: model, firmware, revision, > company)... Really strange, I will dig more about the issue. > > > Stupid and bad workaround: connected a dual port Infiniband. But > production system doesn't wait.. > > > Thank for your help, > Frank > > ------------------------------ > *From:* Aaron Knister > *Sent:* Saturday, June 10, 2017 2:05 PM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found > > Out of curiosity could you send us the output of "ibv_devinfo -v"? > > -Aaron > > Sent from my iPhone > > On Jun 10, 2017, at 06:55, Frank Tower wrote: > > Hi everybody, > > > I don't get why one of our compute node cannot start GPFS over IB. > > > I have the following error: > > > [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no > verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes > > [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and > initialized. > > [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match > (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). > > [I] VERBS RDMA parse verbsPorts mlx4_0/1 > > [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device > mlx4_0 not found > > [I] VERBS RDMA library libibverbs.so unloaded. > > [E] VERBS RDMA failed to start, no valid verbsPorts defined. > > > > I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. > > > I have 2 infinibands card, both have an IP and working well. > > > [root at rdx110 ~]# ibstat -l > > mlx4_0 > > mlx4_1 > > [root at rdx110 ~]# > > > I tried configuration with both card, and no one work with GPFS. > > > I also tried with mlx4_0/1, but same problem. > > > Someone already have the issue ? > > > Kind Regards, > > Frank > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Sun Jun 18 18:53:28 2017 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Sun, 18 Jun 2017 17:53:28 +0000 Subject: [gpfsug-discuss] Infiniband: device mlx4_0 not found Message-ID: There used to be issues with the CX-3 cards and specific ports for if you wanted to use IB and Eth, but that went away in later firmwares, as did a whole load of bits with it being slow to detect media type, so see if you are running an up to date Mellanox firmware (assuming it's a VPI card). On CX-4 there is no auto detect media, but default is IB unless you changed it. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of jcatana at gmail.com [jcatana at gmail.com] Sent: 18 June 2017 16:30 To: gpfsug main discussion list Subject: ?spam? Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Are any cards VPI that can do both eth and ib? I remember reading in documentation that that there is a bus order to having mixed media with mellanox cards. There is a module setting during init where you can set eth ib or auto detect. If the card is on auto it might be coming up eth and making the driver flake out because it's in the wrong order. Responding from my phone so I can't really look it up myself right now about what the proper order is, but maybe this might be some help troubleshooting. On Jun 18, 2017 12:58 AM, "Frank Tower" > wrote: Hi, You were right, ibv_devinfo -v doesn't return something if both card are connected. I didn't checked ibv_* tools, I supposed once IP stack and ibstat OK, the rest should work. I'm stupid ? Anyway, once I disconnect one card, ibv_devinfo show me input but with both cards, I don't have any input except "device not found". And what is weird here, it's that it work only when one card are connected, no matter the card (both are similar: model, firmware, revision, company)... Really strange, I will dig more about the issue. Stupid and bad workaround: connected a dual port Infiniband. But production system doesn't wait.. Thank for your help, Frank ________________________________ From: Aaron Knister > Sent: Saturday, June 10, 2017 2:05 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Out of curiosity could you send us the output of "ibv_devinfo -v"? -Aaron Sent from my iPhone On Jun 10, 2017, at 06:55, Frank Tower > wrote: Hi everybody, I don't get why one of our compute node cannot start GPFS over IB. I have the following error: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). [I] VERBS RDMA parse verbsPorts mlx4_0/1 [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found [I] VERBS RDMA library libibverbs.so unloaded. [E] VERBS RDMA failed to start, no valid verbsPorts defined. I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. I have 2 infinibands card, both have an IP and working well. [root at rdx110 ~]# ibstat -l mlx4_0 mlx4_1 [root at rdx110 ~]# I tried configuration with both card, and no one work with GPFS. I also tried with mlx4_0/1, but same problem. Someone already have the issue ? Kind Regards, Frank _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From eric.wonderley at vt.edu Tue Jun 20 17:03:40 2017 From: eric.wonderley at vt.edu (J. Eric Wonderley) Date: Tue, 20 Jun 2017 12:03:40 -0400 Subject: [gpfsug-discuss] gui related connection fail in gpfs logs Message-ID: These type messages repeat often in our logs: 017-06-20_09:25:13.676-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_09:25:24.292-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_10:00:25.935-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 Is there any way to tell if it is a misconfiguration or communications issue? -------------- next part -------------- An HTML attachment was scrubbed... URL: From NSCHULD at de.ibm.com Wed Jun 21 08:12:36 2017 From: NSCHULD at de.ibm.com (Norbert Schuld) Date: Wed, 21 Jun 2017 09:12:36 +0200 Subject: [gpfsug-discuss] gui related connection fail in gpfs logs In-Reply-To: References: Message-ID: This happens if a the mmhealth system of a node can not forward an event to the GUI - typically on some other node. Resons could be: - Gui is not running - Firewall on used port 80 for older versions of spectrum scale or 443 for newer Mit freundlichen Gr??en / Kind regards Norbert Schuld Dr. IBM Systems Group M925: IBM Spectrum Scale Norbert Software Development Schuld IBM Deutschland R&D GmbH Phone: +49-160 70 70 335 Am Weiher 24 E-Mail: nschuld at de.ibm.com 65451 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina Koederitz /Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "J. Eric Wonderley" To: gpfsug main discussion list Date: 20/06/2017 18:04 Subject: [gpfsug-discuss] gui related connection fail in gpfs logs Sent by: gpfsug-discuss-bounces at spectrumscale.org These type messages repeat often in our logs: 017-06-20_09:25:13.676-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_09:25:24.292-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 2017-06-20_10:00:25.935-0400: [E] An%20attempt%20to%20send%20notification%20to%20the%20GUI%20subsystem%20failed%2E%20response%3Dcurl%3A%20%287%29%20Failed%20connect%20to%20arproto2%2Ear%2Enis%2Eisb%2Einternal%3A443%3B%20Connection%20refused%20rc%3D7 rc=1 Is there any way to tell if it is a misconfiguration or communications issue? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 16690161.gif Type: image/gif Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From ewahl at osc.edu Thu Jun 22 20:37:12 2017 From: ewahl at osc.edu (Edward Wahl) Date: Thu, 22 Jun 2017 15:37:12 -0400 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: References: <18719.1497296477@turing-police.cc.vt.edu> Message-ID: <20170622153712.052d312c@osc.edu> Is there a command to show existing node Address Policy? Or are we left with grep "affinity" on /var/mmfs/gen/mmsdrfs? Ed On Tue, 13 Jun 2017 08:30:18 +0000 "Sobey, Richard A" wrote: > Oh? Nice to know - thanks - will try that method next. > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson > (IT Research Support) Sent: 13 June 2017 09:28 To: gpfsug main discussion > list Subject: Re: [gpfsug-discuss] 'mmces > address move' weirdness? > > Suspending the node doesn't stop the services though, we've done a bunch of > testing by connecting to the "real" IP on the box we wanted to test and that > works fine. > > OK, so you end up connecting to shares like > \\192.168.1.20\sharename, but its perfectly > fine for testing purposes. > > In our experience, suspending the node has been fine for this as it moves the > IP to a "working" node and keeps user service running whilst we test. > > Simon > > From: > > > on behalf of "Sobey, Richard A" > > Reply-To: > "gpfsug-discuss at spectrumscale.org" > > > Date: Tuesday, 13 June 2017 at 09:08 To: > "gpfsug-discuss at spectrumscale.org" > > > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? > > Yes, suspending the node would do it, but in the case where you want to > remove a node from service but keep it running for testing it's not ideal. > > I think you can set the IP address balancing policy to none which might do > what we want. From: > gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon Thompson > (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main discussion > list > > > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? > > mmces node suspend -N > > Is what you want. This will move the address and stop it being assigned one, > otherwise the rebalance will occur. I think you can change the way it > balances, but the default is to distribute. > > Simon > > From: > > > on behalf of "Sobey, Richard A" > > Reply-To: > "gpfsug-discuss at spectrumscale.org" > > > Date: Monday, 12 June 2017 at 21:01 To: > "gpfsug-discuss at spectrumscale.org" > > > Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? > > > I think it's intended but I don't know why. The AUTH service became unhealthy > on one of our CES nodes (SMB only) and we moved its float address elsewhere. > CES decided to move it back again moments later despite the node not being > fit. > > > > Sorry that doesn't really help but at least you're not alone! > > ________________________________ > From:gpfsug-discuss-bounces at spectrumscale.org > > > on behalf of valdis.kletnieks at vt.edu > > Sent: 12 June 2017 > 20:41 To: > gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] 'mmces address move' weirdness? > > So here's our address setup: > > mmces address list > > Address Node Group Attribute > ------------------------------------------------------------------------- > 172.28.45.72 arproto1.ar.nis.isb.internal isb none > 172.28.45.73 arproto2.ar.nis.isb.internal isb none > 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none > 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none > > Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so I try to > move the address over to its pair so I can look around without impacting > users. However, seems like something insists on moving it right back 60 > seconds later... > > Question 1: Is this expected behavior? > Question 2: If it is, what use is 'mmces address move' if it just gets > undone a few seconds later... > > (running on arproto2.ar.nis.vtc.internal): > > ## (date; ip addr show | grep '\.72';mmces address move --ces-ip 172.28.46.72 > --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; ip addr > show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon Jun 12 > 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global > secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 EDT 2017 > Mon Jun 12 15:34:42 EDT 2017 > Mon Jun 12 15:34:43 EDT 2017 > (skipped) > Mon Jun 12 15:35:44 EDT 2017 > Mon Jun 12 15:35:45 EDT 2017 > inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 > Mon Jun 12 15:35:46 EDT 2017 > inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 > Mon Jun 12 15:35:47 EDT 2017 > inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary bond1:0 > ^C -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From laurence at qsplace.co.uk Thu Jun 22 23:05:49 2017 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Thu, 22 Jun 2017 23:05:49 +0100 Subject: [gpfsug-discuss] 'mmces address move' weirdness? In-Reply-To: <20170622153712.052d312c@osc.edu> References: <18719.1497296477@turing-police.cc.vt.edu> <20170622153712.052d312c@osc.edu> Message-ID: <6B843B3B-4C07-459D-B905-10B16E3590A0@qsplace.co.uk> "mmlscluster --ces" will show the address distribution policy. -- Lauz On 22 June 2017 20:37:12 BST, Edward Wahl wrote: > >Is there a command to show existing node Address Policy? >Or are we left with grep "affinity" on /var/mmfs/gen/mmsdrfs? > >Ed > > >On Tue, 13 Jun 2017 08:30:18 +0000 >"Sobey, Richard A" wrote: > >> Oh? Nice to know - thanks - will try that method next. >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon >Thompson >> (IT Research Support) Sent: 13 June 2017 09:28 To: gpfsug main >discussion >> list Subject: Re: [gpfsug-discuss] >'mmces >> address move' weirdness? >> >> Suspending the node doesn't stop the services though, we've done a >bunch of >> testing by connecting to the "real" IP on the box we wanted to test >and that >> works fine. >> >> OK, so you end up connecting to shares like >> \\192.168.1.20\sharename, but its >perfectly >> fine for testing purposes. >> >> In our experience, suspending the node has been fine for this as it >moves the >> IP to a "working" node and keeps user service running whilst we test. >> >> Simon >> >> From: >> >> >> on behalf of "Sobey, Richard A" >> > Reply-To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Date: Tuesday, 13 June 2017 at 09:08 To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? >> >> Yes, suspending the node would do it, but in the case where you want >to >> remove a node from service but keep it running for testing it's not >ideal. >> >> I think you can set the IP address balancing policy to none which >might do >> what we want. From: >> >gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Simon >Thompson >> (IT Research Support) Sent: 12 June 2017 21:06 To: gpfsug main >discussion >> list >> >> >> Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? >> >> mmces node suspend -N >> >> Is what you want. This will move the address and stop it being >assigned one, >> otherwise the rebalance will occur. I think you can change the way it >> balances, but the default is to distribute. >> >> Simon >> >> From: >> >> >> on behalf of "Sobey, Richard A" >> > Reply-To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Date: Monday, 12 June 2017 at 21:01 To: >> >"gpfsug-discuss at spectrumscale.org" >> >> >> Subject: Re: [gpfsug-discuss] 'mmces address move' weirdness? >> >> >> I think it's intended but I don't know why. The AUTH service became >unhealthy >> on one of our CES nodes (SMB only) and we moved its float address >elsewhere. >> CES decided to move it back again moments later despite the node not >being >> fit. >> >> >> >> Sorry that doesn't really help but at least you're not alone! >> >> ________________________________ >> >From:gpfsug-discuss-bounces at spectrumscale.org >> >> >> on behalf of valdis.kletnieks at vt.edu >> > Sent: 12 >June 2017 >> 20:41 To: >> >gpfsug-discuss at spectrumscale.org >> Subject: [gpfsug-discuss] 'mmces address move' weirdness? >> >> So here's our address setup: >> >> mmces address list >> >> Address Node Group >Attribute >> >------------------------------------------------------------------------- >> 172.28.45.72 arproto1.ar.nis.isb.internal isb none >> 172.28.45.73 arproto2.ar.nis.isb.internal isb none >> 172.28.46.72 arproto2.ar.nis.vtc.internal vtc none >> 172.28.46.73 arproto1.ar.nis.vtc.internal vtc none >> >> Having some nfs-ganesha weirdness on arproto2.ar.nis.vtc.internal, so >I try to >> move the address over to its pair so I can look around without >impacting >> users. However, seems like something insists on moving it right back >60 >> seconds later... >> >> Question 1: Is this expected behavior? >> Question 2: If it is, what use is 'mmces address move' if it just >gets >> undone a few seconds later... >> >> (running on arproto2.ar.nis.vtc.internal): >> >> ## (date; ip addr show | grep '\.72';mmces address move --ces-ip >172.28.46.72 >> --ces-node arproto1.ar.nis.vtc.internal; while (/bin/true); do date; >ip addr >> show | grep '\.72'; sleep 1; done;) | tee migrate.not.nailed.down Mon >Jun 12 >> 15:34:33 EDT 2017 inet 172.28.46.72/26 brd 172.28.46.127 scope global >> secondary bond1:0 Mon Jun 12 15:34:40 EDT 2017 Mon Jun 12 15:34:41 >EDT 2017 >> Mon Jun 12 15:34:42 EDT 2017 >> Mon Jun 12 15:34:43 EDT 2017 >> (skipped) >> Mon Jun 12 15:35:44 EDT 2017 >> Mon Jun 12 15:35:45 EDT 2017 >> inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary >bond1:0 >> Mon Jun 12 15:35:46 EDT 2017 >> inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary >bond1:0 >> Mon Jun 12 15:35:47 EDT 2017 >> inet 172.28.46.72/26 brd 172.28.46.127 scope global secondary >bond1:0 >> ^C > > > >-- > >Ed Wahl >Ohio Supercomputer Center >614-292-9302 >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Fri Jun 23 09:13:38 2017 From: john.hearns at asml.com (John Hearns) Date: Fri, 23 Jun 2017 08:13:38 +0000 Subject: [gpfsug-discuss] IO prioritisation / throttling? Message-ID: I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO 'runs wild' it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Fri Jun 23 09:57:46 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Fri, 23 Jun 2017 04:57:46 -0400 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: References: Message-ID: <11158C89-C712-4C79-8B1D-CAA9D3D8641F@gmail.com> I unfortunately don't have an answer other than to perhaps check out this presentation from a recent users group meeting: http://files.gpfsug.org/presentations/2017/Manchester/05_Ellexus_SSUG_Manchester.pdf I've never used the product but it might be able to do what you're asking. Sent from my iPhone > On Jun 23, 2017, at 04:13, John Hearns wrote: > > I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. > We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. > The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down > the production storage. If anyone has a setup like this I would be interested in chatting with you. > Is it feasible to create filesets which have higher/lower priority than others? > > Thankyou for any insights or feedback > John Hearns > -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Fri Jun 23 12:04:23 2017 From: john.hearns at asml.com (John Hearns) Date: Fri, 23 Jun 2017 11:04:23 +0000 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: <11158C89-C712-4C79-8B1D-CAA9D3D8641F@gmail.com> References: <11158C89-C712-4C79-8B1D-CAA9D3D8641F@gmail.com> Message-ID: Aaron, thankyou. I know Rosemary and that is a good company! From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Friday, June 23, 2017 10:58 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] IO prioritisation / throttling? I unfortunately don't have an answer other than to perhaps check out this presentation from a recent users group meeting: http://files.gpfsug.org/presentations/2017/Manchester/05_Ellexus_SSUG_Manchester.pdf I've never used the product but it might be able to do what you're asking. Sent from my iPhone On Jun 23, 2017, at 04:13, John Hearns > wrote: I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kums at us.ibm.com Fri Jun 23 14:36:34 2017 From: kums at us.ibm.com (Kumaran Rajaram) Date: Fri, 23 Jun 2017 09:36:34 -0400 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: References: Message-ID: Hi John, >>We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. >>The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down >>the production storage. You may use the Spectrum Scale Quality of Service for I/O "mmchqos" command (details in link below) to define IOPS limits for the "others" as well as the "maintenance" class for the Dev/Test file-system "pools" (for e.g., mmchqos tds_fs --enable pool=*,other=10000IOPS, maintenance=5000IOPS). This way, the Test and Dev file-system/storage-pools IOPS can be limited/controlled to specified IOPS , giving higher priority to the production GPFS file-system/storage (with production_fs pool=* other=unlimited,maintenance=unlimited - which is the default). https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_mmchqos.htm https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_qosio_describe.htm#qosio_describe My two cents. Regards, -Kums From: John Hearns To: gpfsug main discussion list Date: 06/23/2017 04:14 AM Subject: [gpfsug-discuss] IO prioritisation / throttling? Sent by: gpfsug-discuss-bounces at spectrumscale.org I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Fri Jun 23 15:23:19 2017 From: john.hearns at asml.com (John Hearns) Date: Fri, 23 Jun 2017 14:23:19 +0000 Subject: [gpfsug-discuss] IO prioritisation / throttling? In-Reply-To: References: Message-ID: Thankyou to Kumaran and Aaaron for your help. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Kumaran Rajaram Sent: Friday, June 23, 2017 3:37 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] IO prioritisation / throttling? Hi John, >>We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. >>The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down >>the production storage. You may use the Spectrum Scale Quality of Service for I/O "mmchqos" command (details in link below) to define IOPS limits for the "others" as well as the "maintenance" class for the Dev/Test file-system "pools" (for e.g., mmchqos tds_fs --enable pool=*,other=10000IOPS, maintenance=5000IOPS). This way, the Test and Dev file-system/storage-pools IOPS can be limited/controlled to specified IOPS , giving higher priority to the production GPFS file-system/storage (with production_fs pool=* other=unlimited,maintenance=unlimited - which is the default). https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_mmchqos.htm https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_qosio_describe.htm#qosio_describe My two cents. Regards, -Kums From: John Hearns > To: gpfsug main discussion list > Date: 06/23/2017 04:14 AM Subject: [gpfsug-discuss] IO prioritisation / throttling? Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I guess this is a rather ill-defined question, and I realise it will be open to a lot of interpretations. We have a GPFS Setup using Fujitsu filers and Mellanox infiniband. The desire it to set up an environment for test and development where if IO ?runs wild? it will not bring down the production storage. If anyone has a setup like this I would be interested in chatting with you. Is it feasible to create filesets which have higher/lower priority than others? Thankyou for any insights or feedback John Hearns -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 23 17:06:51 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 23 Jun 2017 16:06:51 +0000 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy Message-ID: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> Hi All, I haven?t been able to find this explicitly documented, so I?m just wanting to confirm that the behavior that I?m expecting is what GPFS is going to do in this scenario? I have a filesystem with data replication set to two. I?m creating a capacity type pool for it right now which will be used to migrate old files to. I only want to use replication of one on the capacity pool. My policy file has two rules, one to move files with an atime > 30 days to the capacity pool, to which I?ve included ?REPLICATE(1)?. The other rule is to move files from the capacity pool back to the system pool if the atime < 14 days. Since data replication is set to two, I am thinking that I do not need to explicitly have a ?REPLICATE(2)? as part of that rule ? is that correct? I.e., I?m wanting to make sure that a file moved to the capacity pool which therefore has its? replication set to one doesn?t keep that same setting even if moved back out of the capacity pool. Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 23 17:58:23 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 23 Jun 2017 12:58:23 -0400 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy In-Reply-To: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> References: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> Message-ID: I believe that is correct. If not, let us know! To recap... when running mmapplypolicy with rules like: ... MIGRATE ... REPLICATE(x) ... will change the replication factor to x, for each file selected by this rule and chosen for execution. ... MIGRATE ... /* no REPLICATE keyword */ will not mess with the replication factor From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/23/2017 12:07 PM Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I haven?t been able to find this explicitly documented, so I?m just wanting to confirm that the behavior that I?m expecting is what GPFS is going to do in this scenario? I have a filesystem with data replication set to two. I?m creating a capacity type pool for it right now which will be used to migrate old files to. I only want to use replication of one on the capacity pool. My policy file has two rules, one to move files with an atime > 30 days to the capacity pool, to which I?ve included ?REPLICATE(1)?. The other rule is to move files from the capacity pool back to the system pool if the atime < 14 days. Since data replication is set to two, I am thinking that I do not need to explicitly have a ?REPLICATE(2)? as part of that rule ? is that correct? I.e., I?m wanting to make sure that a file moved to the capacity pool which therefore has its? replication set to one doesn?t keep that same setting even if moved back out of the capacity pool. Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Fri Jun 23 18:55:15 2017 From: ulmer at ulmer.org (Stephen Ulmer) Date: Fri, 23 Jun 2017 13:55:15 -0400 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy In-Reply-To: References: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> Message-ID: <82717215-1C59-4B12-BAF6-09908044688D@ulmer.org> > On Jun 23, 2017, at 12:58 PM, Marc A Kaplan > wrote: > > I believe that is correct. If not, let us know! > > To recap... when running mmapplypolicy with rules like: > > ... MIGRATE ... REPLICATE(x) ... > > will change the replication factor to x, for each file selected by this rule and chosen for execution. > > ... MIGRATE ... /* no REPLICATE keyword */ > > will not mess with the replication factor > > I think I detect an impedance mismatch... By "not mess with the replication factor" do you mean that after the move: the file will have the default replication factor for the file system the file will retain a replication factor previously set on the file You told Kevin that he was correct and I think he meant the first one, but I read what you said as the second one. Liberty, -- Stephen -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 23 20:28:28 2017 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 23 Jun 2017 15:28:28 -0400 Subject: [gpfsug-discuss] Replication settings when running mmapplypolicy In-Reply-To: <82717215-1C59-4B12-BAF6-09908044688D@ulmer.org> References: <32D51306-988F-4F18-9883-31A00975A9AC@vanderbilt.edu> <82717215-1C59-4B12-BAF6-09908044688D@ulmer.org> Message-ID: Sorry for any confusion. MIGRATing a file does NOT change the replication factor, unless you explicitly use the keyword REPLICATE. The default replication factor, as set/displayed by mm[ch|ls]fs -r only applies at file creation time, unless overriden by a policy SET POOL ... REPLICATE(x) rule. From: Stephen Ulmer To: gpfsug main discussion list Date: 06/23/2017 01:55 PM Subject: Re: [gpfsug-discuss] Replication settings when running mmapplypolicy Sent by: gpfsug-discuss-bounces at spectrumscale.org On Jun 23, 2017, at 12:58 PM, Marc A Kaplan wrote: I believe that is correct. If not, let us know! To recap... when running mmapplypolicy with rules like: ... MIGRATE ... REPLICATE(x) ... will change the replication factor to x, for each file selected by this rule and chosen for execution. ... MIGRATE ... /* no REPLICATE keyword */ will not mess with the replication factor I think I detect an impedance mismatch... By "not mess with the replication factor" do you mean that after the move: the file will have the default replication factor for the file system the file will retain a replication factor previously set on the file You told Kevin that he was correct and I think he meant the first one, but I read what you said as the second one. Liberty, -- Stephen _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From ncapit at atos.net Mon Jun 26 08:49:28 2017 From: ncapit at atos.net (CAPIT, NICOLAS) Date: Mon, 26 Jun 2017 07:49:28 +0000 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads Message-ID: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Hello, I don't know if this behavior/bug was already reported on this ML, so in doubt. Context: - SpectrumScale 4.2.2-3 - client node with 64 cores - OS: RHEL7.3 When a MPI job with 64 processes is launched on the node with 64 cores then the FS freezed (only the output log file of the MPI job is put on the GPFS; so it may be related to the 64 processes writing in a same file???). strace -p 3105 # mmfsd pid stucked Process 3105 attached wait4(-1, # stucked at this point strace ls /gpfs stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC # stucked at this point I have no problem with the other nodes of 28 cores. The GPFS command mmgetstate is working and I am able to use mmshutdown to recover the node. If I put workerThreads=72 on the 64 core node then I am not able to reproduce the freeze and I get the right behavior. Is this a known bug with a number of cores > workerThreads? Best regards, [--] Nicolas Capit -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jun 27 00:57:57 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 26 Jun 2017 19:57:57 -0400 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Message-ID: <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> That's a fascinating bug. When the node is locked up what does "mmdiag --waiters" show from the node in question? I suspect there's more low-level diagnostic data that's helpful for the gurus at IBM but I'm just curious what the waiters look like. -Aaron On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > Hello, > > I don't know if this behavior/bug was already reported on this ML, so in > doubt. > > Context: > > - SpectrumScale 4.2.2-3 > - client node with 64 cores > - OS: RHEL7.3 > > When a MPI job with 64 processes is launched on the node with 64 cores > then the FS freezed (only the output log file of the MPI job is put on > the GPFS; so it may be related to the 64 processes writing in a same > file???). > > strace -p 3105 # mmfsd pid stucked > Process 3105 attached > wait4(-1, # stucked at this point > > strace ls /gpfs > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > # stucked at this point > > I have no problem with the other nodes of 28 cores. > The GPFS command mmgetstate is working and I am able to use mmshutdown > to recover the node. > > > If I put workerThreads=72 on the 64 core node then I am not able to > reproduce the freeze and I get the right behavior. > > Is this a known bug with a number of cores > workerThreads? > > Best regards, > -- > *Nicolas Capit* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From ncapit at atos.net Tue Jun 27 07:59:19 2017 From: ncapit at atos.net (CAPIT, NICOLAS) Date: Tue, 27 Jun 2017 06:59:19 +0000 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net>, <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> Message-ID: <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Hello, When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters"). In the GPFS log file "/var/mmfs/gen/mmfslog" there is nothing and nothing in the dmesg output or system log. The "mmgetstate" command says that the node is "active". The only thing is the freeze of the FS. Best regards, Nicolas Capit ________________________________________ De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de Aaron Knister [aaron.s.knister at nasa.gov] Envoy? : mardi 27 juin 2017 01:57 ? : gpfsug-discuss at spectrumscale.org Objet : Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads That's a fascinating bug. When the node is locked up what does "mmdiag --waiters" show from the node in question? I suspect there's more low-level diagnostic data that's helpful for the gurus at IBM but I'm just curious what the waiters look like. -Aaron On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > Hello, > > I don't know if this behavior/bug was already reported on this ML, so in > doubt. > > Context: > > - SpectrumScale 4.2.2-3 > - client node with 64 cores > - OS: RHEL7.3 > > When a MPI job with 64 processes is launched on the node with 64 cores > then the FS freezed (only the output log file of the MPI job is put on > the GPFS; so it may be related to the 64 processes writing in a same > file???). > > strace -p 3105 # mmfsd pid stucked > Process 3105 attached > wait4(-1, # stucked at this point > > strace ls /gpfs > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > # stucked at this point > > I have no problem with the other nodes of 28 cores. > The GPFS command mmgetstate is working and I am able to use mmshutdown > to recover the node. > > > If I put workerThreads=72 on the 64 core node then I am not able to > reproduce the freeze and I get the right behavior. > > Is this a known bug with a number of cores > workerThreads? > > Best regards, > -- > *Nicolas Capit* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From stuartb at 4gh.net Tue Jun 27 22:34:34 2017 From: stuartb at 4gh.net (Stuart Barkley) Date: Tue, 27 Jun 2017 17:34:34 -0400 (EDT) Subject: [gpfsug-discuss] express edition vs standard edition Message-ID: Does anyone know what controls whether GPFS (4.1.1) thinks it is Express Edition versus Standard Edition? While rebuilding an old cluster from scratch we are getting the message: mmcrfs: Storage pools are not available in the GPFS Express Edition. The Problem Determination Guide says to "Install the GPFS Standard Edition on all nodes in the cluster" which we think we have done. The cluster is just 3 servers and no clients at this point. We have verified that our purchased license is for Standard Version, but have not been able to figure out what controls the GPFS view of this. mmlslicense tells us that we have Express Edition installed. mmchlicense sets server vs client license information, but does not seem to be able to control the edition. Our normal install process is to install gpfs.base-4.1.1-0.x86_64.rpm first and then install gpfs.base-4.1.1-15.x86_64.update.rpm followed by the other needed 4.1.1-15 rpms. I thought maybe we had the wrong gpfs.base and we have re-downloaded Standard Edition RPM files from IBM in case we had the wrong version. However, reinstalling and recreating the cluster does not seem to have addressed this issue. We must be doing something stupid during our install, but I'm pretty sure we used only Standard Edition rpms for our latest attempt. Thanks, Stuart Barkley -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From knop at us.ibm.com Tue Jun 27 22:54:35 2017 From: knop at us.ibm.com (Felipe Knop) Date: Tue, 27 Jun 2017 17:54:35 -0400 Subject: [gpfsug-discuss] express edition vs standard edition In-Reply-To: References: Message-ID: Stuart, I believe you will need to install the gpfs.ext RPMs , otherwise the daemons and commands will think only the Express edition is installed. Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 From: Stuart Barkley To: gpfsug-discuss at spectrumscale.org Date: 06/27/2017 05:35 PM Subject: [gpfsug-discuss] express edition vs standard edition Sent by: gpfsug-discuss-bounces at spectrumscale.org Does anyone know what controls whether GPFS (4.1.1) thinks it is Express Edition versus Standard Edition? While rebuilding an old cluster from scratch we are getting the message: mmcrfs: Storage pools are not available in the GPFS Express Edition. The Problem Determination Guide says to "Install the GPFS Standard Edition on all nodes in the cluster" which we think we have done. The cluster is just 3 servers and no clients at this point. We have verified that our purchased license is for Standard Version, but have not been able to figure out what controls the GPFS view of this. mmlslicense tells us that we have Express Edition installed. mmchlicense sets server vs client license information, but does not seem to be able to control the edition. Our normal install process is to install gpfs.base-4.1.1-0.x86_64.rpm first and then install gpfs.base-4.1.1-15.x86_64.update.rpm followed by the other needed 4.1.1-15 rpms. I thought maybe we had the wrong gpfs.base and we have re-downloaded Standard Edition RPM files from IBM in case we had the wrong version. However, reinstalling and recreating the cluster does not seem to have addressed this issue. We must be doing something stupid during our install, but I'm pretty sure we used only Standard Edition rpms for our latest attempt. Thanks, Stuart Barkley -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuartb at 4gh.net Tue Jun 27 23:48:53 2017 From: stuartb at 4gh.net (Stuart Barkley) Date: Tue, 27 Jun 2017 18:48:53 -0400 (EDT) Subject: [gpfsug-discuss] express edition vs standard edition In-Reply-To: References: Message-ID: On Tue, 27 Jun 2017 at 17:54 -0000, Felipe Knop wrote: > I believe you will need to install the gpfs.ext RPMs , otherwise the > daemons and commands will think only the Express edition is > installed. Yes. This appears to have been my problem. Thanks, Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From scale at us.ibm.com Fri Jun 30 07:57:49 2017 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 30 Jun 2017 14:57:49 +0800 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net>, <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Message-ID: I'm not aware this kind of defects, seems it should not. but lack of data, we don't know what happened. I suggest you can open a PMR for your issue. Thanks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "CAPIT, NICOLAS" To: gpfsug main discussion list Date: 06/27/2017 02:59 PM Subject: Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters"). In the GPFS log file "/var/mmfs/gen/mmfslog" there is nothing and nothing in the dmesg output or system log. The "mmgetstate" command says that the node is "active". The only thing is the freeze of the FS. Best regards, Nicolas Capit ________________________________________ De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de Aaron Knister [aaron.s.knister at nasa.gov] Envoy? : mardi 27 juin 2017 01:57 ? : gpfsug-discuss at spectrumscale.org Objet : Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads That's a fascinating bug. When the node is locked up what does "mmdiag --waiters" show from the node in question? I suspect there's more low-level diagnostic data that's helpful for the gurus at IBM but I'm just curious what the waiters look like. -Aaron On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > Hello, > > I don't know if this behavior/bug was already reported on this ML, so in > doubt. > > Context: > > - SpectrumScale 4.2.2-3 > - client node with 64 cores > - OS: RHEL7.3 > > When a MPI job with 64 processes is launched on the node with 64 cores > then the FS freezed (only the output log file of the MPI job is put on > the GPFS; so it may be related to the 64 processes writing in a same > file???). > > strace -p 3105 # mmfsd pid stucked > Process 3105 attached > wait4(-1, # stucked at this point > > strace ls /gpfs > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > # stucked at this point > > I have no problem with the other nodes of 28 cores. > The GPFS command mmgetstate is working and I am able to use mmshutdown > to recover the node. > > > If I put workerThreads=72 on the 64 core node then I am not able to > reproduce the freeze and I get the right behavior. > > Is this a known bug with a number of cores > workerThreads? > > Best regards, > -- > *Nicolas Capit* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From heiner.billich at psi.ch Fri Jun 30 11:07:10 2017 From: heiner.billich at psi.ch (Billich Heinrich Rainer (PSI)) Date: Fri, 30 Jun 2017 10:07:10 +0000 Subject: [gpfsug-discuss] AFM - how to update directories with deleted files during prefetch Message-ID: <76D2B41B-37A6-410B-9ECE-F5FA4C7FF1EE@psi.ch> Hello I have a short question about AFM prefetch and some more remarks regarding AFM and it?s use for data migration. I understand that many of you have done this for very large amounts of data and number of files. I would welcome an input, comments or remarks. Sorry if this is a bit too long for a mailing list. Short: How can I tell an AFM cache to update a directory when I do prefetch? I know about ?find .? or ?ls ?lsR? but this really is no option for us as it takes too long. Mostly I want to update the directories to make AFM cache aware of file deletions on home. On home I can use a policy run to find all directories which changed since the last update and pass them to prefetch on AFM cache. I know that I can find some workaround based on the directory list, like an ?ls ?lsa? just for those directories, but this doesn?t sound very efficient. And depending on cache effects and timeout settings it may work or not (o.k. ? it will work most time). We do regular file deletions and will accumulated millions of deleted files on cache over time if we don?t update the directories to make AFM cache aware of the deletion. Background: We will use AFM to migrate data on filesets to another cluster. We have to do this several times in the next few months, hence I want to get a reliable and easy to use procedure. The old system is home, the new system is a read-only AFM cache. We want to use ?mmafmctl prefetch? to move the data. Home will be in use while we run the migration. Once almost all data is moved we do a (short) break for a last sync and make the read-only AFM cache a ?normal? fileset. During the break I want to use policy runs and prefetch only and no time consuming ?ls ?lsr? or ?find .? I don?t want to use this metadata intensive posix operation during operation, either. More general: AFM can be used for data migration. But I don?t see how to use it efficiently. How to do incremental transfers, how to ensure that the we really have identical copies before we switch and how to keep the switch time short , i.e. the time when both old and new aren?t accessible for clients, Wish ? maybe an RFE? I can use policy runs to collect all changed items on home since the last update. I wish that I can pass this list to afm prefetch to do all updates on AFM cache, too. Same as backup tools use the list to do incremental backups. And a tool to create policy lists of home and cache and to compare the lists would be nice, too. As you do this during the break/switch it should be fast and reliable and leave no doubts. Kind regards, Heiner From vpuvvada at in.ibm.com Fri Jun 30 13:35:18 2017 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Fri, 30 Jun 2017 18:05:18 +0530 Subject: [gpfsug-discuss] AFM - how to update directories with deleted files during prefetch In-Reply-To: <76D2B41B-37A6-410B-9ECE-F5FA4C7FF1EE@psi.ch> References: <76D2B41B-37A6-410B-9ECE-F5FA4C7FF1EE@psi.ch> Message-ID: What is the version of GPFS ? >Mostly I want to update the directories to make AFM cache aware of file deletions on home. On home I can use a policy run to find all directories which changed since the last >update and pass them to prefetch on AFM cache. AFM prefetch has undocumented option to delete files from cache without doing lookup from home. It supports all types of list files. Find all deleted file from home and prefetch at cache to delete them. mmafmctl device prefetch -j fileset --list-file --delete >Wish ? maybe an RFE? >I can use policy runs to collect all changed items on home since the last update. I wish that I can pass this list to afm prefetch to do all updates on AFM cache, too. Same >as backup tools use the list to do incremental backups. This feature support was already added but undocumented today. This feature will be externalized in future releases. ~Venkat (vpuvvada at in.ibm.com) From: "Billich Heinrich Rainer (PSI)" To: "gpfsug-discuss at spectrumscale.org" Date: 06/30/2017 03:37 PM Subject: [gpfsug-discuss] AFM - how to update directories with deleted files during prefetch Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello I have a short question about AFM prefetch and some more remarks regarding AFM and it?s use for data migration. I understand that many of you have done this for very large amounts of data and number of files. I would welcome an input, comments or remarks. Sorry if this is a bit too long for a mailing list. Short: How can I tell an AFM cache to update a directory when I do prefetch? I know about ?find .? or ?ls ?lsR? but this really is no option for us as it takes too long. Mostly I want to update the directories to make AFM cache aware of file deletions on home. On home I can use a policy run to find all directories which changed since the last update and pass them to prefetch on AFM cache. I know that I can find some workaround based on the directory list, like an ?ls ?lsa? just for those directories, but this doesn?t sound very efficient. And depending on cache effects and timeout settings it may work or not (o.k. ? it will work most time). We do regular file deletions and will accumulated millions of deleted files on cache over time if we don?t update the directories to make AFM cache aware of the deletion. Background: We will use AFM to migrate data on filesets to another cluster. We have to do this several times in the next few months, hence I want to get a reliable and easy to use procedure. The old system is home, the new system is a read-only AFM cache. We want to use ?mmafmctl prefetch? to move the data. Home will be in use while we run the migration. Once almost all data is moved we do a (short) break for a last sync and make the read-only AFM cache a ?normal? fileset. During the break I want to use policy runs and prefetch only and no time consuming ?ls ?lsr? or ?find .? I don?t want to use this metadata intensive posix operation during operation, either. More general: AFM can be used for data migration. But I don?t see how to use it efficiently. How to do incremental transfers, how to ensure that the we really have identical copies before we switch and how to keep the switch time short , i.e. the time when both old and new aren?t accessible for clients, Wish ? maybe an RFE? I can use policy runs to collect all changed items on home since the last update. I wish that I can pass this list to afm prefetch to do all updates on AFM cache, too. Same as backup tools use the list to do incremental backups. And a tool to create policy lists of home and cache and to compare the lists would be nice, too. As you do this during the break/switch it should be fast and reliable and leave no doubts. Kind regards, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From hpc-luke at uconn.edu Fri Jun 30 16:20:27 2017 From: hpc-luke at uconn.edu (hpc-luke at uconn.edu) Date: Fri, 30 Jun 2017 11:20:27 -0400 Subject: [gpfsug-discuss] Mass UID migration suggestions Message-ID: <59566c3b.kqOSy9e2kg80XZ/k%hpc-luke@uconn.edu> Hello, We're trying to change most of our users uids, is there a clean way to migrate all of one users files with say `mmapplypolicy`? We have to change the owner of around 273539588 files, and my estimates for runtime are around 6 days. What we've been doing is indexing all of the files and splitting them up by owner which takes around an hour, and then we were locking the user out while we chown their files. I made it multi threaded as it weirdly gave a 10% speedup despite my expectation that multi threading access from a single node would not give any speedup. Generally I'm looking for advice on how to make the chowning faster. Would spreading the chowning processes over multiple nodes improve performance? Should I not stat the files before running lchown on them, since lchown checks the file before changing it? I saw mention of inodescan(), in an old gpfsug email, which speeds up disk read access, by not guaranteeing that the data is up to date. We have a maintenance day coming up where all users will be locked out, so the file handles(?) from GPFS's perspective will not be able to go stale. Is there a function with similar constraints to inodescan that I can use to speed up this process? Thank you for your time, Luke Storrs-HPC University of Connecticut From aaron.knister at gmail.com Fri Jun 30 16:47:40 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Fri, 30 Jun 2017 11:47:40 -0400 Subject: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads In-Reply-To: References: <441FC013797C0F4B9004428065AD55CE18409C6E@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> <57681d30-7daa-848b-6d64-f74650cf5787@nasa.gov> <441FC013797C0F4B9004428065AD55CE1840BEFC@FRCRPVV9EX6MSX.ww931.my-it-solutions.net> Message-ID: Nicolas, By chance do you have a skylake or kabylake based CPU? Sent from my iPhone > On Jun 30, 2017, at 02:57, IBM Spectrum Scale wrote: > > I'm not aware this kind of defects, seems it should not. but lack of data, we don't know what happened. I suggest you can open a PMR for your issue. Thanks. > > Regards, The Spectrum Scale (GPFS) team > > ------------------------------------------------------------------------------------------------------------------ > If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. > > If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. > > The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. > > "CAPIT, NICOLAS" ---06/27/2017 02:59:59 PM---Hello, When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters") > > From: "CAPIT, NICOLAS" > To: gpfsug main discussion list > Date: 06/27/2017 02:59 PM > Subject: Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > Hello, > > When the node is locked up there is no waiters ("mmdiad --waiters" or "mmfsadm dump waiters"). > In the GPFS log file "/var/mmfs/gen/mmfslog" there is nothing and nothing in the dmesg output or system log. > The "mmgetstate" command says that the node is "active". > The only thing is the freeze of the FS. > > Best regards, > Nicolas Capit > ________________________________________ > De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de Aaron Knister [aaron.s.knister at nasa.gov] > Envoy? : mardi 27 juin 2017 01:57 > ? : gpfsug-discuss at spectrumscale.org > Objet : Re: [gpfsug-discuss] FS freeze on client nodes with nbCores>workerThreads > > That's a fascinating bug. When the node is locked up what does "mmdiag > --waiters" show from the node in question? I suspect there's more > low-level diagnostic data that's helpful for the gurus at IBM but I'm > just curious what the waiters look like. > > -Aaron > > On 6/26/17 3:49 AM, CAPIT, NICOLAS wrote: > > Hello, > > > > I don't know if this behavior/bug was already reported on this ML, so in > > doubt. > > > > Context: > > > > - SpectrumScale 4.2.2-3 > > - client node with 64 cores > > - OS: RHEL7.3 > > > > When a MPI job with 64 processes is launched on the node with 64 cores > > then the FS freezed (only the output log file of the MPI job is put on > > the GPFS; so it may be related to the 64 processes writing in a same > > file???). > > > > strace -p 3105 # mmfsd pid stucked > > Process 3105 attached > > wait4(-1, # stucked at this point > > > > strace ls /gpfs > > stat("/gpfs", {st_mode=S_IFDIR|0755, st_size=131072, ...}) = 0 > > openat(AT_FDCWD, "/gpfs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC > > # stucked at this point > > > > I have no problem with the other nodes of 28 cores. > > The GPFS command mmgetstate is working and I am able to use mmshutdown > > to recover the node. > > > > > > If I put workerThreads=72 on the 64 core node then I am not able to > > reproduce the freeze and I get the right behavior. > > > > Is this a known bug with a number of cores > workerThreads? > > > > Best regards, > > -- > > *Nicolas Capit* > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Jun 30 18:14:07 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 30 Jun 2017 13:14:07 -0400 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: <487469581.449569.1498832342497.JavaMail.webinst@w30112> References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: I'm curious to know why this doesn't affect GSS/ESS? Is it a feature of the additional check-summing done on those platforms? -------- Forwarded Message -------- Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) Date: Fri, 30 Jun 2017 14:19:02 +0000 From: IBM My Notifications To: aaron.s.knister at nasa.gov My Notifications for Storage - 30 Jun 2017 Dear Subscriber (aaron.s.knister at nasa.gov), Here are your updates from IBM My Notifications. Your support Notifications display in English by default. Machine translation based on your IBM profile language setting is added if you specify this option in My defaults within My Notifications. (Note: Not all languages are available at this time, and the English version always takes precedence over the machine translated version.) ------------------------------------------------------------------------------ 1. IBM Spectrum Scale - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error - URL: http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM Spectrum Scale versions where the NSD server is enabled to use RDMA for file IO and the storage used in your GPFS cluster accessed via NSD servers (not fully SAN accessible) includes anything other than IBM Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under these conditions, when the RDMA-enabled network adapter fails, the issue may result in undetected data corruption for file write or read operations. ------------------------------------------------------------------------------ Manage your My Notifications subscriptions, or send questions and comments. - Subscribe or Unsubscribe - https://www.ibm.com/support/mynotifications - Feedback - https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html - Follow us on Twitter - https://twitter.com/IBMStorageSupt To ensure proper delivery please add mynotify at stg.events.ihost.com to your address book. You received this email because you are subscribed to IBM My Notifications as: aaron.s.knister at nasa.gov Please do not reply to this message as it is generated by an automated service machine. (C) International Business Machines Corporation 2017. All rights reserved. From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 30 18:25:56 2017 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 30 Jun 2017 17:25:56 +0000 Subject: [gpfsug-discuss] Mass UID migration suggestions In-Reply-To: <59566c3b.kqOSy9e2kg80XZ/k%hpc-luke@uconn.edu> References: <59566c3b.kqOSy9e2kg80XZ/k%hpc-luke@uconn.edu> Message-ID: <1901859B-7620-42E2-9064-14930AC50EE3@vanderbilt.edu> Hi Luke, I?ve got an off the wall suggestion for you, which may or may not work depending on whether or not you have any UID conflicts with old and new UIDs ? this won?t actually speed things up but it will eliminate the ?downtime? for your users. And the big caveat is that there can?t be any UID conflicts ? i.e. someone?s new UID can?t be someone else?s old UID. Given that ? what if you set an ACL to allow access to both their old and new UIDs ? then change their UID to the new UID ? then chown the files to the new UID and remove the ACL? More work for you, but no downtime for them. We actually may need to do something similar as we will need to change Windows-assigned UID?s based on SIDs to ?correct? UIDs at some point in the future on one of our storage systems. If someone has a better way to solve your problem, I hope they?ll post it to the list, as it may help us as well. HTHAL. Thanks? Kevin On Jun 30, 2017, at 10:20 AM, hpc-luke at uconn.edu wrote: Hello, We're trying to change most of our users uids, is there a clean way to migrate all of one users files with say `mmapplypolicy`? We have to change the owner of around 273539588 files, and my estimates for runtime are around 6 days. What we've been doing is indexing all of the files and splitting them up by owner which takes around an hour, and then we were locking the user out while we chown their files. I made it multi threaded as it weirdly gave a 10% speedup despite my expectation that multi threading access from a single node would not give any speedup. Generally I'm looking for advice on how to make the chowning faster. Would spreading the chowning processes over multiple nodes improve performance? Should I not stat the files before running lchown on them, since lchown checks the file before changing it? I saw mention of inodescan(), in an old gpfsug email, which speeds up disk read access, by not guaranteeing that the data is up to date. We have a maintenance day coming up where all users will be locked out, so the file handles(?) from GPFS's perspective will not be able to go stale. Is there a function with similar constraints to inodescan that I can use to speed up this process? Thank you for your time, Luke Storrs-HPC University of Connecticut _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Fri Jun 30 18:37:30 2017 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Fri, 30 Jun 2017 19:37:30 +0200 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: An HTML attachment was scrubbed... URL: From aaron.knister at gmail.com Fri Jun 30 18:41:43 2017 From: aaron.knister at gmail.com (Aaron Knister) Date: Fri, 30 Jun 2017 13:41:43 -0400 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: Thanks Olaf, that's good to know (and is kind of what I suspected). I've requested a number of times this capability for those of us who can't use or aren't using GNR and the answer is effectively "no". This response is curious to me because I'm sure IBM doesn't believe that data integrity is only important and of value to customers who purchase their hardware *and* software. -Aaron On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser wrote: > yes.. in case of GNR (GPFS native raid) .. we do end-to-end check-summing > ... client --> server --> downToDisk > GNR writes down a chksum to disk (to all pdisks /all "raid" segments ) so > that dropped writes can be detected as well as miss-done writes (bit > flips..) > > > > From: Aaron Knister > To: gpfsug main discussion list > Date: 06/30/2017 07:15 PM > Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): > RDMA-enabled network adapter failure on the NSD server may result in file > IO error (2017.06.30) > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > I'm curious to know why this doesn't affect GSS/ESS? Is it a feature of > the additional check-summing done on those platforms? > > > -------- Forwarded Message -------- > Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled > network adapter > failure on the NSD server may result in file IO error (2017.06.30) > Date: Fri, 30 Jun 2017 14:19:02 +0000 > From: IBM My Notifications > > To: aaron.s.knister at nasa.gov > > > > > My Notifications for Storage - 30 Jun 2017 > > Dear Subscriber (aaron.s.knister at nasa.gov), > > Here are your updates from IBM My Notifications. > > Your support Notifications display in English by default. Machine > translation based on your IBM profile > language setting is added if you specify this option in My defaults > within My Notifications. > (Note: Not all languages are available at this time, and the English > version always takes precedence > over the machine translated version.) > > ------------------------------------------------------------ > ------------------ > 1. IBM Spectrum Scale > > - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure > on the NSD server may result in file IO error > - URL: > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233& > myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_- > OCSTXKQY-OCSWJ00-_-E > - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM > Spectrum Scale versions where the NSD server is enabled to use RDMA for > file IO and the storage used in your GPFS cluster accessed via NSD > servers (not fully SAN accessible) includes anything other than IBM > Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under these > conditions, when the RDMA-enabled network adapter fails, the issue may > result in undetected data corruption for file write or read operations. > > ------------------------------------------------------------ > ------------------ > Manage your My Notifications subscriptions, or send questions and comments. > - Subscribe or Unsubscribe - https://www.ibm.com/support/mynotifications > - Feedback - > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotif > ications.html > > - Follow us on Twitter - https://twitter.com/IBMStorageSupt > > > > To ensure proper delivery please add mynotify at stg.events.ihost.com to > your address book. > You received this email because you are subscribed to IBM My > Notifications as: > aaron.s.knister at nasa.gov > > Please do not reply to this message as it is generated by an automated > service machine. > > (C) International Business Machines Corporation 2017. All rights reserved. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Jun 30 18:53:16 2017 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 30 Jun 2017 13:53:16 -0400 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> Message-ID: <2689cf86-eca2-dab6-c6aa-7fc54d923e55@nasa.gov> In fact the answer was quite literally "no": https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523 (the RFE was declined and the answer was that the "function is already available in GNR environments"). Regarding GNR, see this RFE request https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090 requesting the use of GNR outside of an ESS/GSS environment. It's interesting to note this is the highest voted Public RFE for GPFS that I can see, at least. It too was declined. -Aaron On 6/30/17 1:41 PM, Aaron Knister wrote: > Thanks Olaf, that's good to know (and is kind of what I suspected). I've > requested a number of times this capability for those of us who can't > use or aren't using GNR and the answer is effectively "no". This > response is curious to me because I'm sure IBM doesn't believe that data > integrity is only important and of value to customers who purchase their > hardware *and* software. > > -Aaron > > On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser > wrote: > > yes.. in case of GNR (GPFS native raid) .. we do end-to-end > check-summing ... client --> server --> downToDisk > GNR writes down a chksum to disk (to all pdisks /all "raid" segments > ) so that dropped writes can be detected as well as miss-done > writes (bit flips..) > > > > From: Aaron Knister > > To: gpfsug main discussion list > > Date: 06/30/2017 07:15 PM > Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): > RDMA-enabled network adapter failure on the NSD server may result in > file IO error (2017.06.30) > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------ > > > > I'm curious to know why this doesn't affect GSS/ESS? Is it a feature of > the additional check-summing done on those platforms? > > > -------- Forwarded Message -------- > Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network > adapter > failure on the NSD server may result in file IO error (2017.06.30) > Date: Fri, 30 Jun 2017 14:19:02 +0000 > From: IBM My Notifications > > > To: aaron.s.knister at nasa.gov > > > > > My Notifications for Storage - 30 Jun 2017 > > Dear Subscriber (aaron.s.knister at nasa.gov > ), > > Here are your updates from IBM My Notifications. > > Your support Notifications display in English by default. Machine > translation based on your IBM profile > language setting is added if you specify this option in My defaults > within My Notifications. > (Note: Not all languages are available at this time, and the English > version always takes precedence > over the machine translated version.) > > ------------------------------------------------------------------------------ > 1. IBM Spectrum Scale > > - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter > failure > on the NSD server may result in file IO error > - URL: > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E > > - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM > Spectrum Scale versions where the NSD server is enabled to use RDMA for > file IO and the storage used in your GPFS cluster accessed via NSD > servers (not fully SAN accessible) includes anything other than IBM > Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under these > conditions, when the RDMA-enabled network adapter fails, the issue may > result in undetected data corruption for file write or read operations. > > ------------------------------------------------------------------------------ > Manage your My Notifications subscriptions, or send questions and > comments. > - Subscribe or Unsubscribe - > https://www.ibm.com/support/mynotifications > > - Feedback - > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html > > > - Follow us on Twitter - https://twitter.com/IBMStorageSupt > > > > > To ensure proper delivery please add mynotify at stg.events.ihost.com > to > your address book. > You received this email because you are subscribed to IBM My > Notifications as: > aaron.s.knister at nasa.gov > > Please do not reply to this message as it is generated by an automated > service machine. > > (C) International Business Machines Corporation 2017. All rights > reserved. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From oehmes at gmail.com Fri Jun 30 19:25:28 2017 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 30 Jun 2017 18:25:28 +0000 Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30) In-Reply-To: <2689cf86-eca2-dab6-c6aa-7fc54d923e55@nasa.gov> References: <487469581.449569.1498832342497.JavaMail.webinst@w30112> <2689cf86-eca2-dab6-c6aa-7fc54d923e55@nasa.gov> Message-ID: end-to-end data integrity is very important and the reason it hasn't been done in Scale is not because its not important, its because its very hard to do without impacting performance in a very dramatic way. imagine your raid controller blocksize is 1mb and your filesystem blocksize is 1MB . if your application does a 1 MB write this ends up being a perfect full block , full track de-stage to your raid layer and everything works fine and fast. as soon as you add checksum support you need to add data somehow into this, means your 1MB is no longer 1 MB but 1 MB+checksum. to store this additional data you have multiple options, inline , outside the data block or some combination ,the net is either you need to do more physical i/o's to different places to get both the data and the corresponding checksum or your per block on disc structure becomes bigger than than what your application reads/or writes, both put massive burden on the Storage layer as e.g. a 1 MB write will now, even the blocks are all aligned from the application down to the raid layer, cause a read/modify/write on the raid layer as the data is bigger than the physical track size. so to get end-to-end checksum in Scale outside of ESS the best way is to get GNR as SW to run on generic HW, this is what people should vote for as RFE if they need that functionality. beside end-to-end checksums you get read/write cache and acceleration , fast rebuild and many other goodies as a added bonus. Sven On Fri, Jun 30, 2017 at 10:53 AM Aaron Knister wrote: > In fact the answer was quite literally "no": > > https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523 > (the RFE was declined and the answer was that the "function is already > available in GNR environments"). > > Regarding GNR, see this RFE request > https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090 > requesting the use of GNR outside of an ESS/GSS environment. It's > interesting to note this is the highest voted Public RFE for GPFS that I > can see, at least. It too was declined. > > -Aaron > > On 6/30/17 1:41 PM, Aaron Knister wrote: > > Thanks Olaf, that's good to know (and is kind of what I suspected). I've > > requested a number of times this capability for those of us who can't > > use or aren't using GNR and the answer is effectively "no". This > > response is curious to me because I'm sure IBM doesn't believe that data > > integrity is only important and of value to customers who purchase their > > hardware *and* software. > > > > -Aaron > > > > On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser > > wrote: > > > > yes.. in case of GNR (GPFS native raid) .. we do end-to-end > > check-summing ... client --> server --> downToDisk > > GNR writes down a chksum to disk (to all pdisks /all "raid" segments > > ) so that dropped writes can be detected as well as miss-done > > writes (bit flips..) > > > > > > > > From: Aaron Knister > > > > To: gpfsug main discussion list > > > > Date: 06/30/2017 07:15 PM > > Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): > > RDMA-enabled network adapter failure on the NSD server may result in > > file IO error (2017.06.30) > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > ------------------------------------------------------------------------ > > > > > > > > I'm curious to know why this doesn't affect GSS/ESS? Is it a feature > of > > the additional check-summing done on those platforms? > > > > > > -------- Forwarded Message -------- > > Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network > > adapter > > failure on the NSD server may result in file IO error (2017.06.30) > > Date: Fri, 30 Jun 2017 14:19:02 +0000 > > From: IBM My Notifications > > >> > > To: aaron.s.knister at nasa.gov > > > > > > > > > > My Notifications for Storage - 30 Jun 2017 > > > > Dear Subscriber (aaron.s.knister at nasa.gov > > ), > > > > Here are your updates from IBM My Notifications. > > > > Your support Notifications display in English by default. Machine > > translation based on your IBM profile > > language setting is added if you specify this option in My defaults > > within My Notifications. > > (Note: Not all languages are available at this time, and the English > > version always takes precedence > > over the machine translated version.) > > > > > ------------------------------------------------------------------------------ > > 1. IBM Spectrum Scale > > > > - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter > > failure > > on the NSD server may result in file IO error > > - URL: > > > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E > > < > http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E > > > > - ABSTRACT: IBM has identified an issue with all IBM GPFS and IBM > > Spectrum Scale versions where the NSD server is enabled to use RDMA > for > > file IO and the storage used in your GPFS cluster accessed via NSD > > servers (not fully SAN accessible) includes anything other than IBM > > Elastic Storage Server (ESS) or GPFS Storage Server (GSS); under > these > > conditions, when the RDMA-enabled network adapter fails, the issue > may > > result in undetected data corruption for file write or read > operations. > > > > > ------------------------------------------------------------------------------ > > Manage your My Notifications subscriptions, or send questions and > > comments. > > - Subscribe or Unsubscribe - > > https://www.ibm.com/support/mynotifications > > > > - Feedback - > > > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html > > < > https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html > > > > > > - Follow us on Twitter - https://twitter.com/IBMStorageSupt > > > > > > > > > > To ensure proper delivery please add mynotify at stg.events.ihost.com > > to > > your address book. > > You received this email because you are subscribed to IBM My > > Notifications as: > > aaron.s.knister at nasa.gov > > > > Please do not reply to this message as it is generated by an > automated > > service machine. > > > > (C) International Business Machines Corporation 2017. All rights > > reserved. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: