From mnaineni at in.ibm.com Sun Mar 1 07:53:00 2020 From: mnaineni at in.ibm.com (Malahal R Naineni) Date: Sun, 1 Mar 2020 07:53:00 +0000 Subject: [gpfsug-discuss] Thousands of CLOSE_WAIT IPV6 connections on CES In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From scale at us.ibm.com Mon Mar 2 23:58:12 2020 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 2 Mar 2020 18:58:12 -0500 Subject: [gpfsug-discuss] Policy REGEX question In-Reply-To: <3D2C4651-AD7D-40FD-A1B0-1B22D501B0F3@mbari.org> References: <3D2C4651-AD7D-40FD-A1B0-1B22D501B0F3@mbari.org> Message-ID: The third option is to specify the flavor of regex desired. Right now, if specified, must be one of these: 'x','b','f', 'ix','ib' 'x' extended regular expressions - the default - as implemented by regcomp and regexec library functions with REG_EXTENDED flag 'b' basic regular expressions - without REG_EXTENDED 'f' glob like pattern matching as implemented by fnmatch library function The three 'i' are similar, except upper/lower case is ignored as implemented by the library function with the ignore case flag or option: REG_ICASE or FNM_CASEFOLD We haven't published this because it is not yet fully tested. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Todd Ruston To: gpfsug main discussion list Date: 02/20/2020 03:49 PM Subject: [EXTERNAL] [gpfsug-discuss] Policy REGEX question Sent by: gpfsug-discuss-bounces at spectrumscale.org Greetings, I've been working on creating some new policy rules that will require regular expression matching on path names. As a crutch to help me along, I've used the mmfind command to do some searches and used its policy output as a model. Interestingly, it creates REGEX() functions with an undocumented parameter. For example, the following REGEX expression was created in the WHERE clause by mmfind when searching for a pathname pattern: REGEX(PATH_NAME, '*/xy_survey_*/name/*.tif','f') The Scale policy documentation for REGEX only mentions 2 parameters, not 3: REGEX(String,'Pattern') Returns TRUE if the pattern matches, FALSE if it does not. Pattern is a Posix extended regular expression. (The above is from https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_stringfcts.htm ) Anyone know what that 3rd parameter is, what values are allowed there, and what they mean? My assumption is that it's some sort of selector for type of pattern matching engine, because that pattern (2nd parameter) isn't being handled as a standard regex (e.g. the *'s are treated as wildcards, not zero-or-more repeats). -- Todd E. Ruston Information Systems Manager Monterey Bay Aquarium Research Institute (MBARI) 7700 Sandholdt Road, Moss Landing, CA, 95039 Phone 831-775-1997 Fax 831-775-1652 http://www.mbari.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=NvrrhNye2hSn191BaCICEzG0U2qZ9XkpJrE3TCV_poM&s=UjdRykUGhYCQwUNfT9yEv7qI_cCONPk_Nwz21N3RDQk&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From yeep at robust.my Tue Mar 3 07:56:16 2020 From: yeep at robust.my (T.A. Yeep) Date: Tue, 03 Mar 2020 07:56:16 +0000 Subject: [gpfsug-discuss] Hello from Yeep @ Malaysia Message-ID: Hello everyone, My name is Yeep and my company is Robust HPC (www.robusthpc.com). We are building HPC cluster like animation renderfarm, seismic simulation, Bigdata/AI etc in Kuala Lumpur, Malaysia and wish to explore working with industry peers from South East Asia and China. I find Spectrum Scale a really interesting solution for unstructured data, especially with its AFM and TCT features. I'm also excited about pairing it with ICOS for long term archiving as well as how Scale could benefit for IoT deployment. I love meeting people, if anyone happens to come my place please feel free to say hello. =) -- Best regards T.A. Yeep -------------- next part -------------- An HTML attachment was scrubbed... URL: From agostino.funel at enea.it Wed Mar 4 10:05:26 2020 From: agostino.funel at enea.it (Agostino Funel) Date: Wed, 4 Mar 2020 11:05:26 +0100 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 Message-ID: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Hi, we have a GPFS cluster version 4.2.3.19. We have seen in the official "Administration and Programming Reference" (version 4 Release 2.0, pag. 28) that the read-only mount option (-o ro)? is not explicitly cited. This option is supported in version 5.*. However we tried, as a test, the -o ro mount option on two clients with the following SO and kernels client 1 CentOS Linux release 7.3.1611 (Core) uname -r 3.10.0-514.26.2.el7.x86_64 client 2 CentOS release 6.4 (Final) uname -r 2.6.32-358.23.2.el6.x86_64 and it worked fine. The -o ro option is non permanent in the sense that unmounting and remounting the file system on these clients it turned in r/w original state. Now, we have the necessity of setting read-only the file system on all clients of our cluster. The question is: could we safely use the -o ro option for all clients even if this option is not cited in the official (v. 4 release 2.0) documentation? Thank you very much. Best regards, Agostino Funel -- Agostino Funel DTE-ICT-HPC ENEA P.le E. Fermi 1 80055 Portici (Napoli) Italy Phone: (+39) 081-7723575 Fax: (+39) 081-7723344 E-mail: agostino.funel at enea.it WWW: http://www.afs.enea.it/funel ================================================== Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui e' stata inviata e' da qualificarsi quale strumento aziendale. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679). If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks. ================================================== From luis.bolinches at fi.ibm.com Wed Mar 4 11:19:21 2020 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 4 Mar 2020 11:19:21 +0000 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 In-Reply-To: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Message-ID: Hi >From phone so typos are expected. You maybe would like to look into remo mounts and export the FS only as ro to the client cluster. -- Cheers > On 4. Mar 2020, at 12.05, Agostino Funel wrote: > > Hi, > > we have a GPFS cluster version 4.2.3.19. We have seen in the official > "Administration and Programming Reference" (version 4 Release 2.0, pag. > 28) that the read-only mount option (-o ro) is not explicitly cited. > This option is supported in version 5.*. > > However we tried, as a test, the -o ro mount option on two clients with > the following SO and kernels > > client 1 > > CentOS Linux release 7.3.1611 (Core) > uname -r > 3.10.0-514.26.2.el7.x86_64 > > > client 2 > > CentOS release 6.4 (Final) > uname -r > 2.6.32-358.23.2.el6.x86_64 > > and it worked fine. > > The -o ro option is non permanent in the sense that unmounting and > remounting the file system on these clients it turned in r/w original state. > > Now, we have the necessity of setting read-only the file system on all > clients of our cluster. > > The question is: could we safely use the -o ro option for all clients > even if this option is not cited in the official (v. 4 release 2.0) > documentation? > > Thank you very much. > > Best regards, > > Agostino Funel > > > > > -- > Agostino Funel > DTE-ICT-HPC > ENEA > P.le E. Fermi 1 > 80055 Portici (Napoli) Italy > Phone: (+39) 081-7723575 > Fax: (+39) 081-7723344 > E-mail: agostino.funel at enea.it > WWW: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.afs.enea.it_funel&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=06wvjDocXteVI3UQgtzUEF6p7nx9KL9w1H1JZcBbe3w&s=-gYsQFW6qa06tu1S4OkYufeD-zczy5iX1CwrlkH_2HE&e= > > > > ================================================== > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui e' stata inviata e' da qualificarsi quale strumento aziendale. > La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). > Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie. > > This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679). > If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks. > > ================================================== > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=06wvjDocXteVI3UQgtzUEF6p7nx9KL9w1H1JZcBbe3w&s=wmL2HUe83nsiTZ7XmlW5GLpS7JcNytI9wjkJNmrpBgg&e= > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Wed Mar 4 11:24:01 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 4 Mar 2020 12:24:01 +0100 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 In-Reply-To: <87216c24-a638-64d2-f2ed-704985820661@enea.it> References: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Message-ID: I don?t know the answer ? but as an alternative solution, have you considered splitting the read only clients out into a separate cluster. Then you could enforce the read-only setting using ?mmauth grant ... -a ro?. That should be supported. -jf ons. 4. mar. 2020 kl. 12:05 skrev Agostino Funel : > Hi, > > we have a GPFS cluster version 4.2.3.19. We have seen in the official > "Administration and Programming Reference" (version 4 Release 2.0, pag. > 28) that the read-only mount option (-o ro) is not explicitly cited. > This option is supported in version 5.*. > > However we tried, as a test, the -o ro mount option on two clients with > the following SO and kernels > > client 1 > > CentOS Linux release 7.3.1611 (Core) > uname -r > 3.10.0-514.26.2.el7.x86_64 > > > client 2 > > CentOS release 6.4 (Final) > uname -r > 2.6.32-358.23.2.el6.x86_64 > > and it worked fine. > > The -o ro option is non permanent in the sense that unmounting and > remounting the file system on these clients it turned in r/w original > state. > > Now, we have the necessity of setting read-only the file system on all > clients of our cluster. > > The question is: could we safely use the -o ro option for all clients > even if this option is not cited in the official (v. 4 release 2.0) > documentation? > > Thank you very much. > > Best regards, > > Agostino Funel > > > > > -- > Agostino Funel > DTE-ICT-HPC > ENEA > P.le E. Fermi 1 > 80055 Portici (Napoli) Italy > Phone: (+39) 081-7723575 > Fax: (+39) 081-7723344 > E-mail: agostino.funel at enea.it > WWW: http://www.afs.enea.it/funel > > > > ================================================== > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate e la casella di posta elettronica da cui e' stata inviata > e' da qualificarsi quale strumento aziendale. > La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza > di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. > 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). > Qualora abbiate ricevuto questo documento per errore siete cortesemente > pregati di darne immediata comunicazione al mittente e di provvedere alla > sua distruzione. Grazie. > > This e-mail and any attachments is confidential and may contain privileged > information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is unauthorised > (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE > 2016/679). > If you are not the intended recipient, please delete this message and any > attachments and advise the sender by return e-mail. Thanks. > > ================================================== > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Wed Mar 4 21:02:57 2020 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Wed, 4 Mar 2020 15:02:57 -0600 Subject: [gpfsug-discuss] Hello from Yeep @ Malaysia In-Reply-To: References: Message-ID: Hi Yeep, "Hello" and thanks for reaching out to the team. We will keep an eye out for any future specific questions as you evaluate things further. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "T.A. Yeep" To: gpfsug-discuss at spectrumscale.org Date: 03/03/2020 01:56 AM Subject: [EXTERNAL] [gpfsug-discuss] Hello from Yeep @ Malaysia Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello everyone, My name is Yeep and my company is Robust HPC (www.robusthpc.com). We are building HPC cluster like animation renderfarm, seismic simulation, Bigdata/AI etc in Kuala Lumpur, Malaysia and wish to explore working with industry peers from South East Asia and China. I find Spectrum Scale a really interesting solution for unstructured data, especially with its AFM and TCT features. I'm also excited about pairing it with ICOS for long term archiving as well as how Scale could benefit for IoT deployment. I love meeting people, if anyone happens to come my place please feel free to say hello. =) -- Best regards T.A. Yeep _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=vqlz8Ipd9MevOJqKBqp2z36FwoITbgKqFya8GRFnOAE&s=z4TU3uhB83Yc5sWdafnL_SrbEVpo2sXJJcpVJwR1Dos&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From chair at spectrumscale.org Thu Mar 5 22:34:52 2020 From: chair at spectrumscale.org (Simon Thompson (Spectrum Scale User Group Chair)) Date: Thu, 05 Mar 2020 22:34:52 +0000 Subject: [gpfsug-discuss] SSUG Events 2020 update Message-ID: Registration is open for the May UK meeting. We plan to review 3 weeks before the event in w/c 20th April: current situation number of registrations any travel restrictions in place for speakers/IBM/attendees https://www.spectrumscaleug.org/event/worldwide-uk-2020-spectrum-scale-user-group/ There is also a new user day open for registration at: https://www.spectrumscaleug.org/event/worldwide-uk-spectrum-scale-2020-new-user-day/ Simon From: on behalf of "chair at spectrumscale.org" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Friday, 28 February 2020 at 08:55 To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] SSUG Events 2020 update Hi All, I thought it might be giving a little bit of an update on where we are with events this year. As you may know, SCAsia was cancelled in its entirety due to Covid-19 in Singapore and so there was no SSUG meeting. In the US, we struggled to find a venue to host the spring meeting and now time is a little short to arrange something for the end of March planned date. The IBM Spectrum Scale Strategy Days in Germany in March are currently still planned to happen next week. For the UK meeting (May), we haven?t yet opened registration but are planning to do so next week. We currently believe that as an event with 120-130 attendees, this is probably very low risk, but we?ll keep the current government advice under review as we approach the date. I would suggest that if you are planning to travel internationally to the UK event that you delay booking flights/book refundable transport and ensure you have adequate insurance in place in the event we have to cancel the event. For ISC in June, we currently don?t have a date, nor any firm plans to run an event this year. Simon Thompson UK group chair -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurence at qsplace.co.uk Fri Mar 6 18:30:39 2020 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Fri, 6 Mar 2020 18:30:39 +0000 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 In-Reply-To: References: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Message-ID: <9e43ea92-870a-9aa0-c92a-15f553d0fbc1@qsplace.co.uk> Exporting the filesystem to the remove cluster as RO is an excellent idea and works if all remote clients only need RO. Alternatively (and I believe this is still the case) you can set the "ro" filesystem mount option in the file "/var/mmfs/etc/localMountOptions" or "/var/mmfs/etc/localMountOptions." on all nodes. The issue with changing /etc/fstab is that in my experience GPFS has been known to override it. -- Lauz On 04/03/2020 11:24, Jan-Frode Myklebust wrote: > > I don?t know the answer ? but as an alternative solution, have you > considered splitting the read only clients out into a separate > cluster. Then you could enforce the read-only setting using ?mmauth > grant ... -a ro?. > > That should be supported. > > > > > ? -jf > > ons. 4. mar. 2020 kl. 12:05 skrev Agostino Funel > >: > > Hi, > > we have a GPFS cluster version 4.2.3.19. We have seen in the official > "Administration and Programming Reference" (version 4 Release 2.0, > pag. > 28) that the read-only mount option (-o ro)? is not explicitly cited. > This option is supported in version 5.*. > > However we tried, as a test, the -o ro mount option on two clients > with > the following SO and kernels > > client 1 > > CentOS Linux release 7.3.1611 (Core) > uname -r > 3.10.0-514.26.2.el7.x86_64 > > > client 2 > > CentOS release 6.4 (Final) > uname -r > 2.6.32-358.23.2.el6.x86_64 > > and it worked fine. > > The -o ro option is non permanent in the sense that unmounting and > remounting the file system on these clients it turned in r/w > original state. > > Now, we have the necessity of setting read-only the file system on > all > clients of our cluster. > > The question is: could we safely use the -o ro option for all clients > even if this option is not cited in the official (v. 4 release 2.0) > documentation? > > Thank you very much. > > Best regards, > > Agostino Funel > > > > > -- > Agostino Funel > DTE-ICT-HPC > ENEA > P.le E. Fermi 1 > 80055 Portici (Napoli) Italy > Phone: (+39) 081-7723575 > Fax: (+39) 081-7723344 > E-mail: agostino.funel at enea.it > WWW: http://www.afs.enea.it/funel > > > > ================================================== > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente > alle persone indicate e la casella di posta elettronica da cui e' > stata inviata e' da qualificarsi quale strumento aziendale. > La diffusione, copia o qualsiasi altra azione derivante dalla > conoscenza di queste informazioni sono rigorosamente vietate (art. > 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). > Qualora abbiate ricevuto questo documento per errore siete > cortesemente pregati di darne immediata comunicazione al mittente > e di provvedere alla sua distruzione. Grazie. > > This e-mail and any attachments is confidential and may contain > privileged information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is > unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent > amendments and GDPR UE 2016/679). > If you are not the intended recipient, please delete this message > and any attachments and advise the sender by return e-mail. Thanks. > > ================================================== > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Mar 7 00:52:06 2020 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 6 Mar 2020 18:52:06 -0600 Subject: [gpfsug-discuss] =?utf-8?q?Read-only_mount_option_for_GPFS_versio?= =?utf-8?b?bgk0LjIuMy4xOQ==?= In-Reply-To: <9e43ea92-870a-9aa0-c92a-15f553d0fbc1@qsplace.co.uk> References: <87216c24-a638-64d2-f2ed-704985820661@enea.it> <9e43ea92-870a-9aa0-c92a-15f553d0fbc1@qsplace.co.uk> Message-ID: With regard to your question: "The question is: could we safely use the -o ro option for all clients even if this option is not cited in the official (v. 4 release 2.0) documentation?" The answer is yes, because the '-o ro' option comes from the OS mount command and is not specific to Spectrum Scale's mmmount options (thus not documented that way). So you should be fine there. Next, consider setting that option with mmchfs to make it permanent vs on the mmmount command: mmchfs ?[-o MountOptions] or specifically mmchfs -o ro Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Laurence Horrocks-Barlow To: gpfsug-discuss at spectrumscale.org Date: 03/06/2020 01:03 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 Sent by: gpfsug-discuss-bounces at spectrumscale.org Exporting the filesystem to the remove cluster as RO is an excellent idea and works if all remote clients only need RO. Alternatively (and I believe this is still the case) you can set the "ro" filesystem mount option in the file "/var/mmfs/etc/localMountOptions" or "/var/mmfs/etc/localMountOptions." on all nodes. The issue with changing /etc/fstab is that in my experience GPFS has been known to override it. -- Lauz On 04/03/2020 11:24, Jan-Frode Myklebust wrote: I don?t know the answer ? but as an alternative solution, have you considered splitting the read only clients out into a separate cluster. Then you could enforce the read-only setting using ?mmauth grant ... -a ro?. That should be supported. ? -jf ons. 4. mar. 2020 kl. 12:05 skrev Agostino Funel < agostino.funel at enea.it>: Hi, we have a GPFS cluster version 4.2.3.19. We have seen in the official "Administration and Programming Reference" (version 4 Release 2.0, pag. 28) that the read-only mount option (-o ro)? is not explicitly cited. This option is supported in version 5.*. However we tried, as a test, the -o ro mount option on two clients with the following SO and kernels client 1 CentOS Linux release 7.3.1611 (Core) uname -r 3.10.0-514.26.2.el7.x86_64 client 2 CentOS release 6.4 (Final) uname -r 2.6.32-358.23.2.el6.x86_64 and it worked fine. The -o ro option is non permanent in the sense that unmounting and remounting the file system on these clients it turned in r/w original state. Now, we have the necessity of setting read-only the file system on all clients of our cluster. The question is: could we safely use the -o ro option for all clients even if this option is not cited in the official (v. 4 release 2.0) documentation? Thank you very much. Best regards, Agostino Funel -- Agostino Funel DTE-ICT-HPC ENEA P.le E. Fermi 1 80055 Portici (Napoli) Italy Phone: (+39) 081-7723575 Fax: (+39) 081-7723344 E-mail: agostino.funel at enea.it WWW: http://www.afs.enea.it/funel ================================================== Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui e' stata inviata e' da qualificarsi quale strumento aziendale. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679). If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks. ================================================== _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=Fx-WRuJQxi23Sd8N-mxMO4kCN6P0YBMQYWMxiLZhOEw&s=ggxOhPp6D3BSWttqakUi8m_gp61HBPQgrag0WRQ5_oc&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From cabrillo at ifca.unican.es Mon Mar 9 11:48:45 2020 From: cabrillo at ifca.unican.es (Iban Cabrillo) Date: Mon, 9 Mar 2020 12:48:45 +0100 (CET) Subject: [gpfsug-discuss] Gpfs Standar and JBOD enclosures Message-ID: <855324483.13939921.1583754525009.JavaMail.zimbra@ifca.unican.es> Dear, We have a cluster with more than 1PB running Spectrum Scale Standar 4.2.3-9 version. The filesystem that is running now is configured using a DAS systems with NSDs over a NetApp DDP (distributed raid 6). These days have arrived to our hands a pait of JBOB (D3284) enclosures and 4 new servers, the idea is connect it in pairs by two 2 x (1 enclosure + 2 SAS servers). Is there any way to add this enclosures to the actual Cluster using GNR (something similar) o this option is only available for GSS/ESS? If GNR is not an option, Is there any other way to add this enclosures to the cluster keeping the redundacy (path/server)? Maybe one failure group by enclosure with data replica set to 2 (we will lost the 50% of the sapace)? Regards, I -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Mon Mar 9 12:09:48 2020 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Mon, 9 Mar 2020 12:09:48 +0000 Subject: [gpfsug-discuss] Gpfs Standar and JBOD enclosures In-Reply-To: <855324483.13939921.1583754525009.JavaMail.zimbra@ifca.unican.es> Message-ID: Iban, Spectrum scale native raid will not be an option for your scenario. Scale erasure code edition does not support JBOD enclosures at this point. And Scale Native RAID is only certified for specific hardware ( IBM ESS or Lenovo GSS ) This means you either need to use Hardware raid controllers or a shared nothing disk architecture ( I would recommend the raid controllers ) Regards, Andrew Sent from my iPhone > On 9 Mar 2020, at 21:59, Iban Cabrillo wrote: > > ? > Dear, > We have a cluster with more than 1PB running Spectrum Scale Standar 4.2.3-9 version. > > The filesystem that is running now is configured using a DAS systems with NSDs over a NetApp DDP (distributed raid 6). > These days have arrived to our hands a pait of JBOB (D3284) enclosures and 4 new servers, the idea is connect it in pairs by two 2 x (1 enclosure + 2 SAS servers). > Is there any way to add this enclosures to the actual Cluster using GNR (something similar) o this option is only available for GSS/ESS? > If GNR is not an option, Is there any other way to add this enclosures to the cluster keeping the redundacy (path/server)? Maybe one failure group by enclosure with data replica set to 2 (we will lost the 50% of the sapace)? > > Regards, I > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phgrau at zedat.fu-berlin.de Fri Mar 13 12:41:55 2020 From: phgrau at zedat.fu-berlin.de (Philipp Grau) Date: Fri, 13 Mar 2020 13:41:55 +0100 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip "gpfs_maxstatcache_low". Message-ID: <20200313124155.GA94452@CIS.FU-Berlin.DE> Hello, we have a two node NSD cluster based on a DDN system. Currently we run Spectrum Scale 5.0.4.1 in an HPC environment. Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our current settings are: # mmdiag --config | grep -i cache ! maxFilesToCache 3000000 maxStatCache 10000 maxFilesToCache was tuned during installion and maxStatCache is the according default value. After discussing this issue on the german spectumscale meeting, I understand that it is difficult to give a formula on howto calulate this values. But I learnt that a FilesToCache entry costs about 10 kbytes of memory and a StatCache entry about 500 bytes. And typically maxStatCache should (obviously) be greater than maxFilesToCache. There is a average 100 GB memory usage on our systems (with a total of 265 GB RAM). So setting maxStatCache to at least 3000000 should be no problem. But is that correct or to high/low? Has anyone some hints or thoughts on this topic? Help is welcome. Regards, Philipp -- Philipp Grau | Freie Universitaet Berlin phgrau at ZEDAT.FU-Berlin.DE | Zentraleinrichtung fuer Datenverarbeitung Tel: +49 (30) 838 56583 | Fabeckstr. 32 Fax: +49 (30) 838 56721 | 14195 Berlin From Achim.Rehor at de.ibm.com Fri Mar 13 14:00:48 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Fri, 13 Mar 2020 15:00:48 +0100 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: <20200313124155.GA94452@CIS.FU-Berlin.DE> References: <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Fri Mar 13 14:01:18 2020 From: stockf at us.ibm.com (Frederick Stock) Date: Fri, 13 Mar 2020 14:01:18 +0000 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: <20200313124155.GA94452@CIS.FU-Berlin.DE> References: <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: An HTML attachment was scrubbed... URL: From ckrafft at de.ibm.com Fri Mar 13 14:59:52 2020 From: ckrafft at de.ibm.com (Christoph Krafft) Date: Fri, 13 Mar 2020 15:59:52 +0100 Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node Message-ID: Dear Spectrum Scale User Group, can somebody please share with me answers to the following setup / architectural questions: - What size is typically required for a descOnly NSD, that will only hold a copy of a filesystem descriptor area? - Having a 2-node "core-cluster" and a quorum node at a third site with locally attached descOnly NSDs to that q-node. What are the network latency / bandwidth requirements for traffic flowing between core-cluster-nodes and the q-node? (from my understanding the traffic contains primarily mmfsd communication - including FS descOnly updates - but no data/metadata I/O) Thank you in advance for answers - have a nice weekend. Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany IBM Data Privacy Statement IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4F403830.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From Paul.Sanchez at deshaw.com Fri Mar 13 15:06:14 2020 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 13 Mar 2020 15:06:14 +0000 Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node In-Reply-To: References: Message-ID: <86503c8905ea4db7b4f9e2f5c9c11716@deshaw.com> Per https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_nsdfail.htm and confirmed by several instances in a live environment... > This disk would exist solely to contain a replica of the file system descriptor (that is, it would not contain any file system metadata or data). This disk should be at least 128MB in size. From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Christoph Krafft Sent: Friday, March 13, 2020 11:00 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node This message was sent by an external party. Dear Spectrum Scale User Group, can somebody please share with me answers to the following setup / architectural questions: - What size is typically required for a descOnly NSD, that will only hold a copy of a filesystem descriptor area? - Having a 2-node "core-cluster" and a quorum node at a third site with locally attached descOnly NSDs to that q-node. What are the network latency / bandwidth requirements for traffic flowing between core-cluster-nodes and the q-node? (from my understanding the traffic contains primarily mmfsd communication - including FS descOnly updates - but no data/metadata I/O) Thank you in advance for answers - have a nice weekend. Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group ________________________________ Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH [cid:image002.gif at 01D5F927.69089AB0] Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany ________________________________ IBM Data Privacy Statement IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 166 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 1851 bytes Desc: image002.gif URL: From knop at us.ibm.com Fri Mar 13 15:21:22 2020 From: knop at us.ibm.com (Felipe Knop) Date: Fri, 13 Mar 2020 15:21:22 +0000 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: References: , <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Fri Mar 13 17:31:43 2020 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Fri, 13 Mar 2020 17:31:43 +0000 Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node In-Reply-To: <86503c8905ea4db7b4f9e2f5c9c11716@deshaw.com> References: <86503c8905ea4db7b4f9e2f5c9c11716@deshaw.com>, Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.image001.png at 01D5F927.69089AB0.png Type: image/png Size: 166 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.image002.gif at 01D5F927.69089AB0.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From phgrau at zedat.fu-berlin.de Sat Mar 14 13:12:11 2020 From: phgrau at zedat.fu-berlin.de (Philipp Grau) Date: Sat, 14 Mar 2020 14:12:11 +0100 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: References: <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: <20200314131211.GA717391@CIS.FU-Berlin.DE> Hello all, thank you all a lot for the feedback to my question. I think that I now understand the situation better. I will talk with my coworkers and we will find a better setting for the values... Many thanks, Philipp * Felipe Knop [13.03.20 16:22]: > All, > > Looks to me that the demands of the workload will dictate how many files we > should be cache, that is: maxStatCache + maxFilesToCache . > > The "mix" between maxStatCache and maxFilesToCache depends on how much memory > can be made available. Accessing files from maxFilesToCache is more efficient, > but stat cache entries use much less space. > > With the > > ! maxFilesToCache 3000000 > maxStatCache 10000 > > combination, the stat cache is not providing any significant help, since only > 0.3% of the files that are cached can fit in the stat cache. If enough memory > is available then maxStatCache could be increased to (say) 3000000, at a cost > of 1.4GB. But maxFilesToCache = 3000000 uses up to 27GB. The next questions > are then > > 1) Can such memory become available on the node, given the pagepool size ? > > 2) Does the workload require caching that many files? > > > Felipe > > ---- > Felipe Knop knop at us.ibm.com > GPFS Development and Security > IBM Systems > IBM Building 008 > 2455 South Rd, Poughkeepsie, NY 12601 > (845) 433-9314 T/L 293-9314 > > > > > ----- Original message ----- > From: "Frederick Stock" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip"gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 10:01 AM > > As you have learned there is no simple formula for setting the > maxStatToCache, or for that matter the maxFilesToCache, configuration > values. Memory is certainly one consideration but another is directory > listing operations. The information kept in the stat cache is sufficient > for fulfilling directory listings. If your users are doing directory > listings regularly then a larger stat cache could be helpful. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > stockf at us.ibm.com > > > > ----- Original message ----- > From: Philipp Grau > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: [EXTERNAL] [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip "gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 8:49 AM > > Hello, > > we have a two node NSD cluster based on a DDN system. Currently we > run Spectrum Scale 5.0.4.1 in an HPC environment. > > Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our current > settings are: > > # mmdiag --config | grep -i cache > ! maxFilesToCache 3000000 > maxStatCache 10000 > > maxFilesToCache was tuned during installion and maxStatCache is the > according default value. > > After discussing this issue on the german spectumscale meeting, I > understand that it is difficult to give a formula on howto calulate > this values. > > But I learnt that a FilesToCache entry costs about 10 kbytes of memory > and a StatCache entry about 500 bytes. And typically maxStatCache > should (obviously) be greater than maxFilesToCache. There is a average > 100 GB memory usage on our systems (with a total of 265 GB RAM). > > So setting maxStatCache to at least 3000000 should be no problem. But > is that correct or to high/low? > > Has anyone some hints or thoughts on this topic? Help is welcome. > > Regards, > > Philipp > From TOMP at il.ibm.com Mon Mar 16 08:49:00 2020 From: TOMP at il.ibm.com (Tomer Perry) Date: Mon, 16 Mar 2020 10:49:00 +0200 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: <20200314131211.GA717391@CIS.FU-Berlin.DE> References: <20200313124155.GA94452@CIS.FU-Berlin.DE> <20200314131211.GA717391@CIS.FU-Berlin.DE> Message-ID: Hi, Just remember to take into account the token memory aspect of the chosen values, especially for large clusters ( check https://www.spectrumscaleug.org/wp-content/uploads/2019/05/SSSD19DE-Day-2-B04-Spectrum-Scale-Memory-Usage.pdf for more details). Regards, Tomer Perry From: Philipp Grau To: gpfsug main discussion list Date: 14/03/2020 15:12 Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all, thank you all a lot for the feedback to my question. I think that I now understand the situation better. I will talk with my coworkers and we will find a better setting for the values... Many thanks, Philipp * Felipe Knop [13.03.20 16:22]: > All, > > Looks to me that the demands of the workload will dictate how many files we > should be cache, that is: maxStatCache + maxFilesToCache . > > The "mix" between maxStatCache and maxFilesToCache depends on how much memory > can be made available. Accessing files from maxFilesToCache is more efficient, > but stat cache entries use much less space. > > With the > > ! maxFilesToCache 3000000 > maxStatCache 10000 > > combination, the stat cache is not providing any significant help, since only > 0.3% of the files that are cached can fit in the stat cache. If enough memory > is available then maxStatCache could be increased to (say) 3000000, at a cost > of 1.4GB. But maxFilesToCache = 3000000 uses up to 27GB. The next questions > are then > > 1) Can such memory become available on the node, given the pagepool size ? > > 2) Does the workload require caching that many files? > > > Felipe > > ---- > Felipe Knop knop at us.ibm.com > GPFS Development and Security > IBM Systems > IBM Building 008 > 2455 South Rd, Poughkeepsie, NY 12601 > (845) 433-9314 T/L 293-9314 > > > > > ----- Original message ----- > From: "Frederick Stock" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip"gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 10:01 AM > > As you have learned there is no simple formula for setting the > maxStatToCache, or for that matter the maxFilesToCache, configuration > values. Memory is certainly one consideration but another is directory > listing operations. The information kept in the stat cache is sufficient > for fulfilling directory listings. If your users are doing directory > listings regularly then a larger stat cache could be helpful. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > stockf at us.ibm.com > > > > ----- Original message ----- > From: Philipp Grau > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: [EXTERNAL] [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip "gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 8:49 AM > > Hello, > > we have a two node NSD cluster based on a DDN system. Currently we > run Spectrum Scale 5.0.4.1 in an HPC environment. > > Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our current > settings are: > > # mmdiag --config | grep -i cache > ! maxFilesToCache 3000000 > maxStatCache 10000 > > maxFilesToCache was tuned during installion and maxStatCache is the > according default value. > > After discussing this issue on the german spectumscale meeting, I > understand that it is difficult to give a formula on howto calulate > this values. > > But I learnt that a FilesToCache entry costs about 10 kbytes of memory > and a StatCache entry about 500 bytes. And typically maxStatCache > should (obviously) be greater than maxFilesToCache. There is a average > 100 GB memory usage on our systems (with a total of 265 GB RAM). > > So setting maxStatCache to at least 3000000 should be no problem. But > is that correct or to high/low? > > Has anyone some hints or thoughts on this topic? Help is welcome. > > Regards, > > Philipp > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=mLPyKeOa1gNDrORvEXBgMw&m=OfAPrgGhpUsG419cks6FynnKaWgb84-BkzQXkaxraJw&s=HFKr-6vELN-qgWlEQsLH74gFoLShJnfLN89xNRNGocQ&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Mar 16 22:10:11 2020 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Mon, 16 Mar 2020 22:10:11 +0000 Subject: [gpfsug-discuss] SSUG Events 2020 update In-Reply-To: References: Message-ID: <7C0DB0DD-EEE5-412D-864E-E1048E46E71E@bham.ac.uk> Unfortunately, due to the escalating COVID-19 situation, this year we?re having to cancel the London user group meeting. We?re aware that IBM has cancelled a number of events through to the end of Q2, and whilst we?re independent of IBM, we rely on them to support the group with speakers and so we?ve taken the decision to cancel. Having spoken briefly with Ulf, Kristy and Bob, we?re hoping to bring a digital event of some sort though we need to work out the logistics of doing this and how we make sure it works for the community. We all recognise that the in-person meetings are better, but this year it looks like we?re going to struggle to run one for the foreseeable future! Thanks go to the sponsors who?d signed up to support us for this year?s UK event ? OCF, Lenovo, StarFish and NVIDIA. Simon UK Group Chair From: on behalf of "chair at spectrumscale.org" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Thursday, 5 March 2020 at 22:35 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] SSUG Events 2020 update Registration is open for the May UK meeting. We plan to review 3 weeks before the event in w/c 20th April: * current situation * number of registrations * any travel restrictions in place for speakers/IBM/attendees https://www.spectrumscaleug.org/event/worldwide-uk-2020-spectrum-scale-user-group/ There is also a new user day open for registration at: https://www.spectrumscaleug.org/event/worldwide-uk-spectrum-scale-2020-new-user-day/ Simon From: on behalf of "chair at spectrumscale.org" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Friday, 28 February 2020 at 08:55 To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] SSUG Events 2020 update Hi All, I thought it might be giving a little bit of an update on where we are with events this year. * As you may know, SCAsia was cancelled in its entirety due to Covid-19 in Singapore and so there was no SSUG meeting. * In the US, we struggled to find a venue to host the spring meeting and now time is a little short to arrange something for the end of March planned date. * The IBM Spectrum Scale Strategy Days in Germany in March are currently still planned to happen next week. * For the UK meeting (May), we haven?t yet opened registration but are planning to do so next week. We currently believe that as an event with 120-130 attendees, this is probably very low risk, but we?ll keep the current government advice under review as we approach the date. I would suggest that if you are planning to travel internationally to the UK event that you delay booking flights/book refundable transport and ensure you have adequate insurance in place in the event we have to cancel the event. * For ISC in June, we currently don?t have a date, nor any firm plans to run an event this year. Simon Thompson UK group chair -------------- next part -------------- An HTML attachment was scrubbed... URL: From sandeep.patil at in.ibm.com Tue Mar 17 08:07:58 2020 From: sandeep.patil at in.ibm.com (Sandeep Ramesh) Date: Tue, 17 Mar 2020 08:07:58 +0000 Subject: [gpfsug-discuss] Latest Technical Blogs/Papers on IBM Spectrum Scale (Q3 2019 - Q1 2020) In-Reply-To: References: Message-ID: Dear User Group Members, In continuation to this email thread, here are list of development blogs/Redpaper in the past 2 quarters . We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to this list. Redpaper HIPAA Compliance for Healthcare Workloads on IBM Spectrum Scale http://www.redbooks.ibm.com/abstracts/redp5591.html?Open IBM Spectrum Scale CSI Driver For Container Persistent Storage http://www.redbooks.ibm.com/redpieces/abstracts/redp5589.html?Open Cyber Resiliency Solution for IBM Spectrum Scale , Blueprint http://www.redbooks.ibm.com/abstracts/redp5559.html?Open Enhanced Cyber Security with IBM Spectrum Scale and IBM QRadar http://www.redbooks.ibm.com/abstracts/redp5560.html?Open Monitoring and Managing the IBM Elastic Storage Server Using the GUI http://www.redbooks.ibm.com/abstracts/redp5471.html?Open IBM Hybrid Solution for Scalable Data Solutions using IBM Spectrum Scale http://www.redbooks.ibm.com/abstracts/redp5549.html?Open IBM Spectrum Discover: Metadata Management for Deep Insight of Unstructured Storage http://www.redbooks.ibm.com/abstracts/redp5550.html?Open Monitoring and Managing IBM Spectrum Scale Using the GUI http://www.redbooks.ibm.com/abstracts/redp5458.html?Open IBM Reference Architecture for High Performance Data and AI in Healthcare and Life Sciences, http://www.redbooks.ibm.com/abstracts/redp5481.html?Open Blogs: Why Storage and HIPAA Compliance for AI & Analytics Workloads for Healthcare https://developer.ibm.com/storage/2020/03/17/why-storage-and-hipaa-compliance-for-ai-analytics-workloads-for-healthcare/ Innovation via Integration ? Proactively Securing Your Unstructured Data from Cyber Threats & Attacks --> This was done based on your inputs (as a part of Security Survey) last year on need for Spectrum Scale integrayion with IDS a https://developer.ibm.com/storage/2020/02/24/innovation-via-integration-proactively-securing-your-unstructured-data-from-cyber-threats-attacks/ IBM Spectrum Scale CES HDFS Transparency support https://developer.ibm.com/storage/2020/02/03/ces-hdfs-transparency-support/ How to set up a remote cluster with IBM Spectrum Scale ? steps, limitations and troubleshooting https://developer.ibm.com/storage/2020/01/27/how-to-set-up-a-remote-cluster-with-ibm-spectrum-scale-steps-limitations-and-troubleshooting/ How to use IBM Spectrum Scale with CSI Operator 1.0 on Openshift 4.2 ? sample usage scenario with Tensorflow deployment https://developer.ibm.com/storage/2020/01/20/how-to-use-ibm-spectrum-scale-with-csi-operator-1-0-on-openshift-4-2-sample-usage-scenario-with-tensorflow-deployment/ Achieving WORM like functionality from NFS/SMB clients for data on Spectrum Scale https://developer.ibm.com/storage/2020/01/10/achieving-worm-like-functionality-from-nfs-smb-clients-for-data-on-spectrum-scale/ IBM Spectrum Scale CSI driver video blogs, https://developer.ibm.com/storage/2019/12/26/ibm-spectrum-scale-csi-driver-video-blogs/ IBM Spectrum Scale CSI Driver v1.0.0 released https://developer.ibm.com/storage/2019/12/10/ibm-spectrum-scale-csi-driver-v1-0-0-released/ Now configure IBM? Spectrum Scale with Overlapping UNIXMAP ranges https://developer.ibm.com/storage/2019/11/12/now-configure-ibm-spectrum-scale-with-overlapping-unixmap-ranges/ ?mmadquery?, a Powerful tool helps check AD settings from Spectrum Scale https://developer.ibm.com/storage/2019/11/11/mmadquery-a-powerful-tool-helps-check-ad-settings-from-spectrum-scale/ Spectrum Scale Data Security Modes, https://developer.ibm.com/storage/2019/10/31/spectrum-scale-data-security-modes/ IBM Spectrum Scale for Linux on IBM Z ? What?s new in IBM Spectrum Scale 5.0.4 ? https://developer.ibm.com/storage/2019/10/25/ibm-spectrum-scale-for-linux-on-ibm-z-whats-new-in-ibm-spectrum-scale-5-0-4/ IBM Spectrum Scale installation toolkit ? enhancements over releases ? 5.0.4.0 https://developer.ibm.com/storage/2019/10/18/ibm-spectrum-scale-installation-toolkit-enhancements-over-releases-5-0-4-0/ IBM Spectrum Scale CSI driver beta on GitHub, https://developer.ibm.com/storage/2019/09/26/ibm-spectrum-scale-csi-driver-on-github/ Help Article: Care to be taken when configuring AD with RFC2307 https://developer.ibm.com/storage/2019/09/18/help-article-care-to-be-taken-when-configuring-ad-with-rfc2307/ IBM Spectrum Scale Erasure Code Edition (ECE): Installation Demonstration https://developer.ibm.com/storage/2019/09/10/ibm-spectrum-scale-erasure-code-edition-ece-installation-demonstration/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 09/03/2019 10:58 AM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q2 2019) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q2 2019). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. Redpaper : IBM Power Systems Enterprise AI Solutions (W/ SPECTRUM SCALE) http://www.redbooks.ibm.com/redpieces/abstracts/redp5556.html?Open IBM Spectrum Scale Erasure Code Edition (ECE): Installation Demonstration https://www.youtube.com/watch?v=6If50EvgP-U Blogs: Using IBM Spectrum Scale as platform storage for running containerized Hadoop/Spark workloads https://developer.ibm.com/storage/2019/08/27/using-ibm-spectrum-scale-as-platform-storage-for-running-containerized-hadoop-spark-workloads/ Useful Tools for Spectrum Scale CES NFS https://developer.ibm.com/storage/2019/07/22/useful-tools-for-spectrum-scale-ces-nfs/ How to ensure NFS uses strong encryption algorithms for secure data in motion ? https://developer.ibm.com/storage/2019/07/19/how-to-ensure-nfs-uses-strong-encryption-algorithms-for-secure-data-in-motion/ Introducing IBM Spectrum Scale Erasure Code Edition https://developer.ibm.com/storage/2019/07/07/introducing-ibm-spectrum-scale-erasure-code-edition/ Spectrum Scale: Which Filesystem Encryption Algo to Consider ? https://developer.ibm.com/storage/2019/07/01/spectrum-scale-which-filesystem-encryption-algo-to-consider/ IBM Spectrum Scale HDFS Transparency Apache Hadoop 3.1.x Support https://developer.ibm.com/storage/2019/06/24/ibm-spectrum-scale-hdfs-transparency-apache-hadoop-3-0-x-support/ Enhanced features in Elastic Storage Server (ESS) 5.3.4 https://developer.ibm.com/storage/2019/06/19/enhanced-features-in-elastic-storage-server-ess-5-3-4/ Upgrading IBM Spectrum Scale Erasure Code Edition using installation toolkit https://developer.ibm.com/storage/2019/06/09/upgrading-ibm-spectrum-scale-erasure-code-edition-using-installation-toolkit/ Upgrading IBM Spectrum Scale sync replication / stretch cluster setup in PureApp https://developer.ibm.com/storage/2019/06/06/upgrading-ibm-spectrum-scale-sync-replication-stretch-cluster-setup/ GPFS config remote access with multiple network definitions https://developer.ibm.com/storage/2019/05/30/gpfs-config-remote-access-with-multiple-network-definitions/ IBM Spectrum Scale Erasure Code Edition Fault Tolerance https://developer.ibm.com/storage/2019/05/30/ibm-spectrum-scale-erasure-code-edition-fault-tolerance/ IBM Spectrum Scale for Linux on IBM Z ? What?s new in IBM Spectrum Scale 5.0.3 ? https://developer.ibm.com/storage/2019/05/02/ibm-spectrum-scale-for-linux-on-ibm-z-whats-new-in-ibm-spectrum-scale-5-0-3/ Understanding and Solving WBC_ERR_DOMAIN_NOT_FOUND error with Spectrum Scale https://crk10.wordpress.com/2019/07/21/solving-the-wbc-err-domain-not-found-nt-status-none-mapped-glitch-in-ibm-spectrum-scale/ Understanding and Solving NT_STATUS_INVALID_SID issue for SMB access with Spectrum Scale https://crk10.wordpress.com/2019/07/24/solving-nt_status_invalid_sid-for-smb-share-access-in-ibm-spectrum-scale/ mmadquery primer (apparatus to query Active Directory from IBM Spectrum Scale) https://crk10.wordpress.com/2019/07/27/mmadquery-primer-apparatus-to-query-active-directory-from-ibm-spectrum-scale/ How to configure RHEL host as Active Directory Client using SSSD https://crk10.wordpress.com/2019/07/28/configure-rhel-machine-as-active-directory-client-using-sssd/ How to configure RHEL host as LDAP client using nslcd https://crk10.wordpress.com/2019/07/28/configure-rhel-machine-as-ldap-client-using-nslcd/ Solving NFSv4 AUTH_SYS nobody ownership issue https://crk10.wordpress.com/2019/07/29/nfsv4-auth_sys-nobody-ownership-and-idmapd/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ Consolidation list of all blogs and collaterals. https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Blogs%2C%20White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 04/29/2019 12:12 PM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q1 2019) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q1 2019). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. Spectrum Scale 5.0.3 https://developer.ibm.com/storage/2019/04/24/spectrum-scale-5-0-3/ IBM Spectrum Scale HDFS Transparency Ranger Support https://developer.ibm.com/storage/2019/04/01/ibm-spectrum-scale-hdfs-transparency-ranger-support/ Integration of IBM Aspera Sync with IBM Spectrum Scale: Protecting and Sharing Files Globally, http://www.redbooks.ibm.com/abstracts/redp5527.html?Open Spectrum Scale user group in Singapore, 2019 https://developer.ibm.com/storage/2019/03/14/spectrum-scale-user-group-in-singapore-2019/ 7 traits to use Spectrum Scale to run container workload https://developer.ibm.com/storage/2019/02/26/7-traits-to-use-spectrum-scale-to-run-container-workload/ Health Monitoring of IBM Spectrum Scale Cluster via External Monitoring Framework https://developer.ibm.com/storage/2019/01/22/health-monitoring-of-ibm-spectrum-scale-cluster-via-external-monitoring-framework/ Migrating data from native HDFS to IBM Spectrum Scale based shared storage https://developer.ibm.com/storage/2019/01/18/migrating-data-from-native-hdfs-to-ibm-spectrum-scale-based-shared-storage/ Bulk File Creation useful for Test on Filesystems https://developer.ibm.com/storage/2019/01/16/bulk-file-creation-useful-for-test-on-filesystems/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Blogs%2C%20White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 01/14/2019 06:24 PM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q4 2018) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q4 2018). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. Redpaper: IBM Spectrum Scale and IBM StoredIQ: Identifying and securing your business data to support regulatory requirements http://www.redbooks.ibm.com/abstracts/redp5525.html?Open IBM Spectrum Scale Memory Usage https://www.slideshare.net/tomerperry/ibm-spectrum-scale-memory-usage?qid=50a1dfda-3102-484f-b9d0-14b69fc4800b&v=&b=&from_search=2 Spectrum Scale and Containers https://developer.ibm.com/storage/2018/12/20/spectrum-scale-and-containers/ IBM Elastic Storage Server Performance Graphical Visualization with Grafana https://developer.ibm.com/storage/2018/12/18/ibm-elastic-storage-server-performance-graphical-visualization-with-grafana/ Hadoop Performance for disaggregated compute and storage configurations based on IBM Spectrum Scale Storage https://developer.ibm.com/storage/2018/12/13/hadoop-performance-for-disaggregated-compute-and-storage-configurations-based-on-ibm-spectrum-scale-storage/ EMS HA in ESS LE (Little Endian) environment https://developer.ibm.com/storage/2018/12/07/ems-ha-in-ess-le-little-endian-environment/ What?s new in ESS 5.3.2 https://developer.ibm.com/storage/2018/12/04/whats-new-in-ess-5-3-2/ Administer your Spectrum Scale cluster easily https://developer.ibm.com/storage/2018/11/13/administer-your-spectrum-scale-cluster-easily/ Disaster Recovery using Spectrum Scale?s Active File Management https://developer.ibm.com/storage/2018/11/13/disaster-recovery-using-spectrum-scales-active-file-management/ Recovery Group Failover Procedure of IBM Elastic Storage Server (ESS) https://developer.ibm.com/storage/2018/10/08/recovery-group-failover-procedure-ibm-elastic-storage-server-ess/ Whats new in IBM Elastic Storage Server (ESS) Version 5.3.1 and 5.3.1.1 https://developer.ibm.com/storage/2018/10/04/whats-new-ibm-elastic-storage-server-ess-version-5-3-1-5-3-1-1/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Blogs%2C%20White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 10/03/2018 08:48 PM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q3 2018) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q3 2018). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. How NFS exports became more dynamic with Spectrum Scale 5.0.2 https://developer.ibm.com/storage/2018/10/02/nfs-exports-became-dynamic-spectrum-scale-5-0-2/ HPC storage on AWS (IBM Spectrum Scale) https://developer.ibm.com/storage/2018/10/02/hpc-storage-aws-ibm-spectrum-scale/ Upgrade with Excluding the node(s) using Install-toolkit https://developer.ibm.com/storage/2018/09/30/upgrade-excluding-nodes-using-install-toolkit/ Offline upgrade using Install-toolkit https://developer.ibm.com/storage/2018/09/30/offline-upgrade-using-install-toolkit/ IBM Spectrum Scale for Linux on IBM Z ? What?s new in IBM Spectrum Scale 5.0.2 ? https://developer.ibm.com/storage/2018/09/21/ibm-spectrum-scale-for-linux-on-ibm-z-whats-new-in-ibm-spectrum-scale-5-0-2/ What?s New in IBM Spectrum Scale 5.0.2 ? https://developer.ibm.com/storage/2018/09/15/whats-new-ibm-spectrum-scale-5-0-2/ Starting IBM Spectrum Scale 5.0.2 release, the installation toolkit supports upgrade rerun if fresh upgrade fails. https://developer.ibm.com/storage/2018/09/15/starting-ibm-spectrum-scale-5-0-2-release-installation-toolkit-supports-upgrade-rerun-fresh-upgrade-fails/ IBM Spectrum Scale installation toolkit ? enhancements over releases ? 5.0.2.0 https://developer.ibm.com/storage/2018/09/15/ibm-spectrum-scale-installation-toolkit-enhancements-releases-5-0-2-0/ Announcing HDP 3.0 support with IBM Spectrum Scale https://developer.ibm.com/storage/2018/08/31/announcing-hdp-3-0-support-ibm-spectrum-scale/ IBM Spectrum Scale Tuning Overview for Hadoop Workload https://developer.ibm.com/storage/2018/08/20/ibm-spectrum-scale-tuning-overview-hadoop-workload/ Making the Most of Multicloud Storage https://developer.ibm.com/storage/2018/08/13/making-multicloud-storage/ Disaster Recovery for Transparent Cloud Tiering using SOBAR https://developer.ibm.com/storage/2018/08/13/disaster-recovery-transparent-cloud-tiering-using-sobar/ Your Optimal Choice of AI Storage for Today and Tomorrow https://developer.ibm.com/storage/2018/08/10/spectrum-scale-ai-workloads/ Analyze IBM Spectrum Scale File Access Audit with ELK Stack https://developer.ibm.com/storage/2018/07/30/analyze-ibm-spectrum-scale-file-access-audit-elk-stack/ Mellanox SX1710 40G switch MLAG configuration for IBM ESS https://developer.ibm.com/storage/2018/07/12/mellanox-sx1710-40g-switcher-mlag-configuration/ Protocol Problem Determination Guide for IBM Spectrum Scale? ? SMB and NFS Access issues https://developer.ibm.com/storage/2018/07/10/protocol-problem-determination-guide-ibm-spectrum-scale-smb-nfs-access-issues/ Access Control in IBM Spectrum Scale Object https://developer.ibm.com/storage/2018/07/06/access-control-ibm-spectrum-scale-object/ IBM Spectrum Scale HDFS Transparency Docker support https://developer.ibm.com/storage/2018/07/06/ibm-spectrum-scale-hdfs-transparency-docker-support/ Protocol Problem Determination Guide for IBM Spectrum Scale? ? Log Collection https://developer.ibm.com/storage/2018/07/04/protocol-problem-determination-guide-ibm-spectrum-scale-log-collection/ Redpapers IBM Spectrum Scale Immutability Introduction, Configuration Guidance, and Use Cases http://www.redbooks.ibm.com/abstracts/redp5507.html?Open Certifications Assessment of the immutability function of IBM Spectrum Scale Version 5.0 in accordance to US SEC17a-4f, EU GDPR Article 21 Section 1, German and Swiss laws and regulations in collaboration with KPMG. Certificate: http://www.kpmg.de/bescheinigungen/RequestReport.aspx?DE968667B47544FF83F6CCDCF37E5FB5 Full assessment report: http://www.kpmg.de/bescheinigungen/RequestReport.aspx?B290411BE1224F5A9B4D24663BCD3C5D For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 07/03/2018 12:13 AM Subject: Re: Latest Technical Blogs on Spectrum Scale (Q2 2018) Dear User Group Members, In continuation , here are list of development blogs in the this quarter (Q2 2018). We now have over 100+ developer blogs. As discussed in User Groups, passing it along: IBM Spectrum Scale 5.0.1 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2018/06/15/6494/ IBM Spectrum Scale ILM Policies https://developer.ibm.com/storage/2018/06/02/ibm-spectrum-scale-ilm-policies/ IBM Spectrum Scale 5.0.1 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2018/06/15/6494/ Management GUI enhancements in IBM Spectrum Scale release 5.0.1 https://developer.ibm.com/storage/2018/05/18/management-gui-enhancements-in-ibm-spectrum-scale-release-5-0-1/ Managing IBM Spectrum Scale services through GUI https://developer.ibm.com/storage/2018/05/18/managing-ibm-spectrum-scale-services-through-gui/ Use AWS CLI with IBM Spectrum Scale? object storage https://developer.ibm.com/storage/2018/05/16/use-awscli-with-ibm-spectrum-scale-object-storage/ Hadoop Storage Tiering with IBM Spectrum Scale https://developer.ibm.com/storage/2018/05/09/hadoop-storage-tiering-ibm-spectrum-scale/ How many Files on my Filesystem? https://developer.ibm.com/storage/2018/05/07/many-files-filesystem/ Recording Spectrum Scale Object Stats for Potential Billing like Purpose using Elasticsearch https://developer.ibm.com/storage/2018/05/04/spectrum-scale-object-stats-for-billing-using-elasticsearch/ New features in IBM Elastic Storage Server (ESS) Version 5.3 https://developer.ibm.com/storage/2018/04/09/new-features-ibm-elastic-storage-server-ess-version-5-3/ Using IBM Spectrum Scale for storage in IBM Cloud Private (Missed to send earlier) https://medium.com/ibm-cloud/ibm-spectrum-scale-with-ibm-cloud-private-8bf801796f19 Redpapers Hortonworks Data Platform with IBM Spectrum Scale: Reference Guide for Building an Integrated Solution http://www.redbooks.ibm.com/redpieces/abstracts/redp5448.html, Enabling Hybrid Cloud Storage for IBM Spectrum Scale Using Transparent Cloud Tiering http://www.redbooks.ibm.com/abstracts/redp5411.html?Open SAP HANA and ESS: A Winning Combination (Update) http://www.redbooks.ibm.com/abstracts/redp5436.html?Open Others IBM Spectrum Scale Software Version Recommendation Preventive Service Planning (Updated) http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009703, IDC Infobrief: A Modular Approach to Genomics Infrastructure at Scale in HCLS https://www.ibm.com/common/ssi/cgi-bin/ssialias?htmlfid=37016937USEN& For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 03/27/2018 05:23 PM Subject: Re: Latest Technical Blogs on Spectrum Scale Dear User Group Members, In continuation , here are list of development blogs in the this quarter (Q1 2018). As discussed in User Groups, passing it along: GDPR Compliance and Unstructured Data Storage https://developer.ibm.com/storage/2018/03/27/gdpr-compliance-unstructure-data-storage/ IBM Spectrum Scale for Linux on IBM Z ? Release 5.0 features and highlights https://developer.ibm.com/storage/2018/03/09/ibm-spectrum-scale-linux-ibm-z-release-5-0-features-highlights/ Management GUI enhancements in IBM Spectrum Scale release 5.0.0 https://developer.ibm.com/storage/2018/01/18/gui-enhancements-in-spectrum-scale-release-5-0-0/ IBM Spectrum Scale 5.0.0 ? What?s new in NFS? https://developer.ibm.com/storage/2018/01/18/ibm-spectrum-scale-5-0-0-whats-new-nfs/ Benefits and implementation of Spectrum Scale sudo wrappers https://developer.ibm.com/storage/2018/01/15/benefits-implementation-spectrum-scale-sudo-wrappers/ IBM Spectrum Scale: Big Data and Analytics Solution Brief https://developer.ibm.com/storage/2018/01/15/ibm-spectrum-scale-big-data-analytics-solution-brief/ Variant Sub-blocks in Spectrum Scale 5.0 https://developer.ibm.com/storage/2018/01/11/spectrum-scale-variant-sub-blocks/ Compression support in Spectrum Scale 5.0.0 https://developer.ibm.com/storage/2018/01/11/compression-support-spectrum-scale-5-0-0/ IBM Spectrum Scale Versus Apache Hadoop HDFS https://developer.ibm.com/storage/2018/01/10/spectrumscale_vs_hdfs/ ESS Fault Tolerance https://developer.ibm.com/storage/2018/01/09/ess-fault-tolerance/ Genomic Workloads ? How To Get it Right From Infrastructure Point Of View. https://developer.ibm.com/storage/2018/01/06/genomic-workloads-get-right-infrastructure-point-view/ IBM Spectrum Scale On AWS Cloud : This video explains how to deploy IBM Spectrum Scale on AWS. This solution helps the users who require highly available access to a shared name space across multiple instances with good performance, without requiring an in-depth knowledge of IBM Spectrum Scale. Detailed Demo : https://www.youtube.com/watch?v=6j5Xj_d0bh4 Brief Demo : https://www.youtube.com/watch?v=-aMQKPW_RfY. For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Cc: Doris Conti/Poughkeepsie/IBM at IBMUS Date: 01/10/2018 12:13 PM Subject: Re: Latest Technical Blogs on Spectrum Scale Dear User Group Members, Here are list of development blogs in the last quarter. Passing it to this email group as Doris had got a feedback in the UG meetings to notify the members with the latest updates periodically. Genomic Workloads ? How To Get it Right From Infrastructure Point Of View. https://developer.ibm.com/storage/2018/01/06/genomic-workloads-get-right-infrastructure-point-view/ IBM Spectrum Scale Versus Apache Hadoop HDFS https://developer.ibm.com/storage/2018/01/10/spectrumscale_vs_hdfs/ ESS Fault Tolerance https://developer.ibm.com/storage/2018/01/09/ess-fault-tolerance/ IBM Spectrum Scale MMFSCK ? Savvy Enhancements https://developer.ibm.com/storage/2018/01/05/ibm-spectrum-scale-mmfsck-savvy-enhancements/ ESS Disk Management https://developer.ibm.com/storage/2018/01/02/ess-disk-management/ IBM Spectrum Scale Object Protocol On Ubuntu https://developer.ibm.com/storage/2018/01/01/ibm-spectrum-scale-object-protocol-ubuntu/ IBM Spectrum Scale 5.0 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2017/12/20/ibm-spectrum-scale-5-0-whats-new-object/ A Complete Guide to ? Protocol Problem Determination Guide for IBM Spectrum Scale? ? Part 1 https://developer.ibm.com/storage/2017/12/19/complete-guide-protocol-problem-determination-guide-ibm-spectrum-scale-1/ IBM Spectrum Scale installation toolkit ? enhancements over releases https://developer.ibm.com/storage/2017/12/15/ibm-spectrum-scale-installation-toolkit-enhancements-releases/ Network requirements in an Elastic Storage Server Setup https://developer.ibm.com/storage/2017/12/13/network-requirements-in-an-elastic-storage-server-setup/ Co-resident migration with Transparent cloud tierin https://developer.ibm.com/storage/2017/12/05/co-resident-migration-transparent-cloud-tierin/ IBM Spectrum Scale on Hortonworks HDP Hadoop clusters : A Complete Big Data Solution https://developer.ibm.com/storage/2017/12/05/ibm-spectrum-scale-hortonworks-hdp-hadoop-clusters-complete-big-data-solution/ Big data analytics with Spectrum Scale using remote cluster mount & multi-filesystem support https://developer.ibm.com/storage/2017/11/28/big-data-analytics-spectrum-scale-using-remote-cluster-mount-multi-filesystem-support/ IBM Spectrum Scale HDFS Transparency Short Circuit Write Support https://developer.ibm.com/storage/2017/11/28/ibm-spectrum-scale-hdfs-transparency-short-circuit-write-support/ IBM Spectrum Scale HDFS Transparency Federation Support https://developer.ibm.com/storage/2017/11/27/ibm-spectrum-scale-hdfs-transparency-federation-support/ How to configure and performance tuning different system workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-different-system-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Spark workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-spark-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning database workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-database-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Hadoop workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/24/configure-performance-tuning-hadoop-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ IBM Spectrum Scale Sharing Nothing Cluster Performance Tuning https://developer.ibm.com/storage/2017/11/24/ibm-spectrum-scale-sharing-nothing-cluster-performance-tuning/ How to Configure IBM Spectrum Scale? with NIS based Authentication. https://developer.ibm.com/storage/2017/11/21/configure-ibm-spectrum-scale-nis-based-authentication/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Cc: Doris Conti/Poughkeepsie/IBM at IBMUS Date: 11/16/2017 08:15 PM Subject: Latest Technical Blogs on Spectrum Scale Dear User Group members, Here are the Development Blogs in last 3 months on Spectrum Scale Technical Topics. Spectrum Scale Monitoring ? Know More ? https://developer.ibm.com/storage/2017/11/16/spectrum-scale-monitoring-know/ IBM Spectrum Scale 5.0 Release ? What?s coming ! https://developer.ibm.com/storage/2017/11/14/ibm-spectrum-scale-5-0-release-whats-coming/ Four Essentials things to know for managing data ACLs on IBM Spectrum Scale? from Windows https://developer.ibm.com/storage/2017/11/13/four-essentials-things-know-managing-data-acls-ibm-spectrum-scale-windows/ GSSUTILS: A new way of running SSR, Deploying or Upgrading ESS Server https://developer.ibm.com/storage/2017/11/13/gssutils/ IBM Spectrum Scale Object Authentication https://developer.ibm.com/storage/2017/11/02/spectrum-scale-object-authentication/ Video Surveillance ? Choosing the right storage https://developer.ibm.com/storage/2017/11/02/video-surveillance-choosing-right-storage/ IBM Spectrum scale object deep dive training with problem determination https://www.slideshare.net/SmitaRaut/ibm-spectrum-scale-object-deep-dive-training Spectrum Scale as preferred software defined storage for Ubuntu OpenStack https://developer.ibm.com/storage/2017/09/29/spectrum-scale-preferred-software-defined-storage-ubuntu-openstack/ IBM Elastic Storage Server 2U24 Storage ? an All-Flash offering, a performance workhorse https://developer.ibm.com/storage/2017/10/06/ess-5-2-flash-storage/ A Complete Guide to Configure LDAP-based authentication with IBM Spectrum Scale? for File Access https://developer.ibm.com/storage/2017/09/21/complete-guide-configure-ldap-based-authentication-ibm-spectrum-scale-file-access/ Deploying IBM Spectrum Scale on AWS Quick Start https://developer.ibm.com/storage/2017/09/18/deploy-ibm-spectrum-scale-on-aws-quick-start/ Monitoring Spectrum Scale Object metrics https://developer.ibm.com/storage/2017/09/14/monitoring-spectrum-scale-object-metrics/ Tier your data with ease to Spectrum Scale Private Cloud(s) using Moonwalk Universal https://developer.ibm.com/storage/2017/09/14/tier-data-ease-spectrum-scale-private-clouds-using-moonwalk-universal/ Why do I see owner as ?Nobody? for my export mounted using NFSV4 Protocol on IBM Spectrum Scale?? https://developer.ibm.com/storage/2017/09/08/see-owner-nobody-export-mounted-using-nfsv4-protocol-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory and LDAP https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-ldap/ IBM Spectrum Scale? Authentication using Active Directory and RFC2307 https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-rfc2307/ High Availability Implementation with IBM Spectrum Virtualize and IBM Spectrum Scale https://developer.ibm.com/storage/2017/08/30/high-availability-implementation-ibm-spectrum-virtualize-ibm-spectrum-scale/ 10 Frequently asked Questions on configuring Authentication using AD + AUTO ID mapping on IBM Spectrum Scale?. https://developer.ibm.com/storage/2017/08/04/10-frequently-asked-questions-configuring-authentication-using-ad-auto-id-mapping-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory https://developer.ibm.com/storage/2017/07/30/ibm-spectrum-scale-auth-using-active-directory/ Five cool things that you didn?t know Transparent Cloud Tiering on Spectrum Scale can do https://developer.ibm.com/storage/2017/07/29/five-cool-things-didnt-know-transparent-cloud-tiering-spectrum-scale-can/ IBM Spectrum Scale GUI videos https://developer.ibm.com/storage/2017/07/25/ibm-spectrum-scale-gui-videos/ IBM Spectrum Scale? Authentication ? Planning for NFS Access https://developer.ibm.com/storage/2017/07/24/ibm-spectrum-scale-planning-nfs-access/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media -------------- next part -------------- An HTML attachment was scrubbed... URL: From valleru at cbio.mskcc.org Fri Mar 20 21:18:42 2020 From: valleru at cbio.mskcc.org (Valleru, Lohit/Information Systems) Date: Fri, 20 Mar 2020 17:18:42 -0400 Subject: [gpfsug-discuss] Network switches/architecture for GPFS Message-ID: <35ADE287-4355-430B-83A0-A00C76525822@cbio.mskcc.org> Hello All, I would like to discuss or understand on which ethernet networking switches/architecture seems to work best with GPFS. We had thought about infiniband, but are not yet ready to move to infiniband because of the complexity/upgrade and debugging issues that come with it. Current hardware: We are currently using Arista 7328x 100G core switch for networking among the GPFS clusters and the compute nodes. It is heterogeneous network, with some of the servers on 10G/25G/100G with LACP and without LACP. For example: GPFS storage clusters either have 25G LACP, or 10G LACP, or a single 100G network port. Compute nodes range from 10G to 100G. Login nodes/transfer servers etc have 25G bonded. Most of the servers have Mellanox ConnectX-4 or ConnectX-5 adapters. But we also have few older Intel,Broadcom and Chelsio network cards in the clusters. Most of the transceivers that we use are Mellanox,Finisar,Intel. Issue: We had upgraded to the above switch recently, and we had seen that it is not able to handle the network traffic because of higher NSD servers bandwidth vs lower compute node bandwidth. One issue that we did see was a lot of network discards on the switch side and network congestion with slow IO performance on respective compute nodes. Once we enabled ECN - we did see that it had reduced the network congestion. We do see expels once in a while, but that is mostly related to the network errors or the host not responding. We observed that bonding/LACP does make expels much more trickier, so we have decided to go with no LACP until GPFS code gets better at handling LACP - which I think they are working on. We have heard that our current switch is a shallow buffer switch, and we would need a higher/deep buffer Arista switch to perform better with no congestion/lesser latency and more throughput. On the other side, Mellanox promises to use better ASIC design and buffer architecture with spine leaf design, instead of one deep buffer core switch to get better performance than Arista. Most of the applications that run on the clusters are either genomic applications on CPUs and deep learning applications on GPUs. All of our GPFS storage cluster versions are above 5.0.2 with the compute filesystems at 16M block size on near line rotating disks, and Flash storage at 512K block size. May I know if could feedback from anyone who is using Arista or Mellanox switches on the clusters to understand the pros and cons, stability and the performance numbers of the same? Thank you, Lohit -------------- next part -------------- An HTML attachment was scrubbed... URL: From cblack at nygenome.org Sun Mar 22 00:58:42 2020 From: cblack at nygenome.org (Christopher Black) Date: Sun, 22 Mar 2020 00:58:42 +0000 Subject: [gpfsug-discuss] Network switches/architecture for GPFS In-Reply-To: <35ADE287-4355-430B-83A0-A00C76525822@cbio.mskcc.org> References: <35ADE287-4355-430B-83A0-A00C76525822@cbio.mskcc.org> Message-ID: <74B39595-5718-4B88-B7B8-513942E1983D@nygenome.org> We?ve had good luck moving from older Mellanox 1710 ethernet switches to newer Arista ethernet switches. Our core is a pair of Arista 7508s primarily with 100G cards. Leaf switches are Arista 7280QR for racks with 40Gb-connected servers and 7280SR for racks w/ 10Gb-connected servers. Uplinks from leaf switches to core are multiple 100G connections. Our nsd servers are connected with dual-40Gb connections, each connection a separate Mellanox ConnectX-3 card to spread load and failure across separate pcie slots. Our compute nodes are primarily connected with dual-10Gb connections on Intel x520 or x710 nics (dual-port on a single nic). We also have some Cisco UCS nodes going through Cisco FI?s, these do not perform nearly as well and we?ve had some trouble with them and high bandwidth network storage, especially with defaults. We have some data transfer nodes connected at 2x40Gb, but other than that our only 40Gb-connected nodes are nsd servers. Any server, nsd or compute, uses lacp to bond and has mtu set to 9000. We also set: BONDING_OPTS="mode=4 miimon=100 xmit_hash_policy=layer3+4" For ecn, we have sysctl net.ipv4.tcp_ecn = 2. We also run primarily genomics applications. We have no experience with more recent Mellanox switches, but the ethernet software implementation on their older switches gave us plenty of problems. I?m not the network expert at our site, but they seem to like the Arista software much more than Mellanox. We run some non-default tcp/ip and ethernet settings, primarily from fasterdata.es.net recommendations. IBM?s older wiki notes about linux sysctls sometimes does not match es.net recommendation, and in those cases we generally go w/ es.net, especially as some of the IBM docs were written for older Linux kernels. However, there are some sysctl recommendations from IBM docs that are unique to gpfs (net.core.somaxconn). Regarding non-net tuning to improve gpfs stability, we?ve found the following are also important: vm.min_free_kbytes vm.dirty_bytes vm.dirty_background_bytes It took us a long time to figure out that on systems with lots of memory, many dirty pages could be buffered before being flushed out to network, resulting in a storm of heavy traffic that could cause problems for gpfs disk lease renewals and other control traffic to get through quick enough to avoid expels. For client NIC tuning, we set txqueuelen 10000 but I?ve read that this may not be necessary on newer kernels. On older nics, or even current intel nics with older firmware, we found turning some offload optimizations OFF made things better (gro, lro, gso, lso). I hope this helps you or others running gpfs on ethernet! -Chris From: on behalf of "Valleru, Lohit/Information Systems" Reply-To: gpfsug main discussion list Date: Friday, March 20, 2020 at 5:18 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Network switches/architecture for GPFS Hello All, I would like to discuss or understand on which ethernet networking switches/architecture seems to work best with GPFS. We had thought about infiniband, but are not yet ready to move to infiniband because of the complexity/upgrade and debugging issues that come with it. Current hardware: We are currently using Arista 7328x 100G core switch for networking among the GPFS clusters and the compute nodes. It is heterogeneous network, with some of the servers on 10G/25G/100G with LACP and without LACP. For example: GPFS storage clusters either have 25G LACP, or 10G LACP, or a single 100G network port. Compute nodes range from 10G to 100G. Login nodes/transfer servers etc have 25G bonded. Most of the servers have Mellanox ConnectX-4 or ConnectX-5 adapters. But we also have few older Intel,Broadcom and Chelsio network cards in the clusters. Most of the transceivers that we use are Mellanox,Finisar,Intel. Issue: We had upgraded to the above switch recently, and we had seen that it is not able to handle the network traffic because of higher NSD servers bandwidth vs lower compute node bandwidth. One issue that we did see was a lot of network discards on the switch side and network congestion with slow IO performance on respective compute nodes. Once we enabled ECN - we did see that it had reduced the network congestion. We do see expels once in a while, but that is mostly related to the network errors or the host not responding. We observed that bonding/LACP does make expels much more trickier, so we have decided to go with no LACP until GPFS code gets better at handling LACP - which I think they are working on. We have heard that our current switch is a shallow buffer switch, and we would need a higher/deep buffer Arista switch to perform better with no congestion/lesser latency and more throughput. On the other side, Mellanox promises to use better ASIC design and buffer architecture with spine leaf design, instead of one deep buffer core switch to get better performance than Arista. Most of the applications that run on the clusters are either genomic applications on CPUs and deep learning applications on GPUs. All of our GPFS storage cluster versions are above 5.0.2 with the compute filesystems at 16M block size on near line rotating disks, and Flash storage at 512K block size. May I know if could feedback from anyone who is using Arista or Mellanox switches on the clusters to understand the pros and cons, stability and the performance numbers of the same? Thank you, Lohit ________________________________ This message is for the recipient?s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kraemerf at de.ibm.com Mon Mar 23 07:46:54 2020 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Mon, 23 Mar 2020 08:46:54 +0100 Subject: [gpfsug-discuss] Network switches/architecture for Spectrum Scale (GPFS) In-Reply-To: References: Message-ID: Hi, > May I know if could feedback from anyone who is using Arista or Mellanox switches on the > clusters to understand the pros and cons, stability and the performance numbers of the same? Most current network switches are using Merchant Silicon ASICs instead of custom ASICs as in the past. (This is even true for some Cisco devices.) "Merchant silicon is a term used to described chips, usually ASICs, that are designed and made by an entity other than the company selling the switches in which they are used. I might be tempted to say such switches use off-the-shelf ASICs, though that might imply that I could buy these chips from a retail store." https://www.datacenterknowledge.com/networks/why-merchant-silicon-taking-over-data-center-network-market All of the Merchant Silicon ASICs lines have different family members from entry to high-end features. Switch vendors will pick and choose these ASICs by price and target market for the specific switch models. You can't tell from the "outside" which switch does offer which kind of performance. You need to do more research on the specific switch and use case. Most of the time Mellanox switches does offer very attractive features and performance levels for parallel I/O workloads but there are more options in the market. The NOS (Network OS) which is running on these new networking switches is also not a fixed option. You have some choice here. Some switches can be ordered with different NOS versions. Which NOS is the best fit for Spectrum Scale or other pFS's is still a topic to be researched. Last but not least the question of which NIC card is the best fit in this job is important. SmartNIC: A network interface card (network adapter) that offloads processing tasks that the system CPU would normally handle. Using its own on-board processor, the SmartNIC may be able to perform any combination of encryption/decryption, firewall, TCP/IP and HTTP processing. SmartNICs can be ASIC, FPGA, and System-on-a-Chip (SOC) based. Naturally vendors who make just one kind of NIC seem to insist that only the type of NIC they make should qualify as a SmartNIC. Mellanox is a good choice here but there is wide range of choices and features already in the market. -frank- Appendix: 1 - A Network-centric View of Scalable Storage by Andy Bechtolsheim, Chief Development Officer and Co-Founder, Arista Networks https://youtu.be/__LomgSguSc 2 - Broadcom Ships Tomahawk 4, Industry?s Highest Bandwidth Ethernet Switch Chip at 25.6 Terabits per Second ? Broadcom https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm56990-series https://packetpushers.net/podcast/network-break-264-broadcoms-new-tomahawk-4-hits-25-6tbps-juniper-announces-sd-lan-for-ex-switches/ https://www.telecompaper.com/news/broadcom-ships-tomahawk-4-ethernet-switch-chip--1319338 Broadcom delivered the StrataXGS Tomahawk 4 switch series, demonstrating an unprecedented 25.6 Tbps of Ethernet switching performance in a single device. 3 - Mellanox Introduces Revolutionary ConnectX-6 Dx and BlueField-2 Secure Cloud SmartNICs and I/O Processing Unit Solutions ConnectX-6 Dx and BlueField-2 Provide 200 Gb/s Ethernet and InfiniBand Connectivity, Enabling Next Generation of Clouds, Secure Data Centers and Storage Platforms https://www.mellanox.com/page/press_release_item?id=2195 Frank Kraemer IBM Consulting IT Specialist / Client Technical Architect Am Weiher 24, 65451 Kelsterbach, Germany mailto:kraemerf at de.ibm.com Mobile +49171-3043699 IBM Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Mon Mar 23 18:09:16 2020 From: mweil at wustl.edu (Matt Weil) Date: Mon, 23 Mar 2020 13:09:16 -0500 Subject: [gpfsug-discuss] AFM gateway node scaling Message-ID: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Hello all, Is there any guide and or recommendation as to how to scale this. filesets per gateway node?? Is it necessary to separate NSD server and gateway roles.? Are dedicated gateway nodes licensed as clients? Thanks for any guidance. Matt From peter.chase at metoffice.gov.uk Tue Mar 24 11:08:15 2020 From: peter.chase at metoffice.gov.uk (Chase, Peter) Date: Tue, 24 Mar 2020 11:08:15 +0000 Subject: [gpfsug-discuss] tctwatcher command Message-ID: Hello, The scale 5.0.4 documentation makes reference to a tctwatcher command, but I can't find the executable in any of the 5.0.4-3 RPMs I've deployed (including gpfs.tct.server, gpfs.tct.client, gpfs.kafka and gpfs.librdkafka). I'd wondered if it needed compiling from somewhere in /usr/lpp/mmfs/samples/ or maybe /opt/ibm/MCStore/samples/ but I can't find anything referencing it there either! Does anyone know where it is? Documentation: https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/b1ladm_integratingwfolderintocloud.htm It's also in the 5.0.3 doc. Regards, Pete Chase Mainframe, Virtualization and Storage team Met Office -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Wed Mar 25 10:01:04 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 10:01:04 +0000 Subject: [gpfsug-discuss] mmbackup monitoring Message-ID: What is the best way of monitoring whether or not mmbackup has managed to complete a backup successfully? Traditionally one use a TSM monitoring solution of your choice to make sure nodes where backing up (I am assuming mmbackup is being used in conjunction with TSM here). However mmbackup does not update the backup_end column in the filespaceview table (at least in 4.2) which makes things rather more complicated. The best I can come up with is querying the events table to see if the client schedule completed, but that gives a false sense of security as the schedule completing does not mean the backup completed as far as I know. What solutions are you all using, or does mmbackup in 5.x update the filespaceview table? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From jonathan.buzzard at strath.ac.uk Wed Mar 25 10:09:12 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 10:09:12 +0000 Subject: [gpfsug-discuss] GPFS 5 and supported rhel OS In-Reply-To: References: Message-ID: <91d02fd3-2af7-5880-e1f2-aaf9b1f8040a@strath.ac.uk> On 19/02/2020 23:34, Renata Maria Dart wrote: > Hi, I understand gpfs 4.2.3 is end of support this coming September. A planning question at this stage. Do IBM intend to hold to this date or is/could there be a relaxation due to COVID-19? Basically I was planning to do the upgrade this summer, but what with working from home I am less keen to do a a 4.2.3 to 5.x upgrade while not on hand to the actual hardware. Obviously if we have to we have to, just want to know where we stand. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From carlz at us.ibm.com Wed Mar 25 12:12:38 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Wed, 25 Mar 2020 12:12:38 +0000 Subject: [gpfsug-discuss] GPFS 5 and supported rhel OS Message-ID: <70ECB57D-C012-43FB-89F1-AF5810984DAF@us.ibm.com> So far we have not revisited the EOS date for 4.2.3, but I would not rule it out entirely if the lockdown continues well into the summer. If we did, the next likely EOS date would be April 30th. Even if we do postpone the date for 4.2.3, keep two other dates in mind for planning: - RHEL 6 support is coming to an end in November. We won't support Scale with RHEL 6 once Red Hat stops supporting RHEL 6 - RHEL 7 will be supported with 5.0.5, but not "5.next", the release scheduled for the second half of 2020. So you'll need to plan to adopt RHEL 8 before upgrading to Scale "5.next" As much as possible we are going to try to stick to our release cadence of twice a year even through these difficulties, including designating 5.0.5 for Extended Updates. "Keep Calm and Scale Out". Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com Message: 2 Date: Wed, 25 Mar 2020 10:09:12 +0000 From: Jonathan Buzzard To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] GPFS 5 and supported rhel OS Message-ID: <91d02fd3-2af7-5880-e1f2-aaf9b1f8040a at strath.ac.uk> Content-Type: text/plain; charset=utf-8; format=flowed On 19/02/2020 23:34, Renata Maria Dart wrote: > Hi, I understand gpfs 4.2.3 is end of support this coming September. A planning question at this stage. Do IBM intend to hold to this date or is/could there be a relaxation due to COVID-19? Basically I was planning to do the upgrade this summer, but what with working from home I am less keen to do a a 4.2.3 to 5.x upgrade while not on hand to the actual hardware. Obviously if we have to we have to, just want to know where we stand. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ------------------------------ From skylar2 at u.washington.edu Wed Mar 25 14:15:59 2020 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Wed, 25 Mar 2020 07:15:59 -0700 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: References: Message-ID: <20200325141558.2iwnalq3hwo3r5ha@illiuin> We execute mmbackup via a regular TSM client schedule with an incremental action, with a virtualmountpoint set to an empty, local "canary" directory. mmbackup runs as a preschedule command, and the client -domain parameter is set only to backup the canary directory. dsmc will backup the canary directory as a filespace only if mmbackup succeeds (exits with 0). We can then monitor the canary and infer the status of the associated GPFS filespace or fileset. On Wed, Mar 25, 2020 at 10:01:04AM +0000, Jonathan Buzzard wrote: > > What is the best way of monitoring whether or not mmbackup has managed to > complete a backup successfully? > > Traditionally one use a TSM monitoring solution of your choice to make sure > nodes where backing up (I am assuming mmbackup is being used in conjunction > with TSM here). > > However mmbackup does not update the backup_end column in the filespaceview > table (at least in 4.2) which makes things rather more complicated. > > The best I can come up with is querying the events table to see if the > client schedule completed, but that gives a false sense of security as the > schedule completing does not mean the backup completed as far as I know. > > What solutions are you all using, or does mmbackup in 5.x update the > filespaceview table? -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From pinto at scinet.utoronto.ca Wed Mar 25 14:38:55 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Wed, 25 Mar 2020 10:38:55 -0400 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <20200325141558.2iwnalq3hwo3r5ha@illiuin> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> Message-ID: Additionally, mmbackup creates by default a .mmbackupCfg directory on the root of the fileset where it dumps several files and directories with the progress of the backup. For instance: expiredFiles/, prepFiles/, updatedFiles/, dsminstr.log, ... You may then create a script to search these directories for logs/lists of what has happened, and generate a more detailed report of what happened during the backup. In our case I generate a daily report of how many files and how much data have been sent to the TSM server and deleted for each user, including their paths. You can do more tricks if you want. Jaime On 3/25/2020 10:15:59, Skylar Thompson wrote: > We execute mmbackup via a regular TSM client schedule with an incremental > action, with a virtualmountpoint set to an empty, local "canary" directory. > mmbackup runs as a preschedule command, and the client -domain parameter is > set only to backup the canary directory. dsmc will backup the canary > directory as a filespace only if mmbackup succeeds (exits with 0). We can > then monitor the canary and infer the status of the associated GPFS > filespace or fileset. > > On Wed, Mar 25, 2020 at 10:01:04AM +0000, Jonathan Buzzard wrote: >> >> What is the best way of monitoring whether or not mmbackup has managed to >> complete a backup successfully? >> >> Traditionally one use a TSM monitoring solution of your choice to make sure >> nodes where backing up (I am assuming mmbackup is being used in conjunction >> with TSM here). >> >> However mmbackup does not update the backup_end column in the filespaceview >> table (at least in 4.2) which makes things rather more complicated. >> >> The best I can come up with is querying the events table to see if the >> client schedule completed, but that gives a false sense of security as the >> schedule completing does not mean the backup completed as far as I know. >> >> What solutions are you all using, or does mmbackup in 5.x update the >> filespaceview table? > . . . ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From skylar2 at u.washington.edu Wed Mar 25 15:25:15 2020 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Wed, 25 Mar 2020 08:25:15 -0700 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> Message-ID: <20200325152515.rd2uyh5fckkwdx4p@illiuin> IIRC, I think you need to set 2 in the bit field of the DEBUGmmbackup environment variable. I had a long-term task to see what I could get out of that, but this just reminded me of it and current events might actually let me have time to look into it now... On Wed, Mar 25, 2020 at 10:38:55AM -0400, Jaime Pinto wrote: > Additionally, mmbackup creates by default a .mmbackupCfg directory on the root of the fileset where it dumps several files and directories with the progress of the backup. For instance: expiredFiles/, prepFiles/, updatedFiles/, dsminstr.log, ... > > You may then create a script to search these directories for logs/lists of what has happened, and generate a more detailed report of what happened during the backup. In our case I generate a daily report of how many files and how much data have been sent to the TSM server and deleted for each user, including their paths. You can do more tricks if you want. > > Jaime > > > On 3/25/2020 10:15:59, Skylar Thompson wrote: > > We execute mmbackup via a regular TSM client schedule with an incremental > > action, with a virtualmountpoint set to an empty, local "canary" directory. > > mmbackup runs as a preschedule command, and the client -domain parameter is > > set only to backup the canary directory. dsmc will backup the canary > > directory as a filespace only if mmbackup succeeds (exits with 0). We can > > then monitor the canary and infer the status of the associated GPFS > > filespace or fileset. > > > > On Wed, Mar 25, 2020 at 10:01:04AM +0000, Jonathan Buzzard wrote: > > > > > > What is the best way of monitoring whether or not mmbackup has managed to > > > complete a backup successfully? > > > > > > Traditionally one use a TSM monitoring solution of your choice to make sure > > > nodes where backing up (I am assuming mmbackup is being used in conjunction > > > with TSM here). > > > > > > However mmbackup does not update the backup_end column in the filespaceview > > > table (at least in 4.2) which makes things rather more complicated. > > > > > > The best I can come up with is querying the events table to see if the > > > client schedule completed, but that gives a false sense of security as the > > > schedule completing does not mean the backup completed as far as I know. > > > > > > What solutions are you all using, or does mmbackup in 5.x update the > > > filespaceview table? > > > > . > . > . ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto - Storage Analyst > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From heinrich.billich at id.ethz.ch Wed Mar 25 15:35:55 2020 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Wed, 25 Mar 2020 15:35:55 +0000 Subject: [gpfsug-discuss] GUI timeout when running HW_INVENTORY on little endian ESS server Message-ID: Hello, I did ask about this timeouts when the gui runs HW_INVENTORY before. Now I would like to know what the exact timeout value in the gui code is and if we can change it. I want to argue: If a xCat command takes X seconds but the GUI code timeouts after Y we know the command will fail if X > Y, hence we need to increase Y unless we can reduce X ... It's this function which raises the timeout: at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory If we can't fix the long execution time for the time being, can we raise the timeout value? I know this most likely is a Firmware issue with little endian power systems, but we won't update some more time. Thank you, Heiner debug: Running 'xcat.sh rinv '10.250.***' '*****' 'all' ' on node localhost err: com.ibm.fscc.common.exceptions.FsccException: Error executing rinv command. Exit code = 1; Command output = ; Command error = *****: [****]: Error: timeout at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory(InventoryAndStateHelper.java:92) at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.buildHardwareInventory(InventoryAndStateHelper.java:175) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.inner_run(InventoryRefreshTask.java:94) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.run(InventoryRefreshTask.java:72) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:227) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:199) at com.ibm.fscc.common.newscheduler.RefreshTaskIds.execute(RefreshTaskIds.java:482) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:80) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) ... debug: Running 'mmsysmonc event 'gui' 'xcat_nodelist_ok' ****-i' ***-i' ' on node localhost err: ***-i: Error executing rinv command. Exit code = 1; Command output = ; Command error = nas12io04b: [***]: Error: timeout ,*** -i: Error executing rinv command. Exit code = 1; Command output = ; Command error =***: [***]: Error: timeout err: com.ibm.fscc.cli.CommandException: EFSSG1150C Running specified task was unsuccessful. at com.ibm.fscc.cli.CommandException.createCommandException(CommandException.java:117) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:84) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) EFSSG1150C Running specified task was unsuccessful. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== From heinrich.billich at id.ethz.ch Wed Mar 25 15:53:04 2020 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Wed, 25 Mar 2020 15:53:04 +0000 Subject: [gpfsug-discuss] GUI timeout when running HW_INVENTORY on little endian ESS server In-Reply-To: References: Message-ID: <3C52599F-DC19-47F2-9CF2-90B905D43A1B@id.ethz.ch> Hello, Sorry, I was wrong. Looks like the timeout already happens in xCAT/rinv and the gui just reports it. What to some respect is good - now this is a purely xCAT/hardware issue. The GUI isn't involved any more. Kind regards Heiner /var/log/xcat/command.log: ==================================================== [Date] 2020-03-25 15:03:46 [ClientType] cli [Request] rinv ***** all [Response] ***: Error: timeout [NumberNodes] 1 [ElapsedTime] 97.085 s ==================================================== GUI: HW_INVENTORY * 2020-03-25 15:03:26 681436ms failed CmdRunTask.doExecute nas12io04b-i: Error executing rinv command. Exit code = 1; Command output = ; Command error =***: [**]: Error: timeout ?On 25.03.20, 16:35, "Billich Heinrich Rainer (ID SD)" wrote: Hello, I did ask about this timeouts when the gui runs HW_INVENTORY before. Now I would like to know what the exact timeout value in the gui code is and if we can change it. I want to argue: If a xCat command takes X seconds but the GUI code timeouts after Y we know the command will fail if X > Y, hence we need to increase Y unless we can reduce X ... It's this function which raises the timeout: at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory If we can't fix the long execution time for the time being, can we raise the timeout value? I know this most likely is a Firmware issue with little endian power systems, but we won't update some more time. Thank you, Heiner debug: Running 'xcat.sh rinv '10.250.***' '*****' 'all' ' on node localhost err: com.ibm.fscc.common.exceptions.FsccException: Error executing rinv command. Exit code = 1; Command output = ; Command error = *****: [****]: Error: timeout at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory(InventoryAndStateHelper.java:92) at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.buildHardwareInventory(InventoryAndStateHelper.java:175) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.inner_run(InventoryRefreshTask.java:94) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.run(InventoryRefreshTask.java:72) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:227) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:199) at com.ibm.fscc.common.newscheduler.RefreshTaskIds.execute(RefreshTaskIds.java:482) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:80) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) ... debug: Running 'mmsysmonc event 'gui' 'xcat_nodelist_ok' ****-i' ***-i' ' on node localhost err: ***-i: Error executing rinv command. Exit code = 1; Command output = ; Command error = nas12io04b: [***]: Error: timeout ,*** -i: Error executing rinv command. Exit code = 1; Command output = ; Command error =***: [***]: Error: timeout err: com.ibm.fscc.cli.CommandException: EFSSG1150C Running specified task was unsuccessful. at com.ibm.fscc.cli.CommandException.createCommandException(CommandException.java:117) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:84) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) EFSSG1150C Running specified task was unsuccessful. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== From jonathan.buzzard at strath.ac.uk Wed Mar 25 16:27:27 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 16:27:27 +0000 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <20200325141558.2iwnalq3hwo3r5ha@illiuin> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> Message-ID: <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> On 25/03/2020 14:15, Skylar Thompson wrote: > We execute mmbackup via a regular TSM client schedule with an incremental > action, with a virtualmountpoint set to an empty, local "canary" directory. > mmbackup runs as a preschedule command, and the client -domain parameter is > set only to backup the canary directory. dsmc will backup the canary > directory as a filespace only if mmbackup succeeds (exits with 0). We can > then monitor the canary and infer the status of the associated GPFS > filespace or fileset. > I prefer this approach I think than grovelling around in log files that could easily break on an update. Though there is a better approach which in my view IBM should be using already in mmbackup. It came to me this afternoon that one could use the TSM API for this. After a bit of Googling I find there is an API call dsmUpdateFS, which allows you to update the filespace information on the TSM server. Fields that you can update include DSM_FSUPD_OCCUPANCY DSM_FSUPD_CAPACITY DSM_FSUPD_BACKSTARTDATE DSM_FSUPD_BACKCOMPLETEDATE Information on the API call here https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/api/r_cmd_dsmupdatefs.html How do we submit this as a feature request again? That said in my view it's a bug in mmbackup. The latest in a very long line stretching back well over a decade that make mmbackup less than production ready rather than a feature request :-) I feel a breakout of a text editor and some C code coming on in the meantime. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From skylar2 at u.washington.edu Wed Mar 25 16:32:24 2020 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Wed, 25 Mar 2020 09:32:24 -0700 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> Message-ID: <20200325163224.jbibka2nukfhmc7u@illiuin> On Wed, Mar 25, 2020 at 04:27:27PM +0000, Jonathan Buzzard wrote: > On 25/03/2020 14:15, Skylar Thompson wrote: > > We execute mmbackup via a regular TSM client schedule with an incremental > > action, with a virtualmountpoint set to an empty, local "canary" directory. > > mmbackup runs as a preschedule command, and the client -domain parameter is > > set only to backup the canary directory. dsmc will backup the canary > > directory as a filespace only if mmbackup succeeds (exits with 0). We can > > then monitor the canary and infer the status of the associated GPFS > > filespace or fileset. > > > > I prefer this approach I think than grovelling around in log files that > could easily break on an update. Though there is a better approach which in > my view IBM should be using already in mmbackup. > > It came to me this afternoon that one could use the TSM API for this. After > a bit of Googling I find there is an API call dsmUpdateFS, which allows you > to update the filespace information on the TSM server. > > Fields that you can update include > > DSM_FSUPD_OCCUPANCY > DSM_FSUPD_CAPACITY > DSM_FSUPD_BACKSTARTDATE > DSM_FSUPD_BACKCOMPLETEDATE > > Information on the API call here > > https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/api/r_cmd_dsmupdatefs.html > > How do we submit this as a feature request again? That said in my view it's > a bug in mmbackup. The latest in a very long line stretching back well over > a decade that make mmbackup less than production ready rather than a feature > request :-) > > I feel a breakout of a text editor and some C code coming on in the > meantime. I actually tried using the API years ago to try to do some custom queries, and ran into the problem that custom API clients can only see data from custom API clients; they can't see data from the standard BA client. I contacted IBM about this, and they said it was a safety feature to prevent a rogue/poorly-written client from trashing regular backup/archive data, which makes some sense. Unfortunately, it does mean that IBM would have to be the source of the fix. -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From jonathan.buzzard at strath.ac.uk Wed Mar 25 16:45:54 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 16:45:54 +0000 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <20200325163224.jbibka2nukfhmc7u@illiuin> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> <20200325163224.jbibka2nukfhmc7u@illiuin> Message-ID: On 25/03/2020 16:32, Skylar Thompson wrote: > On Wed, Mar 25, 2020 at 04:27:27PM +0000, Jonathan Buzzard wrote: >> On 25/03/2020 14:15, Skylar Thompson wrote: >>> We execute mmbackup via a regular TSM client schedule with an incremental >>> action, with a virtualmountpoint set to an empty, local "canary" directory. >>> mmbackup runs as a preschedule command, and the client -domain parameter is >>> set only to backup the canary directory. dsmc will backup the canary >>> directory as a filespace only if mmbackup succeeds (exits with 0). We can >>> then monitor the canary and infer the status of the associated GPFS >>> filespace or fileset. >>> >> >> I prefer this approach I think than grovelling around in log files that >> could easily break on an update. Though there is a better approach which in >> my view IBM should be using already in mmbackup. >> >> It came to me this afternoon that one could use the TSM API for this. After >> a bit of Googling I find there is an API call dsmUpdateFS, which allows you >> to update the filespace information on the TSM server. >> >> Fields that you can update include >> >> DSM_FSUPD_OCCUPANCY >> DSM_FSUPD_CAPACITY >> DSM_FSUPD_BACKSTARTDATE >> DSM_FSUPD_BACKCOMPLETEDATE >> >> Information on the API call here >> >> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fsupport%2Fknowledgecenter%2FSSEQVQ_8.1.9%2Fapi%2Fr_cmd_dsmupdatefs.html&data=02%7C01%7Cjonathan.buzzard%40strath.ac.uk%7C8c7605146223442e8a2708d7d0dab99d%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C637207510145541248&sdata=QV9U0WxL5BTtR5%2Fasv1X202d9PqnLXZNg5bzX8KpUOo%3D&reserved=0 >> >> How do we submit this as a feature request again? That said in my view it's >> a bug in mmbackup. The latest in a very long line stretching back well over >> a decade that make mmbackup less than production ready rather than a feature >> request :-) >> >> I feel a breakout of a text editor and some C code coming on in the >> meantime. > > I actually tried using the API years ago to try to do some custom queries, > and ran into the problem that custom API clients can only see data from > custom API clients; they can't see data from the standard BA client. I > contacted IBM about this, and they said it was a safety feature to prevent > a rogue/poorly-written client from trashing regular backup/archive data, > which makes some sense. Unfortunately, it does mean that IBM would have to > be the source of the fix. > Grrr, I had forgotten that. Well then IBM need to fix this. Bug mmbackup does not update the occupancy, capacity, backup start date and backup end date when doing a backup. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From vpuvvada at in.ibm.com Wed Mar 25 16:50:41 2020 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Wed, 25 Mar 2020 22:20:41 +0530 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: Matt, It is recommended to have dedicated AFM gateway nodes. Memory and CPU requirements for AFM gateway node depends on the number of filesets handled by the node and the inode usage of those filesets. Since AFM keeps track of changes in the memory, any network disturbance can cause the memory utilization to go high and which eventually leads to in-memory queue to be dropped. After the queue is dropped, AFM runs recovery to recover the lost operations which is expensive as it involves creating the snapshot, running policy scan, doing readdir from home/secondary and build the list of lost operations. When the gateway node goes down, all the filesets handled by that node distributed to the remaining active gateway nodes. After the gateway node comes back, filesets are transferred back to the original gateway node. When designing the gateway node, make sure that it have enough memory , CPU resources for handling the incoming and outgoing data based on the bandwidth. Limit the filesets per gateway(ex. less than 20 filesets per gateway) so that number of AFM recoveries triggered will be minimal when the queues are lost. Also limit the total number of inodes handled by the gateway node across all the filesets (ex. less than 400 million inodes per gateway). AFM gateway nodes are licensed as server nodes. ~Venkat (vpuvvada at in.ibm.com) From: Matt Weil To: gpfsug-discuss at spectrumscale.org Date: 03/23/2020 11:39 PM Subject: [EXTERNAL] [gpfsug-discuss] AFM gateway node scaling Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all, Is there any guide and or recommendation as to how to scale this. filesets per gateway node? Is it necessary to separate NSD server and gateway roles. Are dedicated gateway nodes licensed as clients? Thanks for any guidance. Matt _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=BosatlBIMbvMZJYB2C0VAcEW4Dr9ApcpPbM9zYSCz7A&s=dmS3n52oSxBzBmWt0E1YgfkPxxwttyfkDBt_sW60f6I&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Wed Mar 25 17:01:33 2020 From: mweil at wustl.edu (Matt Weil) Date: Wed, 25 Mar 2020 12:01:33 -0500 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: thank you thank you... I would like to see that in IBM documentation somewhere. On 3/25/20 11:50 AM, Venkateswara R Puvvada wrote: > Matt, > > It is recommended to have dedicated AFM gateway nodes. Memory and CPU > requirements for AFM gateway node depends on the number of filesets > handled by the node and the inode usage of those filesets. Since AFM > keeps track of changes in the memory, any network disturbance can > cause the memory utilization to go high and which eventually leads to > in-memory queue to be dropped. After the queue is dropped, AFM runs > recovery to recover the lost operations which is expensive as it > involves creating the snapshot, running policy scan, doing readdir > from home/secondary and build the list of ?lost operations. When the > gateway node goes down, all the filesets handled by that node > distributed to the remaining active gateway nodes. After the gateway > node comes back, filesets are transferred back to the original gateway > node. When designing the gateway node, make sure that it have enough > memory , CPU resources for handling the incoming and outgoing data > based on the bandwidth. Limit the filesets per gateway(ex. less than > 20 filesets per gateway) ?so that number of AFM recoveries triggered > will be minimal when the queues are lost. Also limit the total number > of inodes handled by the gateway node across all the filesets (ex. > less than 400 million inodes per gateway). ?AFM gateway nodes are > licensed as server nodes. > > > ~Venkat (vpuvvada at in.ibm.com) > > > > From: ? ? ? ?Matt Weil > To: ? ? ? ?gpfsug-discuss at spectrumscale.org > Date: ? ? ? ?03/23/2020 11:39 PM > Subject: ? ? ? ?[EXTERNAL] [gpfsug-discuss] AFM gateway node scaling > Sent by: ? ? ? ?gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > Hello all, > > Is there any guide and or recommendation as to how to scale this. > > filesets per gateway node?? Is it necessary to separate NSD server and > gateway roles.? Are dedicated gateway nodes licensed as clients? > > Thanks for any guidance. > > Matt > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From vpuvvada at in.ibm.com Thu Mar 26 09:03:45 2020 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Thu, 26 Mar 2020 14:33:45 +0530 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: Most of these recommendations documented in KC, we will add missing information on number of filesets and inodes per gateway in the next release. https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_gatewaynodefailureafm.htm ~Venkat (vpuvvada at in.ibm.com) From: Matt Weil To: gpfsug-discuss at spectrumscale.org Date: 03/25/2020 10:34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] AFM gateway node scaling Sent by: gpfsug-discuss-bounces at spectrumscale.org thank you thank you... I would like to see that in IBM documentation somewhere. On 3/25/20 11:50 AM, Venkateswara R Puvvada wrote: Matt, It is recommended to have dedicated AFM gateway nodes. Memory and CPU requirements for AFM gateway node depends on the number of filesets handled by the node and the inode usage of those filesets. Since AFM keeps track of changes in the memory, any network disturbance can cause the memory utilization to go high and which eventually leads to in-memory queue to be dropped. After the queue is dropped, AFM runs recovery to recover the lost operations which is expensive as it involves creating the snapshot, running policy scan, doing readdir from home/secondary and build the list of lost operations. When the gateway node goes down, all the filesets handled by that node distributed to the remaining active gateway nodes. After the gateway node comes back, filesets are transferred back to the original gateway node. When designing the gateway node, make sure that it have enough memory , CPU resources for handling the incoming and outgoing data based on the bandwidth. Limit the filesets per gateway(ex. less than 20 filesets per gateway) so that number of AFM recoveries triggered will be minimal when the queues are lost. Also limit the total number of inodes handled by the gateway node across all the filesets (ex. less than 400 million inodes per gateway). AFM gateway nodes are licensed as server nodes. ~Venkat (vpuvvada at in.ibm.com) From: Matt Weil To: gpfsug-discuss at spectrumscale.org Date: 03/23/2020 11:39 PM Subject: [EXTERNAL] [gpfsug-discuss] AFM gateway node scaling Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all, Is there any guide and or recommendation as to how to scale this. filesets per gateway node? Is it necessary to separate NSD server and gateway roles. Are dedicated gateway nodes licensed as clients? Thanks for any guidance. Matt _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=7I3h0KRloboZtWXLSSteXUiOKrzQPEohIjQqN8YdDec&s=WiXsp6jbHf0g1paCMaCkruQpEklQOYDQQ4yQaGbksqA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From willi.engeli at id.ethz.ch Thu Mar 26 09:27:07 2020 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Thu, 26 Mar 2020 09:27:07 +0000 Subject: [gpfsug-discuss] AFM gateway node scaling Message-ID: <4ea3a7b7d5504ceca414b489152cbdeb@id.ethz.ch> We are using AFM as well on a relatively small cluster. But if I see the recommended scaling factors, the cluster will grow in Gateway nodes quicker then everywhere else. I have had a recovery during a multiple week period. I understand the reasons for the limits. I would wish that the communication between home and cash system get realized in a way that latencies and number of requests are handled more efficiently. That would allow to set the limitations to Bandwidth and Volume. Kind regards Willi -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5803 bytes Desc: not available URL: From mweil at wustl.edu Thu Mar 26 19:26:14 2020 From: mweil at wustl.edu (Matt Weil) Date: Thu, 26 Mar 2020 14:26:14 -0500 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: Also I found no documentation that advised against having the gateway role on a nsd server.? There was advise to not run the gateway role on a CES node.? What is the recommendation there.? Would a SAN client or shared disk be preferred to keep the latency down. Thanks Matt On 3/26/20 4:03 AM, Venkateswara R Puvvada wrote: > Most of these recommendations documented in KC, we will add missing > information on number of filesets and inodes per gateway in the next > release. > > https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_gatewaynodefailureafm.htm > > ~Venkat (vpuvvada at in.ibm.com) > > > > From: ? ? ? ?Matt Weil > To: ? ? ? ?gpfsug-discuss at spectrumscale.org > Date: ? ? ? ?03/25/2020 10:34 PM > Subject: ? ? ? ?[EXTERNAL] Re: [gpfsug-discuss] AFM gateway node scaling > Sent by: ? ? ? ?gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > thank you thank you... I would like to see that in IBM documentation > somewhere. > > On 3/25/20 11:50 AM, Venkateswara R Puvvada wrote: > Matt, > > It is recommended to have dedicated AFM gateway nodes. Memory and CPU > requirements for AFM gateway node depends on the number of filesets > handled by the node and the inode usage of those filesets. Since AFM > keeps track of changes in the memory, any network disturbance can > cause the memory utilization to go high and which eventually leads to > in-memory queue to be dropped. After the queue is dropped, AFM runs > recovery to recover the lost operations which is expensive as it > involves creating the snapshot, running policy scan, doing readdir > from home/secondary and build the list of ?lost operations. When the > gateway node goes down, all the filesets handled by that node > distributed to the remaining active gateway nodes. After the gateway > node comes back, filesets are transferred back to the original gateway > node. When designing the gateway node, make sure that it have enough > memory , CPU resources for handling the incoming and outgoing data > based on the bandwidth. Limit the filesets per gateway(ex. less than > 20 filesets per gateway) ?so that number of AFM recoveries triggered > will be minimal when the queues are lost. Also limit the total number > of inodes handled by the gateway node across all the filesets (ex. > less than 400 million inodes per gateway). ?AFM gateway nodes are > licensed as server nodes. > > > ~Venkat (_vpuvvada at in.ibm.com_ ) > > > > From: ? ? ? ?Matt Weil __ > To: ? ? ? ?_gpfsug-discuss at spectrumscale.org_ > > Date: ? ? ? ?03/23/2020 11:39 PM > Subject: ? ? ? ?[EXTERNAL] [gpfsug-discuss] AFM gateway node scaling > Sent by: ? ? ? ?_gpfsug-discuss-bounces at spectrumscale.org_ > > > ------------------------------------------------------------------------ > > > > Hello all, > > Is there any guide and or recommendation as to how to scale this. > > filesets per gateway node? ?Is it necessary to separate NSD server and > gateway roles. ?Are dedicated gateway nodes licensed as clients? > > Thanks for any guidance. > > Matt > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org_ > __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_ > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_ > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnaineni at in.ibm.com Sun Mar 1 07:53:00 2020 From: mnaineni at in.ibm.com (Malahal R Naineni) Date: Sun, 1 Mar 2020 07:53:00 +0000 Subject: [gpfsug-discuss] Thousands of CLOSE_WAIT IPV6 connections on CES In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From scale at us.ibm.com Mon Mar 2 23:58:12 2020 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 2 Mar 2020 18:58:12 -0500 Subject: [gpfsug-discuss] Policy REGEX question In-Reply-To: <3D2C4651-AD7D-40FD-A1B0-1B22D501B0F3@mbari.org> References: <3D2C4651-AD7D-40FD-A1B0-1B22D501B0F3@mbari.org> Message-ID: The third option is to specify the flavor of regex desired. Right now, if specified, must be one of these: 'x','b','f', 'ix','ib' 'x' extended regular expressions - the default - as implemented by regcomp and regexec library functions with REG_EXTENDED flag 'b' basic regular expressions - without REG_EXTENDED 'f' glob like pattern matching as implemented by fnmatch library function The three 'i' are similar, except upper/lower case is ignored as implemented by the library function with the ignore case flag or option: REG_ICASE or FNM_CASEFOLD We haven't published this because it is not yet fully tested. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Todd Ruston To: gpfsug main discussion list Date: 02/20/2020 03:49 PM Subject: [EXTERNAL] [gpfsug-discuss] Policy REGEX question Sent by: gpfsug-discuss-bounces at spectrumscale.org Greetings, I've been working on creating some new policy rules that will require regular expression matching on path names. As a crutch to help me along, I've used the mmfind command to do some searches and used its policy output as a model. Interestingly, it creates REGEX() functions with an undocumented parameter. For example, the following REGEX expression was created in the WHERE clause by mmfind when searching for a pathname pattern: REGEX(PATH_NAME, '*/xy_survey_*/name/*.tif','f') The Scale policy documentation for REGEX only mentions 2 parameters, not 3: REGEX(String,'Pattern') Returns TRUE if the pattern matches, FALSE if it does not. Pattern is a Posix extended regular expression. (The above is from https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_stringfcts.htm ) Anyone know what that 3rd parameter is, what values are allowed there, and what they mean? My assumption is that it's some sort of selector for type of pattern matching engine, because that pattern (2nd parameter) isn't being handled as a standard regex (e.g. the *'s are treated as wildcards, not zero-or-more repeats). -- Todd E. Ruston Information Systems Manager Monterey Bay Aquarium Research Institute (MBARI) 7700 Sandholdt Road, Moss Landing, CA, 95039 Phone 831-775-1997 Fax 831-775-1652 http://www.mbari.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=NvrrhNye2hSn191BaCICEzG0U2qZ9XkpJrE3TCV_poM&s=UjdRykUGhYCQwUNfT9yEv7qI_cCONPk_Nwz21N3RDQk&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From yeep at robust.my Tue Mar 3 07:56:16 2020 From: yeep at robust.my (T.A. Yeep) Date: Tue, 03 Mar 2020 07:56:16 +0000 Subject: [gpfsug-discuss] Hello from Yeep @ Malaysia Message-ID: Hello everyone, My name is Yeep and my company is Robust HPC (www.robusthpc.com). We are building HPC cluster like animation renderfarm, seismic simulation, Bigdata/AI etc in Kuala Lumpur, Malaysia and wish to explore working with industry peers from South East Asia and China. I find Spectrum Scale a really interesting solution for unstructured data, especially with its AFM and TCT features. I'm also excited about pairing it with ICOS for long term archiving as well as how Scale could benefit for IoT deployment. I love meeting people, if anyone happens to come my place please feel free to say hello. =) -- Best regards T.A. Yeep -------------- next part -------------- An HTML attachment was scrubbed... URL: From agostino.funel at enea.it Wed Mar 4 10:05:26 2020 From: agostino.funel at enea.it (Agostino Funel) Date: Wed, 4 Mar 2020 11:05:26 +0100 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 Message-ID: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Hi, we have a GPFS cluster version 4.2.3.19. We have seen in the official "Administration and Programming Reference" (version 4 Release 2.0, pag. 28) that the read-only mount option (-o ro)? is not explicitly cited. This option is supported in version 5.*. However we tried, as a test, the -o ro mount option on two clients with the following SO and kernels client 1 CentOS Linux release 7.3.1611 (Core) uname -r 3.10.0-514.26.2.el7.x86_64 client 2 CentOS release 6.4 (Final) uname -r 2.6.32-358.23.2.el6.x86_64 and it worked fine. The -o ro option is non permanent in the sense that unmounting and remounting the file system on these clients it turned in r/w original state. Now, we have the necessity of setting read-only the file system on all clients of our cluster. The question is: could we safely use the -o ro option for all clients even if this option is not cited in the official (v. 4 release 2.0) documentation? Thank you very much. Best regards, Agostino Funel -- Agostino Funel DTE-ICT-HPC ENEA P.le E. Fermi 1 80055 Portici (Napoli) Italy Phone: (+39) 081-7723575 Fax: (+39) 081-7723344 E-mail: agostino.funel at enea.it WWW: http://www.afs.enea.it/funel ================================================== Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui e' stata inviata e' da qualificarsi quale strumento aziendale. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679). If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks. ================================================== From luis.bolinches at fi.ibm.com Wed Mar 4 11:19:21 2020 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 4 Mar 2020 11:19:21 +0000 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 In-Reply-To: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Message-ID: Hi >From phone so typos are expected. You maybe would like to look into remo mounts and export the FS only as ro to the client cluster. -- Cheers > On 4. Mar 2020, at 12.05, Agostino Funel wrote: > > Hi, > > we have a GPFS cluster version 4.2.3.19. We have seen in the official > "Administration and Programming Reference" (version 4 Release 2.0, pag. > 28) that the read-only mount option (-o ro) is not explicitly cited. > This option is supported in version 5.*. > > However we tried, as a test, the -o ro mount option on two clients with > the following SO and kernels > > client 1 > > CentOS Linux release 7.3.1611 (Core) > uname -r > 3.10.0-514.26.2.el7.x86_64 > > > client 2 > > CentOS release 6.4 (Final) > uname -r > 2.6.32-358.23.2.el6.x86_64 > > and it worked fine. > > The -o ro option is non permanent in the sense that unmounting and > remounting the file system on these clients it turned in r/w original state. > > Now, we have the necessity of setting read-only the file system on all > clients of our cluster. > > The question is: could we safely use the -o ro option for all clients > even if this option is not cited in the official (v. 4 release 2.0) > documentation? > > Thank you very much. > > Best regards, > > Agostino Funel > > > > > -- > Agostino Funel > DTE-ICT-HPC > ENEA > P.le E. Fermi 1 > 80055 Portici (Napoli) Italy > Phone: (+39) 081-7723575 > Fax: (+39) 081-7723344 > E-mail: agostino.funel at enea.it > WWW: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.afs.enea.it_funel&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=06wvjDocXteVI3UQgtzUEF6p7nx9KL9w1H1JZcBbe3w&s=-gYsQFW6qa06tu1S4OkYufeD-zczy5iX1CwrlkH_2HE&e= > > > > ================================================== > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui e' stata inviata e' da qualificarsi quale strumento aziendale. > La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). > Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie. > > This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679). > If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks. > > ================================================== > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=06wvjDocXteVI3UQgtzUEF6p7nx9KL9w1H1JZcBbe3w&s=wmL2HUe83nsiTZ7XmlW5GLpS7JcNytI9wjkJNmrpBgg&e= > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Wed Mar 4 11:24:01 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 4 Mar 2020 12:24:01 +0100 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 In-Reply-To: <87216c24-a638-64d2-f2ed-704985820661@enea.it> References: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Message-ID: I don?t know the answer ? but as an alternative solution, have you considered splitting the read only clients out into a separate cluster. Then you could enforce the read-only setting using ?mmauth grant ... -a ro?. That should be supported. -jf ons. 4. mar. 2020 kl. 12:05 skrev Agostino Funel : > Hi, > > we have a GPFS cluster version 4.2.3.19. We have seen in the official > "Administration and Programming Reference" (version 4 Release 2.0, pag. > 28) that the read-only mount option (-o ro) is not explicitly cited. > This option is supported in version 5.*. > > However we tried, as a test, the -o ro mount option on two clients with > the following SO and kernels > > client 1 > > CentOS Linux release 7.3.1611 (Core) > uname -r > 3.10.0-514.26.2.el7.x86_64 > > > client 2 > > CentOS release 6.4 (Final) > uname -r > 2.6.32-358.23.2.el6.x86_64 > > and it worked fine. > > The -o ro option is non permanent in the sense that unmounting and > remounting the file system on these clients it turned in r/w original > state. > > Now, we have the necessity of setting read-only the file system on all > clients of our cluster. > > The question is: could we safely use the -o ro option for all clients > even if this option is not cited in the official (v. 4 release 2.0) > documentation? > > Thank you very much. > > Best regards, > > Agostino Funel > > > > > -- > Agostino Funel > DTE-ICT-HPC > ENEA > P.le E. Fermi 1 > 80055 Portici (Napoli) Italy > Phone: (+39) 081-7723575 > Fax: (+39) 081-7723344 > E-mail: agostino.funel at enea.it > WWW: http://www.afs.enea.it/funel > > > > ================================================== > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate e la casella di posta elettronica da cui e' stata inviata > e' da qualificarsi quale strumento aziendale. > La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza > di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. > 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). > Qualora abbiate ricevuto questo documento per errore siete cortesemente > pregati di darne immediata comunicazione al mittente e di provvedere alla > sua distruzione. Grazie. > > This e-mail and any attachments is confidential and may contain privileged > information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is unauthorised > (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE > 2016/679). > If you are not the intended recipient, please delete this message and any > attachments and advise the sender by return e-mail. Thanks. > > ================================================== > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Wed Mar 4 21:02:57 2020 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Wed, 4 Mar 2020 15:02:57 -0600 Subject: [gpfsug-discuss] Hello from Yeep @ Malaysia In-Reply-To: References: Message-ID: Hi Yeep, "Hello" and thanks for reaching out to the team. We will keep an eye out for any future specific questions as you evaluate things further. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "T.A. Yeep" To: gpfsug-discuss at spectrumscale.org Date: 03/03/2020 01:56 AM Subject: [EXTERNAL] [gpfsug-discuss] Hello from Yeep @ Malaysia Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello everyone, My name is Yeep and my company is Robust HPC (www.robusthpc.com). We are building HPC cluster like animation renderfarm, seismic simulation, Bigdata/AI etc in Kuala Lumpur, Malaysia and wish to explore working with industry peers from South East Asia and China. I find Spectrum Scale a really interesting solution for unstructured data, especially with its AFM and TCT features. I'm also excited about pairing it with ICOS for long term archiving as well as how Scale could benefit for IoT deployment. I love meeting people, if anyone happens to come my place please feel free to say hello. =) -- Best regards T.A. Yeep _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=vqlz8Ipd9MevOJqKBqp2z36FwoITbgKqFya8GRFnOAE&s=z4TU3uhB83Yc5sWdafnL_SrbEVpo2sXJJcpVJwR1Dos&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From chair at spectrumscale.org Thu Mar 5 22:34:52 2020 From: chair at spectrumscale.org (Simon Thompson (Spectrum Scale User Group Chair)) Date: Thu, 05 Mar 2020 22:34:52 +0000 Subject: [gpfsug-discuss] SSUG Events 2020 update Message-ID: Registration is open for the May UK meeting. We plan to review 3 weeks before the event in w/c 20th April: current situation number of registrations any travel restrictions in place for speakers/IBM/attendees https://www.spectrumscaleug.org/event/worldwide-uk-2020-spectrum-scale-user-group/ There is also a new user day open for registration at: https://www.spectrumscaleug.org/event/worldwide-uk-spectrum-scale-2020-new-user-day/ Simon From: on behalf of "chair at spectrumscale.org" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Friday, 28 February 2020 at 08:55 To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] SSUG Events 2020 update Hi All, I thought it might be giving a little bit of an update on where we are with events this year. As you may know, SCAsia was cancelled in its entirety due to Covid-19 in Singapore and so there was no SSUG meeting. In the US, we struggled to find a venue to host the spring meeting and now time is a little short to arrange something for the end of March planned date. The IBM Spectrum Scale Strategy Days in Germany in March are currently still planned to happen next week. For the UK meeting (May), we haven?t yet opened registration but are planning to do so next week. We currently believe that as an event with 120-130 attendees, this is probably very low risk, but we?ll keep the current government advice under review as we approach the date. I would suggest that if you are planning to travel internationally to the UK event that you delay booking flights/book refundable transport and ensure you have adequate insurance in place in the event we have to cancel the event. For ISC in June, we currently don?t have a date, nor any firm plans to run an event this year. Simon Thompson UK group chair -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurence at qsplace.co.uk Fri Mar 6 18:30:39 2020 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Fri, 6 Mar 2020 18:30:39 +0000 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 In-Reply-To: References: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Message-ID: <9e43ea92-870a-9aa0-c92a-15f553d0fbc1@qsplace.co.uk> Exporting the filesystem to the remove cluster as RO is an excellent idea and works if all remote clients only need RO. Alternatively (and I believe this is still the case) you can set the "ro" filesystem mount option in the file "/var/mmfs/etc/localMountOptions" or "/var/mmfs/etc/localMountOptions." on all nodes. The issue with changing /etc/fstab is that in my experience GPFS has been known to override it. -- Lauz On 04/03/2020 11:24, Jan-Frode Myklebust wrote: > > I don?t know the answer ? but as an alternative solution, have you > considered splitting the read only clients out into a separate > cluster. Then you could enforce the read-only setting using ?mmauth > grant ... -a ro?. > > That should be supported. > > > > > ? -jf > > ons. 4. mar. 2020 kl. 12:05 skrev Agostino Funel > >: > > Hi, > > we have a GPFS cluster version 4.2.3.19. We have seen in the official > "Administration and Programming Reference" (version 4 Release 2.0, > pag. > 28) that the read-only mount option (-o ro)? is not explicitly cited. > This option is supported in version 5.*. > > However we tried, as a test, the -o ro mount option on two clients > with > the following SO and kernels > > client 1 > > CentOS Linux release 7.3.1611 (Core) > uname -r > 3.10.0-514.26.2.el7.x86_64 > > > client 2 > > CentOS release 6.4 (Final) > uname -r > 2.6.32-358.23.2.el6.x86_64 > > and it worked fine. > > The -o ro option is non permanent in the sense that unmounting and > remounting the file system on these clients it turned in r/w > original state. > > Now, we have the necessity of setting read-only the file system on > all > clients of our cluster. > > The question is: could we safely use the -o ro option for all clients > even if this option is not cited in the official (v. 4 release 2.0) > documentation? > > Thank you very much. > > Best regards, > > Agostino Funel > > > > > -- > Agostino Funel > DTE-ICT-HPC > ENEA > P.le E. Fermi 1 > 80055 Portici (Napoli) Italy > Phone: (+39) 081-7723575 > Fax: (+39) 081-7723344 > E-mail: agostino.funel at enea.it > WWW: http://www.afs.enea.it/funel > > > > ================================================== > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente > alle persone indicate e la casella di posta elettronica da cui e' > stata inviata e' da qualificarsi quale strumento aziendale. > La diffusione, copia o qualsiasi altra azione derivante dalla > conoscenza di queste informazioni sono rigorosamente vietate (art. > 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). > Qualora abbiate ricevuto questo documento per errore siete > cortesemente pregati di darne immediata comunicazione al mittente > e di provvedere alla sua distruzione. Grazie. > > This e-mail and any attachments is confidential and may contain > privileged information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is > unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent > amendments and GDPR UE 2016/679). > If you are not the intended recipient, please delete this message > and any attachments and advise the sender by return e-mail. Thanks. > > ================================================== > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Mar 7 00:52:06 2020 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 6 Mar 2020 18:52:06 -0600 Subject: [gpfsug-discuss] =?utf-8?q?Read-only_mount_option_for_GPFS_versio?= =?utf-8?b?bgk0LjIuMy4xOQ==?= In-Reply-To: <9e43ea92-870a-9aa0-c92a-15f553d0fbc1@qsplace.co.uk> References: <87216c24-a638-64d2-f2ed-704985820661@enea.it> <9e43ea92-870a-9aa0-c92a-15f553d0fbc1@qsplace.co.uk> Message-ID: With regard to your question: "The question is: could we safely use the -o ro option for all clients even if this option is not cited in the official (v. 4 release 2.0) documentation?" The answer is yes, because the '-o ro' option comes from the OS mount command and is not specific to Spectrum Scale's mmmount options (thus not documented that way). So you should be fine there. Next, consider setting that option with mmchfs to make it permanent vs on the mmmount command: mmchfs ?[-o MountOptions] or specifically mmchfs -o ro Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Laurence Horrocks-Barlow To: gpfsug-discuss at spectrumscale.org Date: 03/06/2020 01:03 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 Sent by: gpfsug-discuss-bounces at spectrumscale.org Exporting the filesystem to the remove cluster as RO is an excellent idea and works if all remote clients only need RO. Alternatively (and I believe this is still the case) you can set the "ro" filesystem mount option in the file "/var/mmfs/etc/localMountOptions" or "/var/mmfs/etc/localMountOptions." on all nodes. The issue with changing /etc/fstab is that in my experience GPFS has been known to override it. -- Lauz On 04/03/2020 11:24, Jan-Frode Myklebust wrote: I don?t know the answer ? but as an alternative solution, have you considered splitting the read only clients out into a separate cluster. Then you could enforce the read-only setting using ?mmauth grant ... -a ro?. That should be supported. ? -jf ons. 4. mar. 2020 kl. 12:05 skrev Agostino Funel < agostino.funel at enea.it>: Hi, we have a GPFS cluster version 4.2.3.19. We have seen in the official "Administration and Programming Reference" (version 4 Release 2.0, pag. 28) that the read-only mount option (-o ro)? is not explicitly cited. This option is supported in version 5.*. However we tried, as a test, the -o ro mount option on two clients with the following SO and kernels client 1 CentOS Linux release 7.3.1611 (Core) uname -r 3.10.0-514.26.2.el7.x86_64 client 2 CentOS release 6.4 (Final) uname -r 2.6.32-358.23.2.el6.x86_64 and it worked fine. The -o ro option is non permanent in the sense that unmounting and remounting the file system on these clients it turned in r/w original state. Now, we have the necessity of setting read-only the file system on all clients of our cluster. The question is: could we safely use the -o ro option for all clients even if this option is not cited in the official (v. 4 release 2.0) documentation? Thank you very much. Best regards, Agostino Funel -- Agostino Funel DTE-ICT-HPC ENEA P.le E. Fermi 1 80055 Portici (Napoli) Italy Phone: (+39) 081-7723575 Fax: (+39) 081-7723344 E-mail: agostino.funel at enea.it WWW: http://www.afs.enea.it/funel ================================================== Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui e' stata inviata e' da qualificarsi quale strumento aziendale. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679). If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks. ================================================== _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=Fx-WRuJQxi23Sd8N-mxMO4kCN6P0YBMQYWMxiLZhOEw&s=ggxOhPp6D3BSWttqakUi8m_gp61HBPQgrag0WRQ5_oc&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From cabrillo at ifca.unican.es Mon Mar 9 11:48:45 2020 From: cabrillo at ifca.unican.es (Iban Cabrillo) Date: Mon, 9 Mar 2020 12:48:45 +0100 (CET) Subject: [gpfsug-discuss] Gpfs Standar and JBOD enclosures Message-ID: <855324483.13939921.1583754525009.JavaMail.zimbra@ifca.unican.es> Dear, We have a cluster with more than 1PB running Spectrum Scale Standar 4.2.3-9 version. The filesystem that is running now is configured using a DAS systems with NSDs over a NetApp DDP (distributed raid 6). These days have arrived to our hands a pait of JBOB (D3284) enclosures and 4 new servers, the idea is connect it in pairs by two 2 x (1 enclosure + 2 SAS servers). Is there any way to add this enclosures to the actual Cluster using GNR (something similar) o this option is only available for GSS/ESS? If GNR is not an option, Is there any other way to add this enclosures to the cluster keeping the redundacy (path/server)? Maybe one failure group by enclosure with data replica set to 2 (we will lost the 50% of the sapace)? Regards, I -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Mon Mar 9 12:09:48 2020 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Mon, 9 Mar 2020 12:09:48 +0000 Subject: [gpfsug-discuss] Gpfs Standar and JBOD enclosures In-Reply-To: <855324483.13939921.1583754525009.JavaMail.zimbra@ifca.unican.es> Message-ID: Iban, Spectrum scale native raid will not be an option for your scenario. Scale erasure code edition does not support JBOD enclosures at this point. And Scale Native RAID is only certified for specific hardware ( IBM ESS or Lenovo GSS ) This means you either need to use Hardware raid controllers or a shared nothing disk architecture ( I would recommend the raid controllers ) Regards, Andrew Sent from my iPhone > On 9 Mar 2020, at 21:59, Iban Cabrillo wrote: > > ? > Dear, > We have a cluster with more than 1PB running Spectrum Scale Standar 4.2.3-9 version. > > The filesystem that is running now is configured using a DAS systems with NSDs over a NetApp DDP (distributed raid 6). > These days have arrived to our hands a pait of JBOB (D3284) enclosures and 4 new servers, the idea is connect it in pairs by two 2 x (1 enclosure + 2 SAS servers). > Is there any way to add this enclosures to the actual Cluster using GNR (something similar) o this option is only available for GSS/ESS? > If GNR is not an option, Is there any other way to add this enclosures to the cluster keeping the redundacy (path/server)? Maybe one failure group by enclosure with data replica set to 2 (we will lost the 50% of the sapace)? > > Regards, I > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phgrau at zedat.fu-berlin.de Fri Mar 13 12:41:55 2020 From: phgrau at zedat.fu-berlin.de (Philipp Grau) Date: Fri, 13 Mar 2020 13:41:55 +0100 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip "gpfs_maxstatcache_low". Message-ID: <20200313124155.GA94452@CIS.FU-Berlin.DE> Hello, we have a two node NSD cluster based on a DDN system. Currently we run Spectrum Scale 5.0.4.1 in an HPC environment. Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our current settings are: # mmdiag --config | grep -i cache ! maxFilesToCache 3000000 maxStatCache 10000 maxFilesToCache was tuned during installion and maxStatCache is the according default value. After discussing this issue on the german spectumscale meeting, I understand that it is difficult to give a formula on howto calulate this values. But I learnt that a FilesToCache entry costs about 10 kbytes of memory and a StatCache entry about 500 bytes. And typically maxStatCache should (obviously) be greater than maxFilesToCache. There is a average 100 GB memory usage on our systems (with a total of 265 GB RAM). So setting maxStatCache to at least 3000000 should be no problem. But is that correct or to high/low? Has anyone some hints or thoughts on this topic? Help is welcome. Regards, Philipp -- Philipp Grau | Freie Universitaet Berlin phgrau at ZEDAT.FU-Berlin.DE | Zentraleinrichtung fuer Datenverarbeitung Tel: +49 (30) 838 56583 | Fabeckstr. 32 Fax: +49 (30) 838 56721 | 14195 Berlin From Achim.Rehor at de.ibm.com Fri Mar 13 14:00:48 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Fri, 13 Mar 2020 15:00:48 +0100 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: <20200313124155.GA94452@CIS.FU-Berlin.DE> References: <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Fri Mar 13 14:01:18 2020 From: stockf at us.ibm.com (Frederick Stock) Date: Fri, 13 Mar 2020 14:01:18 +0000 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: <20200313124155.GA94452@CIS.FU-Berlin.DE> References: <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: An HTML attachment was scrubbed... URL: From ckrafft at de.ibm.com Fri Mar 13 14:59:52 2020 From: ckrafft at de.ibm.com (Christoph Krafft) Date: Fri, 13 Mar 2020 15:59:52 +0100 Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node Message-ID: Dear Spectrum Scale User Group, can somebody please share with me answers to the following setup / architectural questions: - What size is typically required for a descOnly NSD, that will only hold a copy of a filesystem descriptor area? - Having a 2-node "core-cluster" and a quorum node at a third site with locally attached descOnly NSDs to that q-node. What are the network latency / bandwidth requirements for traffic flowing between core-cluster-nodes and the q-node? (from my understanding the traffic contains primarily mmfsd communication - including FS descOnly updates - but no data/metadata I/O) Thank you in advance for answers - have a nice weekend. Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany IBM Data Privacy Statement IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4F403830.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From Paul.Sanchez at deshaw.com Fri Mar 13 15:06:14 2020 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 13 Mar 2020 15:06:14 +0000 Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node In-Reply-To: References: Message-ID: <86503c8905ea4db7b4f9e2f5c9c11716@deshaw.com> Per https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_nsdfail.htm and confirmed by several instances in a live environment... > This disk would exist solely to contain a replica of the file system descriptor (that is, it would not contain any file system metadata or data). This disk should be at least 128MB in size. From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Christoph Krafft Sent: Friday, March 13, 2020 11:00 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node This message was sent by an external party. Dear Spectrum Scale User Group, can somebody please share with me answers to the following setup / architectural questions: - What size is typically required for a descOnly NSD, that will only hold a copy of a filesystem descriptor area? - Having a 2-node "core-cluster" and a quorum node at a third site with locally attached descOnly NSDs to that q-node. What are the network latency / bandwidth requirements for traffic flowing between core-cluster-nodes and the q-node? (from my understanding the traffic contains primarily mmfsd communication - including FS descOnly updates - but no data/metadata I/O) Thank you in advance for answers - have a nice weekend. Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group ________________________________ Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH [cid:image002.gif at 01D5F927.69089AB0] Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany ________________________________ IBM Data Privacy Statement IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 166 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 1851 bytes Desc: image002.gif URL: From knop at us.ibm.com Fri Mar 13 15:21:22 2020 From: knop at us.ibm.com (Felipe Knop) Date: Fri, 13 Mar 2020 15:21:22 +0000 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: References: , <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Fri Mar 13 17:31:43 2020 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Fri, 13 Mar 2020 17:31:43 +0000 Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node In-Reply-To: <86503c8905ea4db7b4f9e2f5c9c11716@deshaw.com> References: <86503c8905ea4db7b4f9e2f5c9c11716@deshaw.com>, Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.image001.png at 01D5F927.69089AB0.png Type: image/png Size: 166 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.image002.gif at 01D5F927.69089AB0.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From phgrau at zedat.fu-berlin.de Sat Mar 14 13:12:11 2020 From: phgrau at zedat.fu-berlin.de (Philipp Grau) Date: Sat, 14 Mar 2020 14:12:11 +0100 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: References: <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: <20200314131211.GA717391@CIS.FU-Berlin.DE> Hello all, thank you all a lot for the feedback to my question. I think that I now understand the situation better. I will talk with my coworkers and we will find a better setting for the values... Many thanks, Philipp * Felipe Knop [13.03.20 16:22]: > All, > > Looks to me that the demands of the workload will dictate how many files we > should be cache, that is: maxStatCache + maxFilesToCache . > > The "mix" between maxStatCache and maxFilesToCache depends on how much memory > can be made available. Accessing files from maxFilesToCache is more efficient, > but stat cache entries use much less space. > > With the > > ! maxFilesToCache 3000000 > maxStatCache 10000 > > combination, the stat cache is not providing any significant help, since only > 0.3% of the files that are cached can fit in the stat cache. If enough memory > is available then maxStatCache could be increased to (say) 3000000, at a cost > of 1.4GB. But maxFilesToCache = 3000000 uses up to 27GB. The next questions > are then > > 1) Can such memory become available on the node, given the pagepool size ? > > 2) Does the workload require caching that many files? > > > Felipe > > ---- > Felipe Knop knop at us.ibm.com > GPFS Development and Security > IBM Systems > IBM Building 008 > 2455 South Rd, Poughkeepsie, NY 12601 > (845) 433-9314 T/L 293-9314 > > > > > ----- Original message ----- > From: "Frederick Stock" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip"gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 10:01 AM > > As you have learned there is no simple formula for setting the > maxStatToCache, or for that matter the maxFilesToCache, configuration > values. Memory is certainly one consideration but another is directory > listing operations. The information kept in the stat cache is sufficient > for fulfilling directory listings. If your users are doing directory > listings regularly then a larger stat cache could be helpful. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > stockf at us.ibm.com > > > > ----- Original message ----- > From: Philipp Grau > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: [EXTERNAL] [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip "gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 8:49 AM > > Hello, > > we have a two node NSD cluster based on a DDN system. Currently we > run Spectrum Scale 5.0.4.1 in an HPC environment. > > Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our current > settings are: > > # mmdiag --config | grep -i cache > ! maxFilesToCache 3000000 > maxStatCache 10000 > > maxFilesToCache was tuned during installion and maxStatCache is the > according default value. > > After discussing this issue on the german spectumscale meeting, I > understand that it is difficult to give a formula on howto calulate > this values. > > But I learnt that a FilesToCache entry costs about 10 kbytes of memory > and a StatCache entry about 500 bytes. And typically maxStatCache > should (obviously) be greater than maxFilesToCache. There is a average > 100 GB memory usage on our systems (with a total of 265 GB RAM). > > So setting maxStatCache to at least 3000000 should be no problem. But > is that correct or to high/low? > > Has anyone some hints or thoughts on this topic? Help is welcome. > > Regards, > > Philipp > From TOMP at il.ibm.com Mon Mar 16 08:49:00 2020 From: TOMP at il.ibm.com (Tomer Perry) Date: Mon, 16 Mar 2020 10:49:00 +0200 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: <20200314131211.GA717391@CIS.FU-Berlin.DE> References: <20200313124155.GA94452@CIS.FU-Berlin.DE> <20200314131211.GA717391@CIS.FU-Berlin.DE> Message-ID: Hi, Just remember to take into account the token memory aspect of the chosen values, especially for large clusters ( check https://www.spectrumscaleug.org/wp-content/uploads/2019/05/SSSD19DE-Day-2-B04-Spectrum-Scale-Memory-Usage.pdf for more details). Regards, Tomer Perry From: Philipp Grau To: gpfsug main discussion list Date: 14/03/2020 15:12 Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all, thank you all a lot for the feedback to my question. I think that I now understand the situation better. I will talk with my coworkers and we will find a better setting for the values... Many thanks, Philipp * Felipe Knop [13.03.20 16:22]: > All, > > Looks to me that the demands of the workload will dictate how many files we > should be cache, that is: maxStatCache + maxFilesToCache . > > The "mix" between maxStatCache and maxFilesToCache depends on how much memory > can be made available. Accessing files from maxFilesToCache is more efficient, > but stat cache entries use much less space. > > With the > > ! maxFilesToCache 3000000 > maxStatCache 10000 > > combination, the stat cache is not providing any significant help, since only > 0.3% of the files that are cached can fit in the stat cache. If enough memory > is available then maxStatCache could be increased to (say) 3000000, at a cost > of 1.4GB. But maxFilesToCache = 3000000 uses up to 27GB. The next questions > are then > > 1) Can such memory become available on the node, given the pagepool size ? > > 2) Does the workload require caching that many files? > > > Felipe > > ---- > Felipe Knop knop at us.ibm.com > GPFS Development and Security > IBM Systems > IBM Building 008 > 2455 South Rd, Poughkeepsie, NY 12601 > (845) 433-9314 T/L 293-9314 > > > > > ----- Original message ----- > From: "Frederick Stock" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip"gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 10:01 AM > > As you have learned there is no simple formula for setting the > maxStatToCache, or for that matter the maxFilesToCache, configuration > values. Memory is certainly one consideration but another is directory > listing operations. The information kept in the stat cache is sufficient > for fulfilling directory listings. If your users are doing directory > listings regularly then a larger stat cache could be helpful. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > stockf at us.ibm.com > > > > ----- Original message ----- > From: Philipp Grau > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: [EXTERNAL] [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip "gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 8:49 AM > > Hello, > > we have a two node NSD cluster based on a DDN system. Currently we > run Spectrum Scale 5.0.4.1 in an HPC environment. > > Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our current > settings are: > > # mmdiag --config | grep -i cache > ! maxFilesToCache 3000000 > maxStatCache 10000 > > maxFilesToCache was tuned during installion and maxStatCache is the > according default value. > > After discussing this issue on the german spectumscale meeting, I > understand that it is difficult to give a formula on howto calulate > this values. > > But I learnt that a FilesToCache entry costs about 10 kbytes of memory > and a StatCache entry about 500 bytes. And typically maxStatCache > should (obviously) be greater than maxFilesToCache. There is a average > 100 GB memory usage on our systems (with a total of 265 GB RAM). > > So setting maxStatCache to at least 3000000 should be no problem. But > is that correct or to high/low? > > Has anyone some hints or thoughts on this topic? Help is welcome. > > Regards, > > Philipp > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=mLPyKeOa1gNDrORvEXBgMw&m=OfAPrgGhpUsG419cks6FynnKaWgb84-BkzQXkaxraJw&s=HFKr-6vELN-qgWlEQsLH74gFoLShJnfLN89xNRNGocQ&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Mar 16 22:10:11 2020 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Mon, 16 Mar 2020 22:10:11 +0000 Subject: [gpfsug-discuss] SSUG Events 2020 update In-Reply-To: References: Message-ID: <7C0DB0DD-EEE5-412D-864E-E1048E46E71E@bham.ac.uk> Unfortunately, due to the escalating COVID-19 situation, this year we?re having to cancel the London user group meeting. We?re aware that IBM has cancelled a number of events through to the end of Q2, and whilst we?re independent of IBM, we rely on them to support the group with speakers and so we?ve taken the decision to cancel. Having spoken briefly with Ulf, Kristy and Bob, we?re hoping to bring a digital event of some sort though we need to work out the logistics of doing this and how we make sure it works for the community. We all recognise that the in-person meetings are better, but this year it looks like we?re going to struggle to run one for the foreseeable future! Thanks go to the sponsors who?d signed up to support us for this year?s UK event ? OCF, Lenovo, StarFish and NVIDIA. Simon UK Group Chair From: on behalf of "chair at spectrumscale.org" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Thursday, 5 March 2020 at 22:35 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] SSUG Events 2020 update Registration is open for the May UK meeting. We plan to review 3 weeks before the event in w/c 20th April: * current situation * number of registrations * any travel restrictions in place for speakers/IBM/attendees https://www.spectrumscaleug.org/event/worldwide-uk-2020-spectrum-scale-user-group/ There is also a new user day open for registration at: https://www.spectrumscaleug.org/event/worldwide-uk-spectrum-scale-2020-new-user-day/ Simon From: on behalf of "chair at spectrumscale.org" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Friday, 28 February 2020 at 08:55 To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] SSUG Events 2020 update Hi All, I thought it might be giving a little bit of an update on where we are with events this year. * As you may know, SCAsia was cancelled in its entirety due to Covid-19 in Singapore and so there was no SSUG meeting. * In the US, we struggled to find a venue to host the spring meeting and now time is a little short to arrange something for the end of March planned date. * The IBM Spectrum Scale Strategy Days in Germany in March are currently still planned to happen next week. * For the UK meeting (May), we haven?t yet opened registration but are planning to do so next week. We currently believe that as an event with 120-130 attendees, this is probably very low risk, but we?ll keep the current government advice under review as we approach the date. I would suggest that if you are planning to travel internationally to the UK event that you delay booking flights/book refundable transport and ensure you have adequate insurance in place in the event we have to cancel the event. * For ISC in June, we currently don?t have a date, nor any firm plans to run an event this year. Simon Thompson UK group chair -------------- next part -------------- An HTML attachment was scrubbed... URL: From sandeep.patil at in.ibm.com Tue Mar 17 08:07:58 2020 From: sandeep.patil at in.ibm.com (Sandeep Ramesh) Date: Tue, 17 Mar 2020 08:07:58 +0000 Subject: [gpfsug-discuss] Latest Technical Blogs/Papers on IBM Spectrum Scale (Q3 2019 - Q1 2020) In-Reply-To: References: Message-ID: Dear User Group Members, In continuation to this email thread, here are list of development blogs/Redpaper in the past 2 quarters . We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to this list. Redpaper HIPAA Compliance for Healthcare Workloads on IBM Spectrum Scale http://www.redbooks.ibm.com/abstracts/redp5591.html?Open IBM Spectrum Scale CSI Driver For Container Persistent Storage http://www.redbooks.ibm.com/redpieces/abstracts/redp5589.html?Open Cyber Resiliency Solution for IBM Spectrum Scale , Blueprint http://www.redbooks.ibm.com/abstracts/redp5559.html?Open Enhanced Cyber Security with IBM Spectrum Scale and IBM QRadar http://www.redbooks.ibm.com/abstracts/redp5560.html?Open Monitoring and Managing the IBM Elastic Storage Server Using the GUI http://www.redbooks.ibm.com/abstracts/redp5471.html?Open IBM Hybrid Solution for Scalable Data Solutions using IBM Spectrum Scale http://www.redbooks.ibm.com/abstracts/redp5549.html?Open IBM Spectrum Discover: Metadata Management for Deep Insight of Unstructured Storage http://www.redbooks.ibm.com/abstracts/redp5550.html?Open Monitoring and Managing IBM Spectrum Scale Using the GUI http://www.redbooks.ibm.com/abstracts/redp5458.html?Open IBM Reference Architecture for High Performance Data and AI in Healthcare and Life Sciences, http://www.redbooks.ibm.com/abstracts/redp5481.html?Open Blogs: Why Storage and HIPAA Compliance for AI & Analytics Workloads for Healthcare https://developer.ibm.com/storage/2020/03/17/why-storage-and-hipaa-compliance-for-ai-analytics-workloads-for-healthcare/ Innovation via Integration ? Proactively Securing Your Unstructured Data from Cyber Threats & Attacks --> This was done based on your inputs (as a part of Security Survey) last year on need for Spectrum Scale integrayion with IDS a https://developer.ibm.com/storage/2020/02/24/innovation-via-integration-proactively-securing-your-unstructured-data-from-cyber-threats-attacks/ IBM Spectrum Scale CES HDFS Transparency support https://developer.ibm.com/storage/2020/02/03/ces-hdfs-transparency-support/ How to set up a remote cluster with IBM Spectrum Scale ? steps, limitations and troubleshooting https://developer.ibm.com/storage/2020/01/27/how-to-set-up-a-remote-cluster-with-ibm-spectrum-scale-steps-limitations-and-troubleshooting/ How to use IBM Spectrum Scale with CSI Operator 1.0 on Openshift 4.2 ? sample usage scenario with Tensorflow deployment https://developer.ibm.com/storage/2020/01/20/how-to-use-ibm-spectrum-scale-with-csi-operator-1-0-on-openshift-4-2-sample-usage-scenario-with-tensorflow-deployment/ Achieving WORM like functionality from NFS/SMB clients for data on Spectrum Scale https://developer.ibm.com/storage/2020/01/10/achieving-worm-like-functionality-from-nfs-smb-clients-for-data-on-spectrum-scale/ IBM Spectrum Scale CSI driver video blogs, https://developer.ibm.com/storage/2019/12/26/ibm-spectrum-scale-csi-driver-video-blogs/ IBM Spectrum Scale CSI Driver v1.0.0 released https://developer.ibm.com/storage/2019/12/10/ibm-spectrum-scale-csi-driver-v1-0-0-released/ Now configure IBM? Spectrum Scale with Overlapping UNIXMAP ranges https://developer.ibm.com/storage/2019/11/12/now-configure-ibm-spectrum-scale-with-overlapping-unixmap-ranges/ ?mmadquery?, a Powerful tool helps check AD settings from Spectrum Scale https://developer.ibm.com/storage/2019/11/11/mmadquery-a-powerful-tool-helps-check-ad-settings-from-spectrum-scale/ Spectrum Scale Data Security Modes, https://developer.ibm.com/storage/2019/10/31/spectrum-scale-data-security-modes/ IBM Spectrum Scale for Linux on IBM Z ? What?s new in IBM Spectrum Scale 5.0.4 ? https://developer.ibm.com/storage/2019/10/25/ibm-spectrum-scale-for-linux-on-ibm-z-whats-new-in-ibm-spectrum-scale-5-0-4/ IBM Spectrum Scale installation toolkit ? enhancements over releases ? 5.0.4.0 https://developer.ibm.com/storage/2019/10/18/ibm-spectrum-scale-installation-toolkit-enhancements-over-releases-5-0-4-0/ IBM Spectrum Scale CSI driver beta on GitHub, https://developer.ibm.com/storage/2019/09/26/ibm-spectrum-scale-csi-driver-on-github/ Help Article: Care to be taken when configuring AD with RFC2307 https://developer.ibm.com/storage/2019/09/18/help-article-care-to-be-taken-when-configuring-ad-with-rfc2307/ IBM Spectrum Scale Erasure Code Edition (ECE): Installation Demonstration https://developer.ibm.com/storage/2019/09/10/ibm-spectrum-scale-erasure-code-edition-ece-installation-demonstration/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 09/03/2019 10:58 AM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q2 2019) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q2 2019). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. Redpaper : IBM Power Systems Enterprise AI Solutions (W/ SPECTRUM SCALE) http://www.redbooks.ibm.com/redpieces/abstracts/redp5556.html?Open IBM Spectrum Scale Erasure Code Edition (ECE): Installation Demonstration https://www.youtube.com/watch?v=6If50EvgP-U Blogs: Using IBM Spectrum Scale as platform storage for running containerized Hadoop/Spark workloads https://developer.ibm.com/storage/2019/08/27/using-ibm-spectrum-scale-as-platform-storage-for-running-containerized-hadoop-spark-workloads/ Useful Tools for Spectrum Scale CES NFS https://developer.ibm.com/storage/2019/07/22/useful-tools-for-spectrum-scale-ces-nfs/ How to ensure NFS uses strong encryption algorithms for secure data in motion ? https://developer.ibm.com/storage/2019/07/19/how-to-ensure-nfs-uses-strong-encryption-algorithms-for-secure-data-in-motion/ Introducing IBM Spectrum Scale Erasure Code Edition https://developer.ibm.com/storage/2019/07/07/introducing-ibm-spectrum-scale-erasure-code-edition/ Spectrum Scale: Which Filesystem Encryption Algo to Consider ? https://developer.ibm.com/storage/2019/07/01/spectrum-scale-which-filesystem-encryption-algo-to-consider/ IBM Spectrum Scale HDFS Transparency Apache Hadoop 3.1.x Support https://developer.ibm.com/storage/2019/06/24/ibm-spectrum-scale-hdfs-transparency-apache-hadoop-3-0-x-support/ Enhanced features in Elastic Storage Server (ESS) 5.3.4 https://developer.ibm.com/storage/2019/06/19/enhanced-features-in-elastic-storage-server-ess-5-3-4/ Upgrading IBM Spectrum Scale Erasure Code Edition using installation toolkit https://developer.ibm.com/storage/2019/06/09/upgrading-ibm-spectrum-scale-erasure-code-edition-using-installation-toolkit/ Upgrading IBM Spectrum Scale sync replication / stretch cluster setup in PureApp https://developer.ibm.com/storage/2019/06/06/upgrading-ibm-spectrum-scale-sync-replication-stretch-cluster-setup/ GPFS config remote access with multiple network definitions https://developer.ibm.com/storage/2019/05/30/gpfs-config-remote-access-with-multiple-network-definitions/ IBM Spectrum Scale Erasure Code Edition Fault Tolerance https://developer.ibm.com/storage/2019/05/30/ibm-spectrum-scale-erasure-code-edition-fault-tolerance/ IBM Spectrum Scale for Linux on IBM Z ? What?s new in IBM Spectrum Scale 5.0.3 ? https://developer.ibm.com/storage/2019/05/02/ibm-spectrum-scale-for-linux-on-ibm-z-whats-new-in-ibm-spectrum-scale-5-0-3/ Understanding and Solving WBC_ERR_DOMAIN_NOT_FOUND error with Spectrum Scale https://crk10.wordpress.com/2019/07/21/solving-the-wbc-err-domain-not-found-nt-status-none-mapped-glitch-in-ibm-spectrum-scale/ Understanding and Solving NT_STATUS_INVALID_SID issue for SMB access with Spectrum Scale https://crk10.wordpress.com/2019/07/24/solving-nt_status_invalid_sid-for-smb-share-access-in-ibm-spectrum-scale/ mmadquery primer (apparatus to query Active Directory from IBM Spectrum Scale) https://crk10.wordpress.com/2019/07/27/mmadquery-primer-apparatus-to-query-active-directory-from-ibm-spectrum-scale/ How to configure RHEL host as Active Directory Client using SSSD https://crk10.wordpress.com/2019/07/28/configure-rhel-machine-as-active-directory-client-using-sssd/ How to configure RHEL host as LDAP client using nslcd https://crk10.wordpress.com/2019/07/28/configure-rhel-machine-as-ldap-client-using-nslcd/ Solving NFSv4 AUTH_SYS nobody ownership issue https://crk10.wordpress.com/2019/07/29/nfsv4-auth_sys-nobody-ownership-and-idmapd/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ Consolidation list of all blogs and collaterals. https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Blogs%2C%20White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 04/29/2019 12:12 PM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q1 2019) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q1 2019). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. Spectrum Scale 5.0.3 https://developer.ibm.com/storage/2019/04/24/spectrum-scale-5-0-3/ IBM Spectrum Scale HDFS Transparency Ranger Support https://developer.ibm.com/storage/2019/04/01/ibm-spectrum-scale-hdfs-transparency-ranger-support/ Integration of IBM Aspera Sync with IBM Spectrum Scale: Protecting and Sharing Files Globally, http://www.redbooks.ibm.com/abstracts/redp5527.html?Open Spectrum Scale user group in Singapore, 2019 https://developer.ibm.com/storage/2019/03/14/spectrum-scale-user-group-in-singapore-2019/ 7 traits to use Spectrum Scale to run container workload https://developer.ibm.com/storage/2019/02/26/7-traits-to-use-spectrum-scale-to-run-container-workload/ Health Monitoring of IBM Spectrum Scale Cluster via External Monitoring Framework https://developer.ibm.com/storage/2019/01/22/health-monitoring-of-ibm-spectrum-scale-cluster-via-external-monitoring-framework/ Migrating data from native HDFS to IBM Spectrum Scale based shared storage https://developer.ibm.com/storage/2019/01/18/migrating-data-from-native-hdfs-to-ibm-spectrum-scale-based-shared-storage/ Bulk File Creation useful for Test on Filesystems https://developer.ibm.com/storage/2019/01/16/bulk-file-creation-useful-for-test-on-filesystems/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Blogs%2C%20White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 01/14/2019 06:24 PM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q4 2018) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q4 2018). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. Redpaper: IBM Spectrum Scale and IBM StoredIQ: Identifying and securing your business data to support regulatory requirements http://www.redbooks.ibm.com/abstracts/redp5525.html?Open IBM Spectrum Scale Memory Usage https://www.slideshare.net/tomerperry/ibm-spectrum-scale-memory-usage?qid=50a1dfda-3102-484f-b9d0-14b69fc4800b&v=&b=&from_search=2 Spectrum Scale and Containers https://developer.ibm.com/storage/2018/12/20/spectrum-scale-and-containers/ IBM Elastic Storage Server Performance Graphical Visualization with Grafana https://developer.ibm.com/storage/2018/12/18/ibm-elastic-storage-server-performance-graphical-visualization-with-grafana/ Hadoop Performance for disaggregated compute and storage configurations based on IBM Spectrum Scale Storage https://developer.ibm.com/storage/2018/12/13/hadoop-performance-for-disaggregated-compute-and-storage-configurations-based-on-ibm-spectrum-scale-storage/ EMS HA in ESS LE (Little Endian) environment https://developer.ibm.com/storage/2018/12/07/ems-ha-in-ess-le-little-endian-environment/ What?s new in ESS 5.3.2 https://developer.ibm.com/storage/2018/12/04/whats-new-in-ess-5-3-2/ Administer your Spectrum Scale cluster easily https://developer.ibm.com/storage/2018/11/13/administer-your-spectrum-scale-cluster-easily/ Disaster Recovery using Spectrum Scale?s Active File Management https://developer.ibm.com/storage/2018/11/13/disaster-recovery-using-spectrum-scales-active-file-management/ Recovery Group Failover Procedure of IBM Elastic Storage Server (ESS) https://developer.ibm.com/storage/2018/10/08/recovery-group-failover-procedure-ibm-elastic-storage-server-ess/ Whats new in IBM Elastic Storage Server (ESS) Version 5.3.1 and 5.3.1.1 https://developer.ibm.com/storage/2018/10/04/whats-new-ibm-elastic-storage-server-ess-version-5-3-1-5-3-1-1/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Blogs%2C%20White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 10/03/2018 08:48 PM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q3 2018) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q3 2018). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. How NFS exports became more dynamic with Spectrum Scale 5.0.2 https://developer.ibm.com/storage/2018/10/02/nfs-exports-became-dynamic-spectrum-scale-5-0-2/ HPC storage on AWS (IBM Spectrum Scale) https://developer.ibm.com/storage/2018/10/02/hpc-storage-aws-ibm-spectrum-scale/ Upgrade with Excluding the node(s) using Install-toolkit https://developer.ibm.com/storage/2018/09/30/upgrade-excluding-nodes-using-install-toolkit/ Offline upgrade using Install-toolkit https://developer.ibm.com/storage/2018/09/30/offline-upgrade-using-install-toolkit/ IBM Spectrum Scale for Linux on IBM Z ? What?s new in IBM Spectrum Scale 5.0.2 ? https://developer.ibm.com/storage/2018/09/21/ibm-spectrum-scale-for-linux-on-ibm-z-whats-new-in-ibm-spectrum-scale-5-0-2/ What?s New in IBM Spectrum Scale 5.0.2 ? https://developer.ibm.com/storage/2018/09/15/whats-new-ibm-spectrum-scale-5-0-2/ Starting IBM Spectrum Scale 5.0.2 release, the installation toolkit supports upgrade rerun if fresh upgrade fails. https://developer.ibm.com/storage/2018/09/15/starting-ibm-spectrum-scale-5-0-2-release-installation-toolkit-supports-upgrade-rerun-fresh-upgrade-fails/ IBM Spectrum Scale installation toolkit ? enhancements over releases ? 5.0.2.0 https://developer.ibm.com/storage/2018/09/15/ibm-spectrum-scale-installation-toolkit-enhancements-releases-5-0-2-0/ Announcing HDP 3.0 support with IBM Spectrum Scale https://developer.ibm.com/storage/2018/08/31/announcing-hdp-3-0-support-ibm-spectrum-scale/ IBM Spectrum Scale Tuning Overview for Hadoop Workload https://developer.ibm.com/storage/2018/08/20/ibm-spectrum-scale-tuning-overview-hadoop-workload/ Making the Most of Multicloud Storage https://developer.ibm.com/storage/2018/08/13/making-multicloud-storage/ Disaster Recovery for Transparent Cloud Tiering using SOBAR https://developer.ibm.com/storage/2018/08/13/disaster-recovery-transparent-cloud-tiering-using-sobar/ Your Optimal Choice of AI Storage for Today and Tomorrow https://developer.ibm.com/storage/2018/08/10/spectrum-scale-ai-workloads/ Analyze IBM Spectrum Scale File Access Audit with ELK Stack https://developer.ibm.com/storage/2018/07/30/analyze-ibm-spectrum-scale-file-access-audit-elk-stack/ Mellanox SX1710 40G switch MLAG configuration for IBM ESS https://developer.ibm.com/storage/2018/07/12/mellanox-sx1710-40g-switcher-mlag-configuration/ Protocol Problem Determination Guide for IBM Spectrum Scale? ? SMB and NFS Access issues https://developer.ibm.com/storage/2018/07/10/protocol-problem-determination-guide-ibm-spectrum-scale-smb-nfs-access-issues/ Access Control in IBM Spectrum Scale Object https://developer.ibm.com/storage/2018/07/06/access-control-ibm-spectrum-scale-object/ IBM Spectrum Scale HDFS Transparency Docker support https://developer.ibm.com/storage/2018/07/06/ibm-spectrum-scale-hdfs-transparency-docker-support/ Protocol Problem Determination Guide for IBM Spectrum Scale? ? Log Collection https://developer.ibm.com/storage/2018/07/04/protocol-problem-determination-guide-ibm-spectrum-scale-log-collection/ Redpapers IBM Spectrum Scale Immutability Introduction, Configuration Guidance, and Use Cases http://www.redbooks.ibm.com/abstracts/redp5507.html?Open Certifications Assessment of the immutability function of IBM Spectrum Scale Version 5.0 in accordance to US SEC17a-4f, EU GDPR Article 21 Section 1, German and Swiss laws and regulations in collaboration with KPMG. Certificate: http://www.kpmg.de/bescheinigungen/RequestReport.aspx?DE968667B47544FF83F6CCDCF37E5FB5 Full assessment report: http://www.kpmg.de/bescheinigungen/RequestReport.aspx?B290411BE1224F5A9B4D24663BCD3C5D For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 07/03/2018 12:13 AM Subject: Re: Latest Technical Blogs on Spectrum Scale (Q2 2018) Dear User Group Members, In continuation , here are list of development blogs in the this quarter (Q2 2018). We now have over 100+ developer blogs. As discussed in User Groups, passing it along: IBM Spectrum Scale 5.0.1 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2018/06/15/6494/ IBM Spectrum Scale ILM Policies https://developer.ibm.com/storage/2018/06/02/ibm-spectrum-scale-ilm-policies/ IBM Spectrum Scale 5.0.1 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2018/06/15/6494/ Management GUI enhancements in IBM Spectrum Scale release 5.0.1 https://developer.ibm.com/storage/2018/05/18/management-gui-enhancements-in-ibm-spectrum-scale-release-5-0-1/ Managing IBM Spectrum Scale services through GUI https://developer.ibm.com/storage/2018/05/18/managing-ibm-spectrum-scale-services-through-gui/ Use AWS CLI with IBM Spectrum Scale? object storage https://developer.ibm.com/storage/2018/05/16/use-awscli-with-ibm-spectrum-scale-object-storage/ Hadoop Storage Tiering with IBM Spectrum Scale https://developer.ibm.com/storage/2018/05/09/hadoop-storage-tiering-ibm-spectrum-scale/ How many Files on my Filesystem? https://developer.ibm.com/storage/2018/05/07/many-files-filesystem/ Recording Spectrum Scale Object Stats for Potential Billing like Purpose using Elasticsearch https://developer.ibm.com/storage/2018/05/04/spectrum-scale-object-stats-for-billing-using-elasticsearch/ New features in IBM Elastic Storage Server (ESS) Version 5.3 https://developer.ibm.com/storage/2018/04/09/new-features-ibm-elastic-storage-server-ess-version-5-3/ Using IBM Spectrum Scale for storage in IBM Cloud Private (Missed to send earlier) https://medium.com/ibm-cloud/ibm-spectrum-scale-with-ibm-cloud-private-8bf801796f19 Redpapers Hortonworks Data Platform with IBM Spectrum Scale: Reference Guide for Building an Integrated Solution http://www.redbooks.ibm.com/redpieces/abstracts/redp5448.html, Enabling Hybrid Cloud Storage for IBM Spectrum Scale Using Transparent Cloud Tiering http://www.redbooks.ibm.com/abstracts/redp5411.html?Open SAP HANA and ESS: A Winning Combination (Update) http://www.redbooks.ibm.com/abstracts/redp5436.html?Open Others IBM Spectrum Scale Software Version Recommendation Preventive Service Planning (Updated) http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009703, IDC Infobrief: A Modular Approach to Genomics Infrastructure at Scale in HCLS https://www.ibm.com/common/ssi/cgi-bin/ssialias?htmlfid=37016937USEN& For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 03/27/2018 05:23 PM Subject: Re: Latest Technical Blogs on Spectrum Scale Dear User Group Members, In continuation , here are list of development blogs in the this quarter (Q1 2018). As discussed in User Groups, passing it along: GDPR Compliance and Unstructured Data Storage https://developer.ibm.com/storage/2018/03/27/gdpr-compliance-unstructure-data-storage/ IBM Spectrum Scale for Linux on IBM Z ? Release 5.0 features and highlights https://developer.ibm.com/storage/2018/03/09/ibm-spectrum-scale-linux-ibm-z-release-5-0-features-highlights/ Management GUI enhancements in IBM Spectrum Scale release 5.0.0 https://developer.ibm.com/storage/2018/01/18/gui-enhancements-in-spectrum-scale-release-5-0-0/ IBM Spectrum Scale 5.0.0 ? What?s new in NFS? https://developer.ibm.com/storage/2018/01/18/ibm-spectrum-scale-5-0-0-whats-new-nfs/ Benefits and implementation of Spectrum Scale sudo wrappers https://developer.ibm.com/storage/2018/01/15/benefits-implementation-spectrum-scale-sudo-wrappers/ IBM Spectrum Scale: Big Data and Analytics Solution Brief https://developer.ibm.com/storage/2018/01/15/ibm-spectrum-scale-big-data-analytics-solution-brief/ Variant Sub-blocks in Spectrum Scale 5.0 https://developer.ibm.com/storage/2018/01/11/spectrum-scale-variant-sub-blocks/ Compression support in Spectrum Scale 5.0.0 https://developer.ibm.com/storage/2018/01/11/compression-support-spectrum-scale-5-0-0/ IBM Spectrum Scale Versus Apache Hadoop HDFS https://developer.ibm.com/storage/2018/01/10/spectrumscale_vs_hdfs/ ESS Fault Tolerance https://developer.ibm.com/storage/2018/01/09/ess-fault-tolerance/ Genomic Workloads ? How To Get it Right From Infrastructure Point Of View. https://developer.ibm.com/storage/2018/01/06/genomic-workloads-get-right-infrastructure-point-view/ IBM Spectrum Scale On AWS Cloud : This video explains how to deploy IBM Spectrum Scale on AWS. This solution helps the users who require highly available access to a shared name space across multiple instances with good performance, without requiring an in-depth knowledge of IBM Spectrum Scale. Detailed Demo : https://www.youtube.com/watch?v=6j5Xj_d0bh4 Brief Demo : https://www.youtube.com/watch?v=-aMQKPW_RfY. For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Cc: Doris Conti/Poughkeepsie/IBM at IBMUS Date: 01/10/2018 12:13 PM Subject: Re: Latest Technical Blogs on Spectrum Scale Dear User Group Members, Here are list of development blogs in the last quarter. Passing it to this email group as Doris had got a feedback in the UG meetings to notify the members with the latest updates periodically. Genomic Workloads ? How To Get it Right From Infrastructure Point Of View. https://developer.ibm.com/storage/2018/01/06/genomic-workloads-get-right-infrastructure-point-view/ IBM Spectrum Scale Versus Apache Hadoop HDFS https://developer.ibm.com/storage/2018/01/10/spectrumscale_vs_hdfs/ ESS Fault Tolerance https://developer.ibm.com/storage/2018/01/09/ess-fault-tolerance/ IBM Spectrum Scale MMFSCK ? Savvy Enhancements https://developer.ibm.com/storage/2018/01/05/ibm-spectrum-scale-mmfsck-savvy-enhancements/ ESS Disk Management https://developer.ibm.com/storage/2018/01/02/ess-disk-management/ IBM Spectrum Scale Object Protocol On Ubuntu https://developer.ibm.com/storage/2018/01/01/ibm-spectrum-scale-object-protocol-ubuntu/ IBM Spectrum Scale 5.0 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2017/12/20/ibm-spectrum-scale-5-0-whats-new-object/ A Complete Guide to ? Protocol Problem Determination Guide for IBM Spectrum Scale? ? Part 1 https://developer.ibm.com/storage/2017/12/19/complete-guide-protocol-problem-determination-guide-ibm-spectrum-scale-1/ IBM Spectrum Scale installation toolkit ? enhancements over releases https://developer.ibm.com/storage/2017/12/15/ibm-spectrum-scale-installation-toolkit-enhancements-releases/ Network requirements in an Elastic Storage Server Setup https://developer.ibm.com/storage/2017/12/13/network-requirements-in-an-elastic-storage-server-setup/ Co-resident migration with Transparent cloud tierin https://developer.ibm.com/storage/2017/12/05/co-resident-migration-transparent-cloud-tierin/ IBM Spectrum Scale on Hortonworks HDP Hadoop clusters : A Complete Big Data Solution https://developer.ibm.com/storage/2017/12/05/ibm-spectrum-scale-hortonworks-hdp-hadoop-clusters-complete-big-data-solution/ Big data analytics with Spectrum Scale using remote cluster mount & multi-filesystem support https://developer.ibm.com/storage/2017/11/28/big-data-analytics-spectrum-scale-using-remote-cluster-mount-multi-filesystem-support/ IBM Spectrum Scale HDFS Transparency Short Circuit Write Support https://developer.ibm.com/storage/2017/11/28/ibm-spectrum-scale-hdfs-transparency-short-circuit-write-support/ IBM Spectrum Scale HDFS Transparency Federation Support https://developer.ibm.com/storage/2017/11/27/ibm-spectrum-scale-hdfs-transparency-federation-support/ How to configure and performance tuning different system workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-different-system-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Spark workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-spark-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning database workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-database-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Hadoop workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/24/configure-performance-tuning-hadoop-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ IBM Spectrum Scale Sharing Nothing Cluster Performance Tuning https://developer.ibm.com/storage/2017/11/24/ibm-spectrum-scale-sharing-nothing-cluster-performance-tuning/ How to Configure IBM Spectrum Scale? with NIS based Authentication. https://developer.ibm.com/storage/2017/11/21/configure-ibm-spectrum-scale-nis-based-authentication/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Cc: Doris Conti/Poughkeepsie/IBM at IBMUS Date: 11/16/2017 08:15 PM Subject: Latest Technical Blogs on Spectrum Scale Dear User Group members, Here are the Development Blogs in last 3 months on Spectrum Scale Technical Topics. Spectrum Scale Monitoring ? Know More ? https://developer.ibm.com/storage/2017/11/16/spectrum-scale-monitoring-know/ IBM Spectrum Scale 5.0 Release ? What?s coming ! https://developer.ibm.com/storage/2017/11/14/ibm-spectrum-scale-5-0-release-whats-coming/ Four Essentials things to know for managing data ACLs on IBM Spectrum Scale? from Windows https://developer.ibm.com/storage/2017/11/13/four-essentials-things-know-managing-data-acls-ibm-spectrum-scale-windows/ GSSUTILS: A new way of running SSR, Deploying or Upgrading ESS Server https://developer.ibm.com/storage/2017/11/13/gssutils/ IBM Spectrum Scale Object Authentication https://developer.ibm.com/storage/2017/11/02/spectrum-scale-object-authentication/ Video Surveillance ? Choosing the right storage https://developer.ibm.com/storage/2017/11/02/video-surveillance-choosing-right-storage/ IBM Spectrum scale object deep dive training with problem determination https://www.slideshare.net/SmitaRaut/ibm-spectrum-scale-object-deep-dive-training Spectrum Scale as preferred software defined storage for Ubuntu OpenStack https://developer.ibm.com/storage/2017/09/29/spectrum-scale-preferred-software-defined-storage-ubuntu-openstack/ IBM Elastic Storage Server 2U24 Storage ? an All-Flash offering, a performance workhorse https://developer.ibm.com/storage/2017/10/06/ess-5-2-flash-storage/ A Complete Guide to Configure LDAP-based authentication with IBM Spectrum Scale? for File Access https://developer.ibm.com/storage/2017/09/21/complete-guide-configure-ldap-based-authentication-ibm-spectrum-scale-file-access/ Deploying IBM Spectrum Scale on AWS Quick Start https://developer.ibm.com/storage/2017/09/18/deploy-ibm-spectrum-scale-on-aws-quick-start/ Monitoring Spectrum Scale Object metrics https://developer.ibm.com/storage/2017/09/14/monitoring-spectrum-scale-object-metrics/ Tier your data with ease to Spectrum Scale Private Cloud(s) using Moonwalk Universal https://developer.ibm.com/storage/2017/09/14/tier-data-ease-spectrum-scale-private-clouds-using-moonwalk-universal/ Why do I see owner as ?Nobody? for my export mounted using NFSV4 Protocol on IBM Spectrum Scale?? https://developer.ibm.com/storage/2017/09/08/see-owner-nobody-export-mounted-using-nfsv4-protocol-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory and LDAP https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-ldap/ IBM Spectrum Scale? Authentication using Active Directory and RFC2307 https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-rfc2307/ High Availability Implementation with IBM Spectrum Virtualize and IBM Spectrum Scale https://developer.ibm.com/storage/2017/08/30/high-availability-implementation-ibm-spectrum-virtualize-ibm-spectrum-scale/ 10 Frequently asked Questions on configuring Authentication using AD + AUTO ID mapping on IBM Spectrum Scale?. https://developer.ibm.com/storage/2017/08/04/10-frequently-asked-questions-configuring-authentication-using-ad-auto-id-mapping-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory https://developer.ibm.com/storage/2017/07/30/ibm-spectrum-scale-auth-using-active-directory/ Five cool things that you didn?t know Transparent Cloud Tiering on Spectrum Scale can do https://developer.ibm.com/storage/2017/07/29/five-cool-things-didnt-know-transparent-cloud-tiering-spectrum-scale-can/ IBM Spectrum Scale GUI videos https://developer.ibm.com/storage/2017/07/25/ibm-spectrum-scale-gui-videos/ IBM Spectrum Scale? Authentication ? Planning for NFS Access https://developer.ibm.com/storage/2017/07/24/ibm-spectrum-scale-planning-nfs-access/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media -------------- next part -------------- An HTML attachment was scrubbed... URL: From valleru at cbio.mskcc.org Fri Mar 20 21:18:42 2020 From: valleru at cbio.mskcc.org (Valleru, Lohit/Information Systems) Date: Fri, 20 Mar 2020 17:18:42 -0400 Subject: [gpfsug-discuss] Network switches/architecture for GPFS Message-ID: <35ADE287-4355-430B-83A0-A00C76525822@cbio.mskcc.org> Hello All, I would like to discuss or understand on which ethernet networking switches/architecture seems to work best with GPFS. We had thought about infiniband, but are not yet ready to move to infiniband because of the complexity/upgrade and debugging issues that come with it. Current hardware: We are currently using Arista 7328x 100G core switch for networking among the GPFS clusters and the compute nodes. It is heterogeneous network, with some of the servers on 10G/25G/100G with LACP and without LACP. For example: GPFS storage clusters either have 25G LACP, or 10G LACP, or a single 100G network port. Compute nodes range from 10G to 100G. Login nodes/transfer servers etc have 25G bonded. Most of the servers have Mellanox ConnectX-4 or ConnectX-5 adapters. But we also have few older Intel,Broadcom and Chelsio network cards in the clusters. Most of the transceivers that we use are Mellanox,Finisar,Intel. Issue: We had upgraded to the above switch recently, and we had seen that it is not able to handle the network traffic because of higher NSD servers bandwidth vs lower compute node bandwidth. One issue that we did see was a lot of network discards on the switch side and network congestion with slow IO performance on respective compute nodes. Once we enabled ECN - we did see that it had reduced the network congestion. We do see expels once in a while, but that is mostly related to the network errors or the host not responding. We observed that bonding/LACP does make expels much more trickier, so we have decided to go with no LACP until GPFS code gets better at handling LACP - which I think they are working on. We have heard that our current switch is a shallow buffer switch, and we would need a higher/deep buffer Arista switch to perform better with no congestion/lesser latency and more throughput. On the other side, Mellanox promises to use better ASIC design and buffer architecture with spine leaf design, instead of one deep buffer core switch to get better performance than Arista. Most of the applications that run on the clusters are either genomic applications on CPUs and deep learning applications on GPUs. All of our GPFS storage cluster versions are above 5.0.2 with the compute filesystems at 16M block size on near line rotating disks, and Flash storage at 512K block size. May I know if could feedback from anyone who is using Arista or Mellanox switches on the clusters to understand the pros and cons, stability and the performance numbers of the same? Thank you, Lohit -------------- next part -------------- An HTML attachment was scrubbed... URL: From cblack at nygenome.org Sun Mar 22 00:58:42 2020 From: cblack at nygenome.org (Christopher Black) Date: Sun, 22 Mar 2020 00:58:42 +0000 Subject: [gpfsug-discuss] Network switches/architecture for GPFS In-Reply-To: <35ADE287-4355-430B-83A0-A00C76525822@cbio.mskcc.org> References: <35ADE287-4355-430B-83A0-A00C76525822@cbio.mskcc.org> Message-ID: <74B39595-5718-4B88-B7B8-513942E1983D@nygenome.org> We?ve had good luck moving from older Mellanox 1710 ethernet switches to newer Arista ethernet switches. Our core is a pair of Arista 7508s primarily with 100G cards. Leaf switches are Arista 7280QR for racks with 40Gb-connected servers and 7280SR for racks w/ 10Gb-connected servers. Uplinks from leaf switches to core are multiple 100G connections. Our nsd servers are connected with dual-40Gb connections, each connection a separate Mellanox ConnectX-3 card to spread load and failure across separate pcie slots. Our compute nodes are primarily connected with dual-10Gb connections on Intel x520 or x710 nics (dual-port on a single nic). We also have some Cisco UCS nodes going through Cisco FI?s, these do not perform nearly as well and we?ve had some trouble with them and high bandwidth network storage, especially with defaults. We have some data transfer nodes connected at 2x40Gb, but other than that our only 40Gb-connected nodes are nsd servers. Any server, nsd or compute, uses lacp to bond and has mtu set to 9000. We also set: BONDING_OPTS="mode=4 miimon=100 xmit_hash_policy=layer3+4" For ecn, we have sysctl net.ipv4.tcp_ecn = 2. We also run primarily genomics applications. We have no experience with more recent Mellanox switches, but the ethernet software implementation on their older switches gave us plenty of problems. I?m not the network expert at our site, but they seem to like the Arista software much more than Mellanox. We run some non-default tcp/ip and ethernet settings, primarily from fasterdata.es.net recommendations. IBM?s older wiki notes about linux sysctls sometimes does not match es.net recommendation, and in those cases we generally go w/ es.net, especially as some of the IBM docs were written for older Linux kernels. However, there are some sysctl recommendations from IBM docs that are unique to gpfs (net.core.somaxconn). Regarding non-net tuning to improve gpfs stability, we?ve found the following are also important: vm.min_free_kbytes vm.dirty_bytes vm.dirty_background_bytes It took us a long time to figure out that on systems with lots of memory, many dirty pages could be buffered before being flushed out to network, resulting in a storm of heavy traffic that could cause problems for gpfs disk lease renewals and other control traffic to get through quick enough to avoid expels. For client NIC tuning, we set txqueuelen 10000 but I?ve read that this may not be necessary on newer kernels. On older nics, or even current intel nics with older firmware, we found turning some offload optimizations OFF made things better (gro, lro, gso, lso). I hope this helps you or others running gpfs on ethernet! -Chris From: on behalf of "Valleru, Lohit/Information Systems" Reply-To: gpfsug main discussion list Date: Friday, March 20, 2020 at 5:18 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Network switches/architecture for GPFS Hello All, I would like to discuss or understand on which ethernet networking switches/architecture seems to work best with GPFS. We had thought about infiniband, but are not yet ready to move to infiniband because of the complexity/upgrade and debugging issues that come with it. Current hardware: We are currently using Arista 7328x 100G core switch for networking among the GPFS clusters and the compute nodes. It is heterogeneous network, with some of the servers on 10G/25G/100G with LACP and without LACP. For example: GPFS storage clusters either have 25G LACP, or 10G LACP, or a single 100G network port. Compute nodes range from 10G to 100G. Login nodes/transfer servers etc have 25G bonded. Most of the servers have Mellanox ConnectX-4 or ConnectX-5 adapters. But we also have few older Intel,Broadcom and Chelsio network cards in the clusters. Most of the transceivers that we use are Mellanox,Finisar,Intel. Issue: We had upgraded to the above switch recently, and we had seen that it is not able to handle the network traffic because of higher NSD servers bandwidth vs lower compute node bandwidth. One issue that we did see was a lot of network discards on the switch side and network congestion with slow IO performance on respective compute nodes. Once we enabled ECN - we did see that it had reduced the network congestion. We do see expels once in a while, but that is mostly related to the network errors or the host not responding. We observed that bonding/LACP does make expels much more trickier, so we have decided to go with no LACP until GPFS code gets better at handling LACP - which I think they are working on. We have heard that our current switch is a shallow buffer switch, and we would need a higher/deep buffer Arista switch to perform better with no congestion/lesser latency and more throughput. On the other side, Mellanox promises to use better ASIC design and buffer architecture with spine leaf design, instead of one deep buffer core switch to get better performance than Arista. Most of the applications that run on the clusters are either genomic applications on CPUs and deep learning applications on GPUs. All of our GPFS storage cluster versions are above 5.0.2 with the compute filesystems at 16M block size on near line rotating disks, and Flash storage at 512K block size. May I know if could feedback from anyone who is using Arista or Mellanox switches on the clusters to understand the pros and cons, stability and the performance numbers of the same? Thank you, Lohit ________________________________ This message is for the recipient?s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kraemerf at de.ibm.com Mon Mar 23 07:46:54 2020 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Mon, 23 Mar 2020 08:46:54 +0100 Subject: [gpfsug-discuss] Network switches/architecture for Spectrum Scale (GPFS) In-Reply-To: References: Message-ID: Hi, > May I know if could feedback from anyone who is using Arista or Mellanox switches on the > clusters to understand the pros and cons, stability and the performance numbers of the same? Most current network switches are using Merchant Silicon ASICs instead of custom ASICs as in the past. (This is even true for some Cisco devices.) "Merchant silicon is a term used to described chips, usually ASICs, that are designed and made by an entity other than the company selling the switches in which they are used. I might be tempted to say such switches use off-the-shelf ASICs, though that might imply that I could buy these chips from a retail store." https://www.datacenterknowledge.com/networks/why-merchant-silicon-taking-over-data-center-network-market All of the Merchant Silicon ASICs lines have different family members from entry to high-end features. Switch vendors will pick and choose these ASICs by price and target market for the specific switch models. You can't tell from the "outside" which switch does offer which kind of performance. You need to do more research on the specific switch and use case. Most of the time Mellanox switches does offer very attractive features and performance levels for parallel I/O workloads but there are more options in the market. The NOS (Network OS) which is running on these new networking switches is also not a fixed option. You have some choice here. Some switches can be ordered with different NOS versions. Which NOS is the best fit for Spectrum Scale or other pFS's is still a topic to be researched. Last but not least the question of which NIC card is the best fit in this job is important. SmartNIC: A network interface card (network adapter) that offloads processing tasks that the system CPU would normally handle. Using its own on-board processor, the SmartNIC may be able to perform any combination of encryption/decryption, firewall, TCP/IP and HTTP processing. SmartNICs can be ASIC, FPGA, and System-on-a-Chip (SOC) based. Naturally vendors who make just one kind of NIC seem to insist that only the type of NIC they make should qualify as a SmartNIC. Mellanox is a good choice here but there is wide range of choices and features already in the market. -frank- Appendix: 1 - A Network-centric View of Scalable Storage by Andy Bechtolsheim, Chief Development Officer and Co-Founder, Arista Networks https://youtu.be/__LomgSguSc 2 - Broadcom Ships Tomahawk 4, Industry?s Highest Bandwidth Ethernet Switch Chip at 25.6 Terabits per Second ? Broadcom https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm56990-series https://packetpushers.net/podcast/network-break-264-broadcoms-new-tomahawk-4-hits-25-6tbps-juniper-announces-sd-lan-for-ex-switches/ https://www.telecompaper.com/news/broadcom-ships-tomahawk-4-ethernet-switch-chip--1319338 Broadcom delivered the StrataXGS Tomahawk 4 switch series, demonstrating an unprecedented 25.6 Tbps of Ethernet switching performance in a single device. 3 - Mellanox Introduces Revolutionary ConnectX-6 Dx and BlueField-2 Secure Cloud SmartNICs and I/O Processing Unit Solutions ConnectX-6 Dx and BlueField-2 Provide 200 Gb/s Ethernet and InfiniBand Connectivity, Enabling Next Generation of Clouds, Secure Data Centers and Storage Platforms https://www.mellanox.com/page/press_release_item?id=2195 Frank Kraemer IBM Consulting IT Specialist / Client Technical Architect Am Weiher 24, 65451 Kelsterbach, Germany mailto:kraemerf at de.ibm.com Mobile +49171-3043699 IBM Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Mon Mar 23 18:09:16 2020 From: mweil at wustl.edu (Matt Weil) Date: Mon, 23 Mar 2020 13:09:16 -0500 Subject: [gpfsug-discuss] AFM gateway node scaling Message-ID: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Hello all, Is there any guide and or recommendation as to how to scale this. filesets per gateway node?? Is it necessary to separate NSD server and gateway roles.? Are dedicated gateway nodes licensed as clients? Thanks for any guidance. Matt From peter.chase at metoffice.gov.uk Tue Mar 24 11:08:15 2020 From: peter.chase at metoffice.gov.uk (Chase, Peter) Date: Tue, 24 Mar 2020 11:08:15 +0000 Subject: [gpfsug-discuss] tctwatcher command Message-ID: Hello, The scale 5.0.4 documentation makes reference to a tctwatcher command, but I can't find the executable in any of the 5.0.4-3 RPMs I've deployed (including gpfs.tct.server, gpfs.tct.client, gpfs.kafka and gpfs.librdkafka). I'd wondered if it needed compiling from somewhere in /usr/lpp/mmfs/samples/ or maybe /opt/ibm/MCStore/samples/ but I can't find anything referencing it there either! Does anyone know where it is? Documentation: https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/b1ladm_integratingwfolderintocloud.htm It's also in the 5.0.3 doc. Regards, Pete Chase Mainframe, Virtualization and Storage team Met Office -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Wed Mar 25 10:01:04 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 10:01:04 +0000 Subject: [gpfsug-discuss] mmbackup monitoring Message-ID: What is the best way of monitoring whether or not mmbackup has managed to complete a backup successfully? Traditionally one use a TSM monitoring solution of your choice to make sure nodes where backing up (I am assuming mmbackup is being used in conjunction with TSM here). However mmbackup does not update the backup_end column in the filespaceview table (at least in 4.2) which makes things rather more complicated. The best I can come up with is querying the events table to see if the client schedule completed, but that gives a false sense of security as the schedule completing does not mean the backup completed as far as I know. What solutions are you all using, or does mmbackup in 5.x update the filespaceview table? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From jonathan.buzzard at strath.ac.uk Wed Mar 25 10:09:12 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 10:09:12 +0000 Subject: [gpfsug-discuss] GPFS 5 and supported rhel OS In-Reply-To: References: Message-ID: <91d02fd3-2af7-5880-e1f2-aaf9b1f8040a@strath.ac.uk> On 19/02/2020 23:34, Renata Maria Dart wrote: > Hi, I understand gpfs 4.2.3 is end of support this coming September. A planning question at this stage. Do IBM intend to hold to this date or is/could there be a relaxation due to COVID-19? Basically I was planning to do the upgrade this summer, but what with working from home I am less keen to do a a 4.2.3 to 5.x upgrade while not on hand to the actual hardware. Obviously if we have to we have to, just want to know where we stand. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From carlz at us.ibm.com Wed Mar 25 12:12:38 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Wed, 25 Mar 2020 12:12:38 +0000 Subject: [gpfsug-discuss] GPFS 5 and supported rhel OS Message-ID: <70ECB57D-C012-43FB-89F1-AF5810984DAF@us.ibm.com> So far we have not revisited the EOS date for 4.2.3, but I would not rule it out entirely if the lockdown continues well into the summer. If we did, the next likely EOS date would be April 30th. Even if we do postpone the date for 4.2.3, keep two other dates in mind for planning: - RHEL 6 support is coming to an end in November. We won't support Scale with RHEL 6 once Red Hat stops supporting RHEL 6 - RHEL 7 will be supported with 5.0.5, but not "5.next", the release scheduled for the second half of 2020. So you'll need to plan to adopt RHEL 8 before upgrading to Scale "5.next" As much as possible we are going to try to stick to our release cadence of twice a year even through these difficulties, including designating 5.0.5 for Extended Updates. "Keep Calm and Scale Out". Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com Message: 2 Date: Wed, 25 Mar 2020 10:09:12 +0000 From: Jonathan Buzzard To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] GPFS 5 and supported rhel OS Message-ID: <91d02fd3-2af7-5880-e1f2-aaf9b1f8040a at strath.ac.uk> Content-Type: text/plain; charset=utf-8; format=flowed On 19/02/2020 23:34, Renata Maria Dart wrote: > Hi, I understand gpfs 4.2.3 is end of support this coming September. A planning question at this stage. Do IBM intend to hold to this date or is/could there be a relaxation due to COVID-19? Basically I was planning to do the upgrade this summer, but what with working from home I am less keen to do a a 4.2.3 to 5.x upgrade while not on hand to the actual hardware. Obviously if we have to we have to, just want to know where we stand. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ------------------------------ From skylar2 at u.washington.edu Wed Mar 25 14:15:59 2020 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Wed, 25 Mar 2020 07:15:59 -0700 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: References: Message-ID: <20200325141558.2iwnalq3hwo3r5ha@illiuin> We execute mmbackup via a regular TSM client schedule with an incremental action, with a virtualmountpoint set to an empty, local "canary" directory. mmbackup runs as a preschedule command, and the client -domain parameter is set only to backup the canary directory. dsmc will backup the canary directory as a filespace only if mmbackup succeeds (exits with 0). We can then monitor the canary and infer the status of the associated GPFS filespace or fileset. On Wed, Mar 25, 2020 at 10:01:04AM +0000, Jonathan Buzzard wrote: > > What is the best way of monitoring whether or not mmbackup has managed to > complete a backup successfully? > > Traditionally one use a TSM monitoring solution of your choice to make sure > nodes where backing up (I am assuming mmbackup is being used in conjunction > with TSM here). > > However mmbackup does not update the backup_end column in the filespaceview > table (at least in 4.2) which makes things rather more complicated. > > The best I can come up with is querying the events table to see if the > client schedule completed, but that gives a false sense of security as the > schedule completing does not mean the backup completed as far as I know. > > What solutions are you all using, or does mmbackup in 5.x update the > filespaceview table? -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From pinto at scinet.utoronto.ca Wed Mar 25 14:38:55 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Wed, 25 Mar 2020 10:38:55 -0400 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <20200325141558.2iwnalq3hwo3r5ha@illiuin> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> Message-ID: Additionally, mmbackup creates by default a .mmbackupCfg directory on the root of the fileset where it dumps several files and directories with the progress of the backup. For instance: expiredFiles/, prepFiles/, updatedFiles/, dsminstr.log, ... You may then create a script to search these directories for logs/lists of what has happened, and generate a more detailed report of what happened during the backup. In our case I generate a daily report of how many files and how much data have been sent to the TSM server and deleted for each user, including their paths. You can do more tricks if you want. Jaime On 3/25/2020 10:15:59, Skylar Thompson wrote: > We execute mmbackup via a regular TSM client schedule with an incremental > action, with a virtualmountpoint set to an empty, local "canary" directory. > mmbackup runs as a preschedule command, and the client -domain parameter is > set only to backup the canary directory. dsmc will backup the canary > directory as a filespace only if mmbackup succeeds (exits with 0). We can > then monitor the canary and infer the status of the associated GPFS > filespace or fileset. > > On Wed, Mar 25, 2020 at 10:01:04AM +0000, Jonathan Buzzard wrote: >> >> What is the best way of monitoring whether or not mmbackup has managed to >> complete a backup successfully? >> >> Traditionally one use a TSM monitoring solution of your choice to make sure >> nodes where backing up (I am assuming mmbackup is being used in conjunction >> with TSM here). >> >> However mmbackup does not update the backup_end column in the filespaceview >> table (at least in 4.2) which makes things rather more complicated. >> >> The best I can come up with is querying the events table to see if the >> client schedule completed, but that gives a false sense of security as the >> schedule completing does not mean the backup completed as far as I know. >> >> What solutions are you all using, or does mmbackup in 5.x update the >> filespaceview table? > . . . ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From skylar2 at u.washington.edu Wed Mar 25 15:25:15 2020 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Wed, 25 Mar 2020 08:25:15 -0700 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> Message-ID: <20200325152515.rd2uyh5fckkwdx4p@illiuin> IIRC, I think you need to set 2 in the bit field of the DEBUGmmbackup environment variable. I had a long-term task to see what I could get out of that, but this just reminded me of it and current events might actually let me have time to look into it now... On Wed, Mar 25, 2020 at 10:38:55AM -0400, Jaime Pinto wrote: > Additionally, mmbackup creates by default a .mmbackupCfg directory on the root of the fileset where it dumps several files and directories with the progress of the backup. For instance: expiredFiles/, prepFiles/, updatedFiles/, dsminstr.log, ... > > You may then create a script to search these directories for logs/lists of what has happened, and generate a more detailed report of what happened during the backup. In our case I generate a daily report of how many files and how much data have been sent to the TSM server and deleted for each user, including their paths. You can do more tricks if you want. > > Jaime > > > On 3/25/2020 10:15:59, Skylar Thompson wrote: > > We execute mmbackup via a regular TSM client schedule with an incremental > > action, with a virtualmountpoint set to an empty, local "canary" directory. > > mmbackup runs as a preschedule command, and the client -domain parameter is > > set only to backup the canary directory. dsmc will backup the canary > > directory as a filespace only if mmbackup succeeds (exits with 0). We can > > then monitor the canary and infer the status of the associated GPFS > > filespace or fileset. > > > > On Wed, Mar 25, 2020 at 10:01:04AM +0000, Jonathan Buzzard wrote: > > > > > > What is the best way of monitoring whether or not mmbackup has managed to > > > complete a backup successfully? > > > > > > Traditionally one use a TSM monitoring solution of your choice to make sure > > > nodes where backing up (I am assuming mmbackup is being used in conjunction > > > with TSM here). > > > > > > However mmbackup does not update the backup_end column in the filespaceview > > > table (at least in 4.2) which makes things rather more complicated. > > > > > > The best I can come up with is querying the events table to see if the > > > client schedule completed, but that gives a false sense of security as the > > > schedule completing does not mean the backup completed as far as I know. > > > > > > What solutions are you all using, or does mmbackup in 5.x update the > > > filespaceview table? > > > > . > . > . ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto - Storage Analyst > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From heinrich.billich at id.ethz.ch Wed Mar 25 15:35:55 2020 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Wed, 25 Mar 2020 15:35:55 +0000 Subject: [gpfsug-discuss] GUI timeout when running HW_INVENTORY on little endian ESS server Message-ID: Hello, I did ask about this timeouts when the gui runs HW_INVENTORY before. Now I would like to know what the exact timeout value in the gui code is and if we can change it. I want to argue: If a xCat command takes X seconds but the GUI code timeouts after Y we know the command will fail if X > Y, hence we need to increase Y unless we can reduce X ... It's this function which raises the timeout: at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory If we can't fix the long execution time for the time being, can we raise the timeout value? I know this most likely is a Firmware issue with little endian power systems, but we won't update some more time. Thank you, Heiner debug: Running 'xcat.sh rinv '10.250.***' '*****' 'all' ' on node localhost err: com.ibm.fscc.common.exceptions.FsccException: Error executing rinv command. Exit code = 1; Command output = ; Command error = *****: [****]: Error: timeout at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory(InventoryAndStateHelper.java:92) at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.buildHardwareInventory(InventoryAndStateHelper.java:175) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.inner_run(InventoryRefreshTask.java:94) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.run(InventoryRefreshTask.java:72) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:227) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:199) at com.ibm.fscc.common.newscheduler.RefreshTaskIds.execute(RefreshTaskIds.java:482) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:80) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) ... debug: Running 'mmsysmonc event 'gui' 'xcat_nodelist_ok' ****-i' ***-i' ' on node localhost err: ***-i: Error executing rinv command. Exit code = 1; Command output = ; Command error = nas12io04b: [***]: Error: timeout ,*** -i: Error executing rinv command. Exit code = 1; Command output = ; Command error =***: [***]: Error: timeout err: com.ibm.fscc.cli.CommandException: EFSSG1150C Running specified task was unsuccessful. at com.ibm.fscc.cli.CommandException.createCommandException(CommandException.java:117) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:84) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) EFSSG1150C Running specified task was unsuccessful. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== From heinrich.billich at id.ethz.ch Wed Mar 25 15:53:04 2020 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Wed, 25 Mar 2020 15:53:04 +0000 Subject: [gpfsug-discuss] GUI timeout when running HW_INVENTORY on little endian ESS server In-Reply-To: References: Message-ID: <3C52599F-DC19-47F2-9CF2-90B905D43A1B@id.ethz.ch> Hello, Sorry, I was wrong. Looks like the timeout already happens in xCAT/rinv and the gui just reports it. What to some respect is good - now this is a purely xCAT/hardware issue. The GUI isn't involved any more. Kind regards Heiner /var/log/xcat/command.log: ==================================================== [Date] 2020-03-25 15:03:46 [ClientType] cli [Request] rinv ***** all [Response] ***: Error: timeout [NumberNodes] 1 [ElapsedTime] 97.085 s ==================================================== GUI: HW_INVENTORY * 2020-03-25 15:03:26 681436ms failed CmdRunTask.doExecute nas12io04b-i: Error executing rinv command. Exit code = 1; Command output = ; Command error =***: [**]: Error: timeout ?On 25.03.20, 16:35, "Billich Heinrich Rainer (ID SD)" wrote: Hello, I did ask about this timeouts when the gui runs HW_INVENTORY before. Now I would like to know what the exact timeout value in the gui code is and if we can change it. I want to argue: If a xCat command takes X seconds but the GUI code timeouts after Y we know the command will fail if X > Y, hence we need to increase Y unless we can reduce X ... It's this function which raises the timeout: at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory If we can't fix the long execution time for the time being, can we raise the timeout value? I know this most likely is a Firmware issue with little endian power systems, but we won't update some more time. Thank you, Heiner debug: Running 'xcat.sh rinv '10.250.***' '*****' 'all' ' on node localhost err: com.ibm.fscc.common.exceptions.FsccException: Error executing rinv command. Exit code = 1; Command output = ; Command error = *****: [****]: Error: timeout at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory(InventoryAndStateHelper.java:92) at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.buildHardwareInventory(InventoryAndStateHelper.java:175) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.inner_run(InventoryRefreshTask.java:94) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.run(InventoryRefreshTask.java:72) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:227) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:199) at com.ibm.fscc.common.newscheduler.RefreshTaskIds.execute(RefreshTaskIds.java:482) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:80) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) ... debug: Running 'mmsysmonc event 'gui' 'xcat_nodelist_ok' ****-i' ***-i' ' on node localhost err: ***-i: Error executing rinv command. Exit code = 1; Command output = ; Command error = nas12io04b: [***]: Error: timeout ,*** -i: Error executing rinv command. Exit code = 1; Command output = ; Command error =***: [***]: Error: timeout err: com.ibm.fscc.cli.CommandException: EFSSG1150C Running specified task was unsuccessful. at com.ibm.fscc.cli.CommandException.createCommandException(CommandException.java:117) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:84) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) EFSSG1150C Running specified task was unsuccessful. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== From jonathan.buzzard at strath.ac.uk Wed Mar 25 16:27:27 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 16:27:27 +0000 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <20200325141558.2iwnalq3hwo3r5ha@illiuin> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> Message-ID: <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> On 25/03/2020 14:15, Skylar Thompson wrote: > We execute mmbackup via a regular TSM client schedule with an incremental > action, with a virtualmountpoint set to an empty, local "canary" directory. > mmbackup runs as a preschedule command, and the client -domain parameter is > set only to backup the canary directory. dsmc will backup the canary > directory as a filespace only if mmbackup succeeds (exits with 0). We can > then monitor the canary and infer the status of the associated GPFS > filespace or fileset. > I prefer this approach I think than grovelling around in log files that could easily break on an update. Though there is a better approach which in my view IBM should be using already in mmbackup. It came to me this afternoon that one could use the TSM API for this. After a bit of Googling I find there is an API call dsmUpdateFS, which allows you to update the filespace information on the TSM server. Fields that you can update include DSM_FSUPD_OCCUPANCY DSM_FSUPD_CAPACITY DSM_FSUPD_BACKSTARTDATE DSM_FSUPD_BACKCOMPLETEDATE Information on the API call here https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/api/r_cmd_dsmupdatefs.html How do we submit this as a feature request again? That said in my view it's a bug in mmbackup. The latest in a very long line stretching back well over a decade that make mmbackup less than production ready rather than a feature request :-) I feel a breakout of a text editor and some C code coming on in the meantime. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From skylar2 at u.washington.edu Wed Mar 25 16:32:24 2020 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Wed, 25 Mar 2020 09:32:24 -0700 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> Message-ID: <20200325163224.jbibka2nukfhmc7u@illiuin> On Wed, Mar 25, 2020 at 04:27:27PM +0000, Jonathan Buzzard wrote: > On 25/03/2020 14:15, Skylar Thompson wrote: > > We execute mmbackup via a regular TSM client schedule with an incremental > > action, with a virtualmountpoint set to an empty, local "canary" directory. > > mmbackup runs as a preschedule command, and the client -domain parameter is > > set only to backup the canary directory. dsmc will backup the canary > > directory as a filespace only if mmbackup succeeds (exits with 0). We can > > then monitor the canary and infer the status of the associated GPFS > > filespace or fileset. > > > > I prefer this approach I think than grovelling around in log files that > could easily break on an update. Though there is a better approach which in > my view IBM should be using already in mmbackup. > > It came to me this afternoon that one could use the TSM API for this. After > a bit of Googling I find there is an API call dsmUpdateFS, which allows you > to update the filespace information on the TSM server. > > Fields that you can update include > > DSM_FSUPD_OCCUPANCY > DSM_FSUPD_CAPACITY > DSM_FSUPD_BACKSTARTDATE > DSM_FSUPD_BACKCOMPLETEDATE > > Information on the API call here > > https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/api/r_cmd_dsmupdatefs.html > > How do we submit this as a feature request again? That said in my view it's > a bug in mmbackup. The latest in a very long line stretching back well over > a decade that make mmbackup less than production ready rather than a feature > request :-) > > I feel a breakout of a text editor and some C code coming on in the > meantime. I actually tried using the API years ago to try to do some custom queries, and ran into the problem that custom API clients can only see data from custom API clients; they can't see data from the standard BA client. I contacted IBM about this, and they said it was a safety feature to prevent a rogue/poorly-written client from trashing regular backup/archive data, which makes some sense. Unfortunately, it does mean that IBM would have to be the source of the fix. -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From jonathan.buzzard at strath.ac.uk Wed Mar 25 16:45:54 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 16:45:54 +0000 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <20200325163224.jbibka2nukfhmc7u@illiuin> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> <20200325163224.jbibka2nukfhmc7u@illiuin> Message-ID: On 25/03/2020 16:32, Skylar Thompson wrote: > On Wed, Mar 25, 2020 at 04:27:27PM +0000, Jonathan Buzzard wrote: >> On 25/03/2020 14:15, Skylar Thompson wrote: >>> We execute mmbackup via a regular TSM client schedule with an incremental >>> action, with a virtualmountpoint set to an empty, local "canary" directory. >>> mmbackup runs as a preschedule command, and the client -domain parameter is >>> set only to backup the canary directory. dsmc will backup the canary >>> directory as a filespace only if mmbackup succeeds (exits with 0). We can >>> then monitor the canary and infer the status of the associated GPFS >>> filespace or fileset. >>> >> >> I prefer this approach I think than grovelling around in log files that >> could easily break on an update. Though there is a better approach which in >> my view IBM should be using already in mmbackup. >> >> It came to me this afternoon that one could use the TSM API for this. After >> a bit of Googling I find there is an API call dsmUpdateFS, which allows you >> to update the filespace information on the TSM server. >> >> Fields that you can update include >> >> DSM_FSUPD_OCCUPANCY >> DSM_FSUPD_CAPACITY >> DSM_FSUPD_BACKSTARTDATE >> DSM_FSUPD_BACKCOMPLETEDATE >> >> Information on the API call here >> >> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fsupport%2Fknowledgecenter%2FSSEQVQ_8.1.9%2Fapi%2Fr_cmd_dsmupdatefs.html&data=02%7C01%7Cjonathan.buzzard%40strath.ac.uk%7C8c7605146223442e8a2708d7d0dab99d%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C637207510145541248&sdata=QV9U0WxL5BTtR5%2Fasv1X202d9PqnLXZNg5bzX8KpUOo%3D&reserved=0 >> >> How do we submit this as a feature request again? That said in my view it's >> a bug in mmbackup. The latest in a very long line stretching back well over >> a decade that make mmbackup less than production ready rather than a feature >> request :-) >> >> I feel a breakout of a text editor and some C code coming on in the >> meantime. > > I actually tried using the API years ago to try to do some custom queries, > and ran into the problem that custom API clients can only see data from > custom API clients; they can't see data from the standard BA client. I > contacted IBM about this, and they said it was a safety feature to prevent > a rogue/poorly-written client from trashing regular backup/archive data, > which makes some sense. Unfortunately, it does mean that IBM would have to > be the source of the fix. > Grrr, I had forgotten that. Well then IBM need to fix this. Bug mmbackup does not update the occupancy, capacity, backup start date and backup end date when doing a backup. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From vpuvvada at in.ibm.com Wed Mar 25 16:50:41 2020 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Wed, 25 Mar 2020 22:20:41 +0530 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: Matt, It is recommended to have dedicated AFM gateway nodes. Memory and CPU requirements for AFM gateway node depends on the number of filesets handled by the node and the inode usage of those filesets. Since AFM keeps track of changes in the memory, any network disturbance can cause the memory utilization to go high and which eventually leads to in-memory queue to be dropped. After the queue is dropped, AFM runs recovery to recover the lost operations which is expensive as it involves creating the snapshot, running policy scan, doing readdir from home/secondary and build the list of lost operations. When the gateway node goes down, all the filesets handled by that node distributed to the remaining active gateway nodes. After the gateway node comes back, filesets are transferred back to the original gateway node. When designing the gateway node, make sure that it have enough memory , CPU resources for handling the incoming and outgoing data based on the bandwidth. Limit the filesets per gateway(ex. less than 20 filesets per gateway) so that number of AFM recoveries triggered will be minimal when the queues are lost. Also limit the total number of inodes handled by the gateway node across all the filesets (ex. less than 400 million inodes per gateway). AFM gateway nodes are licensed as server nodes. ~Venkat (vpuvvada at in.ibm.com) From: Matt Weil To: gpfsug-discuss at spectrumscale.org Date: 03/23/2020 11:39 PM Subject: [EXTERNAL] [gpfsug-discuss] AFM gateway node scaling Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all, Is there any guide and or recommendation as to how to scale this. filesets per gateway node? Is it necessary to separate NSD server and gateway roles. Are dedicated gateway nodes licensed as clients? Thanks for any guidance. Matt _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=BosatlBIMbvMZJYB2C0VAcEW4Dr9ApcpPbM9zYSCz7A&s=dmS3n52oSxBzBmWt0E1YgfkPxxwttyfkDBt_sW60f6I&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Wed Mar 25 17:01:33 2020 From: mweil at wustl.edu (Matt Weil) Date: Wed, 25 Mar 2020 12:01:33 -0500 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: thank you thank you... I would like to see that in IBM documentation somewhere. On 3/25/20 11:50 AM, Venkateswara R Puvvada wrote: > Matt, > > It is recommended to have dedicated AFM gateway nodes. Memory and CPU > requirements for AFM gateway node depends on the number of filesets > handled by the node and the inode usage of those filesets. Since AFM > keeps track of changes in the memory, any network disturbance can > cause the memory utilization to go high and which eventually leads to > in-memory queue to be dropped. After the queue is dropped, AFM runs > recovery to recover the lost operations which is expensive as it > involves creating the snapshot, running policy scan, doing readdir > from home/secondary and build the list of ?lost operations. When the > gateway node goes down, all the filesets handled by that node > distributed to the remaining active gateway nodes. After the gateway > node comes back, filesets are transferred back to the original gateway > node. When designing the gateway node, make sure that it have enough > memory , CPU resources for handling the incoming and outgoing data > based on the bandwidth. Limit the filesets per gateway(ex. less than > 20 filesets per gateway) ?so that number of AFM recoveries triggered > will be minimal when the queues are lost. Also limit the total number > of inodes handled by the gateway node across all the filesets (ex. > less than 400 million inodes per gateway). ?AFM gateway nodes are > licensed as server nodes. > > > ~Venkat (vpuvvada at in.ibm.com) > > > > From: ? ? ? ?Matt Weil > To: ? ? ? ?gpfsug-discuss at spectrumscale.org > Date: ? ? ? ?03/23/2020 11:39 PM > Subject: ? ? ? ?[EXTERNAL] [gpfsug-discuss] AFM gateway node scaling > Sent by: ? ? ? ?gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > Hello all, > > Is there any guide and or recommendation as to how to scale this. > > filesets per gateway node?? Is it necessary to separate NSD server and > gateway roles.? Are dedicated gateway nodes licensed as clients? > > Thanks for any guidance. > > Matt > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From vpuvvada at in.ibm.com Thu Mar 26 09:03:45 2020 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Thu, 26 Mar 2020 14:33:45 +0530 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: Most of these recommendations documented in KC, we will add missing information on number of filesets and inodes per gateway in the next release. https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_gatewaynodefailureafm.htm ~Venkat (vpuvvada at in.ibm.com) From: Matt Weil To: gpfsug-discuss at spectrumscale.org Date: 03/25/2020 10:34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] AFM gateway node scaling Sent by: gpfsug-discuss-bounces at spectrumscale.org thank you thank you... I would like to see that in IBM documentation somewhere. On 3/25/20 11:50 AM, Venkateswara R Puvvada wrote: Matt, It is recommended to have dedicated AFM gateway nodes. Memory and CPU requirements for AFM gateway node depends on the number of filesets handled by the node and the inode usage of those filesets. Since AFM keeps track of changes in the memory, any network disturbance can cause the memory utilization to go high and which eventually leads to in-memory queue to be dropped. After the queue is dropped, AFM runs recovery to recover the lost operations which is expensive as it involves creating the snapshot, running policy scan, doing readdir from home/secondary and build the list of lost operations. When the gateway node goes down, all the filesets handled by that node distributed to the remaining active gateway nodes. After the gateway node comes back, filesets are transferred back to the original gateway node. When designing the gateway node, make sure that it have enough memory , CPU resources for handling the incoming and outgoing data based on the bandwidth. Limit the filesets per gateway(ex. less than 20 filesets per gateway) so that number of AFM recoveries triggered will be minimal when the queues are lost. Also limit the total number of inodes handled by the gateway node across all the filesets (ex. less than 400 million inodes per gateway). AFM gateway nodes are licensed as server nodes. ~Venkat (vpuvvada at in.ibm.com) From: Matt Weil To: gpfsug-discuss at spectrumscale.org Date: 03/23/2020 11:39 PM Subject: [EXTERNAL] [gpfsug-discuss] AFM gateway node scaling Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all, Is there any guide and or recommendation as to how to scale this. filesets per gateway node? Is it necessary to separate NSD server and gateway roles. Are dedicated gateway nodes licensed as clients? Thanks for any guidance. Matt _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=7I3h0KRloboZtWXLSSteXUiOKrzQPEohIjQqN8YdDec&s=WiXsp6jbHf0g1paCMaCkruQpEklQOYDQQ4yQaGbksqA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From willi.engeli at id.ethz.ch Thu Mar 26 09:27:07 2020 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Thu, 26 Mar 2020 09:27:07 +0000 Subject: [gpfsug-discuss] AFM gateway node scaling Message-ID: <4ea3a7b7d5504ceca414b489152cbdeb@id.ethz.ch> We are using AFM as well on a relatively small cluster. But if I see the recommended scaling factors, the cluster will grow in Gateway nodes quicker then everywhere else. I have had a recovery during a multiple week period. I understand the reasons for the limits. I would wish that the communication between home and cash system get realized in a way that latencies and number of requests are handled more efficiently. That would allow to set the limitations to Bandwidth and Volume. Kind regards Willi -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5803 bytes Desc: not available URL: From mweil at wustl.edu Thu Mar 26 19:26:14 2020 From: mweil at wustl.edu (Matt Weil) Date: Thu, 26 Mar 2020 14:26:14 -0500 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: Also I found no documentation that advised against having the gateway role on a nsd server.? There was advise to not run the gateway role on a CES node.? What is the recommendation there.? Would a SAN client or shared disk be preferred to keep the latency down. Thanks Matt On 3/26/20 4:03 AM, Venkateswara R Puvvada wrote: > Most of these recommendations documented in KC, we will add missing > information on number of filesets and inodes per gateway in the next > release. > > https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_gatewaynodefailureafm.htm > > ~Venkat (vpuvvada at in.ibm.com) > > > > From: ? ? ? ?Matt Weil > To: ? ? ? ?gpfsug-discuss at spectrumscale.org > Date: ? ? ? ?03/25/2020 10:34 PM > Subject: ? ? ? ?[EXTERNAL] Re: [gpfsug-discuss] AFM gateway node scaling > Sent by: ? ? ? ?gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > thank you thank you... I would like to see that in IBM documentation > somewhere. > > On 3/25/20 11:50 AM, Venkateswara R Puvvada wrote: > Matt, > > It is recommended to have dedicated AFM gateway nodes. Memory and CPU > requirements for AFM gateway node depends on the number of filesets > handled by the node and the inode usage of those filesets. Since AFM > keeps track of changes in the memory, any network disturbance can > cause the memory utilization to go high and which eventually leads to > in-memory queue to be dropped. After the queue is dropped, AFM runs > recovery to recover the lost operations which is expensive as it > involves creating the snapshot, running policy scan, doing readdir > from home/secondary and build the list of ?lost operations. When the > gateway node goes down, all the filesets handled by that node > distributed to the remaining active gateway nodes. After the gateway > node comes back, filesets are transferred back to the original gateway > node. When designing the gateway node, make sure that it have enough > memory , CPU resources for handling the incoming and outgoing data > based on the bandwidth. Limit the filesets per gateway(ex. less than > 20 filesets per gateway) ?so that number of AFM recoveries triggered > will be minimal when the queues are lost. Also limit the total number > of inodes handled by the gateway node across all the filesets (ex. > less than 400 million inodes per gateway). ?AFM gateway nodes are > licensed as server nodes. > > > ~Venkat (_vpuvvada at in.ibm.com_ ) > > > > From: ? ? ? ?Matt Weil __ > To: ? ? ? ?_gpfsug-discuss at spectrumscale.org_ > > Date: ? ? ? ?03/23/2020 11:39 PM > Subject: ? ? ? ?[EXTERNAL] [gpfsug-discuss] AFM gateway node scaling > Sent by: ? ? ? ?_gpfsug-discuss-bounces at spectrumscale.org_ > > > ------------------------------------------------------------------------ > > > > Hello all, > > Is there any guide and or recommendation as to how to scale this. > > filesets per gateway node? ?Is it necessary to separate NSD server and > gateway roles. ?Are dedicated gateway nodes licensed as clients? > > Thanks for any guidance. > > Matt > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org_ > __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_ > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_ > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnaineni at in.ibm.com Sun Mar 1 07:53:00 2020 From: mnaineni at in.ibm.com (Malahal R Naineni) Date: Sun, 1 Mar 2020 07:53:00 +0000 Subject: [gpfsug-discuss] Thousands of CLOSE_WAIT IPV6 connections on CES In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From scale at us.ibm.com Mon Mar 2 23:58:12 2020 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 2 Mar 2020 18:58:12 -0500 Subject: [gpfsug-discuss] Policy REGEX question In-Reply-To: <3D2C4651-AD7D-40FD-A1B0-1B22D501B0F3@mbari.org> References: <3D2C4651-AD7D-40FD-A1B0-1B22D501B0F3@mbari.org> Message-ID: The third option is to specify the flavor of regex desired. Right now, if specified, must be one of these: 'x','b','f', 'ix','ib' 'x' extended regular expressions - the default - as implemented by regcomp and regexec library functions with REG_EXTENDED flag 'b' basic regular expressions - without REG_EXTENDED 'f' glob like pattern matching as implemented by fnmatch library function The three 'i' are similar, except upper/lower case is ignored as implemented by the library function with the ignore case flag or option: REG_ICASE or FNM_CASEFOLD We haven't published this because it is not yet fully tested. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Todd Ruston To: gpfsug main discussion list Date: 02/20/2020 03:49 PM Subject: [EXTERNAL] [gpfsug-discuss] Policy REGEX question Sent by: gpfsug-discuss-bounces at spectrumscale.org Greetings, I've been working on creating some new policy rules that will require regular expression matching on path names. As a crutch to help me along, I've used the mmfind command to do some searches and used its policy output as a model. Interestingly, it creates REGEX() functions with an undocumented parameter. For example, the following REGEX expression was created in the WHERE clause by mmfind when searching for a pathname pattern: REGEX(PATH_NAME, '*/xy_survey_*/name/*.tif','f') The Scale policy documentation for REGEX only mentions 2 parameters, not 3: REGEX(String,'Pattern') Returns TRUE if the pattern matches, FALSE if it does not. Pattern is a Posix extended regular expression. (The above is from https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.2/com.ibm.spectrum.scale.v5r02.doc/bl1adv_stringfcts.htm ) Anyone know what that 3rd parameter is, what values are allowed there, and what they mean? My assumption is that it's some sort of selector for type of pattern matching engine, because that pattern (2nd parameter) isn't being handled as a standard regex (e.g. the *'s are treated as wildcards, not zero-or-more repeats). -- Todd E. Ruston Information Systems Manager Monterey Bay Aquarium Research Institute (MBARI) 7700 Sandholdt Road, Moss Landing, CA, 95039 Phone 831-775-1997 Fax 831-775-1652 http://www.mbari.org _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=NvrrhNye2hSn191BaCICEzG0U2qZ9XkpJrE3TCV_poM&s=UjdRykUGhYCQwUNfT9yEv7qI_cCONPk_Nwz21N3RDQk&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From yeep at robust.my Tue Mar 3 07:56:16 2020 From: yeep at robust.my (T.A. Yeep) Date: Tue, 03 Mar 2020 07:56:16 +0000 Subject: [gpfsug-discuss] Hello from Yeep @ Malaysia Message-ID: Hello everyone, My name is Yeep and my company is Robust HPC (www.robusthpc.com). We are building HPC cluster like animation renderfarm, seismic simulation, Bigdata/AI etc in Kuala Lumpur, Malaysia and wish to explore working with industry peers from South East Asia and China. I find Spectrum Scale a really interesting solution for unstructured data, especially with its AFM and TCT features. I'm also excited about pairing it with ICOS for long term archiving as well as how Scale could benefit for IoT deployment. I love meeting people, if anyone happens to come my place please feel free to say hello. =) -- Best regards T.A. Yeep -------------- next part -------------- An HTML attachment was scrubbed... URL: From agostino.funel at enea.it Wed Mar 4 10:05:26 2020 From: agostino.funel at enea.it (Agostino Funel) Date: Wed, 4 Mar 2020 11:05:26 +0100 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 Message-ID: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Hi, we have a GPFS cluster version 4.2.3.19. We have seen in the official "Administration and Programming Reference" (version 4 Release 2.0, pag. 28) that the read-only mount option (-o ro)? is not explicitly cited. This option is supported in version 5.*. However we tried, as a test, the -o ro mount option on two clients with the following SO and kernels client 1 CentOS Linux release 7.3.1611 (Core) uname -r 3.10.0-514.26.2.el7.x86_64 client 2 CentOS release 6.4 (Final) uname -r 2.6.32-358.23.2.el6.x86_64 and it worked fine. The -o ro option is non permanent in the sense that unmounting and remounting the file system on these clients it turned in r/w original state. Now, we have the necessity of setting read-only the file system on all clients of our cluster. The question is: could we safely use the -o ro option for all clients even if this option is not cited in the official (v. 4 release 2.0) documentation? Thank you very much. Best regards, Agostino Funel -- Agostino Funel DTE-ICT-HPC ENEA P.le E. Fermi 1 80055 Portici (Napoli) Italy Phone: (+39) 081-7723575 Fax: (+39) 081-7723344 E-mail: agostino.funel at enea.it WWW: http://www.afs.enea.it/funel ================================================== Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui e' stata inviata e' da qualificarsi quale strumento aziendale. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679). If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks. ================================================== From luis.bolinches at fi.ibm.com Wed Mar 4 11:19:21 2020 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 4 Mar 2020 11:19:21 +0000 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 In-Reply-To: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Message-ID: Hi >From phone so typos are expected. You maybe would like to look into remo mounts and export the FS only as ro to the client cluster. -- Cheers > On 4. Mar 2020, at 12.05, Agostino Funel wrote: > > Hi, > > we have a GPFS cluster version 4.2.3.19. We have seen in the official > "Administration and Programming Reference" (version 4 Release 2.0, pag. > 28) that the read-only mount option (-o ro) is not explicitly cited. > This option is supported in version 5.*. > > However we tried, as a test, the -o ro mount option on two clients with > the following SO and kernels > > client 1 > > CentOS Linux release 7.3.1611 (Core) > uname -r > 3.10.0-514.26.2.el7.x86_64 > > > client 2 > > CentOS release 6.4 (Final) > uname -r > 2.6.32-358.23.2.el6.x86_64 > > and it worked fine. > > The -o ro option is non permanent in the sense that unmounting and > remounting the file system on these clients it turned in r/w original state. > > Now, we have the necessity of setting read-only the file system on all > clients of our cluster. > > The question is: could we safely use the -o ro option for all clients > even if this option is not cited in the official (v. 4 release 2.0) > documentation? > > Thank you very much. > > Best regards, > > Agostino Funel > > > > > -- > Agostino Funel > DTE-ICT-HPC > ENEA > P.le E. Fermi 1 > 80055 Portici (Napoli) Italy > Phone: (+39) 081-7723575 > Fax: (+39) 081-7723344 > E-mail: agostino.funel at enea.it > WWW: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.afs.enea.it_funel&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=06wvjDocXteVI3UQgtzUEF6p7nx9KL9w1H1JZcBbe3w&s=-gYsQFW6qa06tu1S4OkYufeD-zczy5iX1CwrlkH_2HE&e= > > > > ================================================== > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui e' stata inviata e' da qualificarsi quale strumento aziendale. > La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). > Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie. > > This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679). > If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks. > > ================================================== > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=06wvjDocXteVI3UQgtzUEF6p7nx9KL9w1H1JZcBbe3w&s=wmL2HUe83nsiTZ7XmlW5GLpS7JcNytI9wjkJNmrpBgg&e= > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Wed Mar 4 11:24:01 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 4 Mar 2020 12:24:01 +0100 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 In-Reply-To: <87216c24-a638-64d2-f2ed-704985820661@enea.it> References: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Message-ID: I don?t know the answer ? but as an alternative solution, have you considered splitting the read only clients out into a separate cluster. Then you could enforce the read-only setting using ?mmauth grant ... -a ro?. That should be supported. -jf ons. 4. mar. 2020 kl. 12:05 skrev Agostino Funel : > Hi, > > we have a GPFS cluster version 4.2.3.19. We have seen in the official > "Administration and Programming Reference" (version 4 Release 2.0, pag. > 28) that the read-only mount option (-o ro) is not explicitly cited. > This option is supported in version 5.*. > > However we tried, as a test, the -o ro mount option on two clients with > the following SO and kernels > > client 1 > > CentOS Linux release 7.3.1611 (Core) > uname -r > 3.10.0-514.26.2.el7.x86_64 > > > client 2 > > CentOS release 6.4 (Final) > uname -r > 2.6.32-358.23.2.el6.x86_64 > > and it worked fine. > > The -o ro option is non permanent in the sense that unmounting and > remounting the file system on these clients it turned in r/w original > state. > > Now, we have the necessity of setting read-only the file system on all > clients of our cluster. > > The question is: could we safely use the -o ro option for all clients > even if this option is not cited in the official (v. 4 release 2.0) > documentation? > > Thank you very much. > > Best regards, > > Agostino Funel > > > > > -- > Agostino Funel > DTE-ICT-HPC > ENEA > P.le E. Fermi 1 > 80055 Portici (Napoli) Italy > Phone: (+39) 081-7723575 > Fax: (+39) 081-7723344 > E-mail: agostino.funel at enea.it > WWW: http://www.afs.enea.it/funel > > > > ================================================== > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate e la casella di posta elettronica da cui e' stata inviata > e' da qualificarsi quale strumento aziendale. > La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza > di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. > 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). > Qualora abbiate ricevuto questo documento per errore siete cortesemente > pregati di darne immediata comunicazione al mittente e di provvedere alla > sua distruzione. Grazie. > > This e-mail and any attachments is confidential and may contain privileged > information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is unauthorised > (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE > 2016/679). > If you are not the intended recipient, please delete this message and any > attachments and advise the sender by return e-mail. Thanks. > > ================================================== > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Wed Mar 4 21:02:57 2020 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Wed, 4 Mar 2020 15:02:57 -0600 Subject: [gpfsug-discuss] Hello from Yeep @ Malaysia In-Reply-To: References: Message-ID: Hi Yeep, "Hello" and thanks for reaching out to the team. We will keep an eye out for any future specific questions as you evaluate things further. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "T.A. Yeep" To: gpfsug-discuss at spectrumscale.org Date: 03/03/2020 01:56 AM Subject: [EXTERNAL] [gpfsug-discuss] Hello from Yeep @ Malaysia Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello everyone, My name is Yeep and my company is Robust HPC (www.robusthpc.com). We are building HPC cluster like animation renderfarm, seismic simulation, Bigdata/AI etc in Kuala Lumpur, Malaysia and wish to explore working with industry peers from South East Asia and China. I find Spectrum Scale a really interesting solution for unstructured data, especially with its AFM and TCT features. I'm also excited about pairing it with ICOS for long term archiving as well as how Scale could benefit for IoT deployment. I love meeting people, if anyone happens to come my place please feel free to say hello. =) -- Best regards T.A. Yeep _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=vqlz8Ipd9MevOJqKBqp2z36FwoITbgKqFya8GRFnOAE&s=z4TU3uhB83Yc5sWdafnL_SrbEVpo2sXJJcpVJwR1Dos&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From chair at spectrumscale.org Thu Mar 5 22:34:52 2020 From: chair at spectrumscale.org (Simon Thompson (Spectrum Scale User Group Chair)) Date: Thu, 05 Mar 2020 22:34:52 +0000 Subject: [gpfsug-discuss] SSUG Events 2020 update Message-ID: Registration is open for the May UK meeting. We plan to review 3 weeks before the event in w/c 20th April: current situation number of registrations any travel restrictions in place for speakers/IBM/attendees https://www.spectrumscaleug.org/event/worldwide-uk-2020-spectrum-scale-user-group/ There is also a new user day open for registration at: https://www.spectrumscaleug.org/event/worldwide-uk-spectrum-scale-2020-new-user-day/ Simon From: on behalf of "chair at spectrumscale.org" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Friday, 28 February 2020 at 08:55 To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] SSUG Events 2020 update Hi All, I thought it might be giving a little bit of an update on where we are with events this year. As you may know, SCAsia was cancelled in its entirety due to Covid-19 in Singapore and so there was no SSUG meeting. In the US, we struggled to find a venue to host the spring meeting and now time is a little short to arrange something for the end of March planned date. The IBM Spectrum Scale Strategy Days in Germany in March are currently still planned to happen next week. For the UK meeting (May), we haven?t yet opened registration but are planning to do so next week. We currently believe that as an event with 120-130 attendees, this is probably very low risk, but we?ll keep the current government advice under review as we approach the date. I would suggest that if you are planning to travel internationally to the UK event that you delay booking flights/book refundable transport and ensure you have adequate insurance in place in the event we have to cancel the event. For ISC in June, we currently don?t have a date, nor any firm plans to run an event this year. Simon Thompson UK group chair -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurence at qsplace.co.uk Fri Mar 6 18:30:39 2020 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Fri, 6 Mar 2020 18:30:39 +0000 Subject: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 In-Reply-To: References: <87216c24-a638-64d2-f2ed-704985820661@enea.it> Message-ID: <9e43ea92-870a-9aa0-c92a-15f553d0fbc1@qsplace.co.uk> Exporting the filesystem to the remove cluster as RO is an excellent idea and works if all remote clients only need RO. Alternatively (and I believe this is still the case) you can set the "ro" filesystem mount option in the file "/var/mmfs/etc/localMountOptions" or "/var/mmfs/etc/localMountOptions." on all nodes. The issue with changing /etc/fstab is that in my experience GPFS has been known to override it. -- Lauz On 04/03/2020 11:24, Jan-Frode Myklebust wrote: > > I don?t know the answer ? but as an alternative solution, have you > considered splitting the read only clients out into a separate > cluster. Then you could enforce the read-only setting using ?mmauth > grant ... -a ro?. > > That should be supported. > > > > > ? -jf > > ons. 4. mar. 2020 kl. 12:05 skrev Agostino Funel > >: > > Hi, > > we have a GPFS cluster version 4.2.3.19. We have seen in the official > "Administration and Programming Reference" (version 4 Release 2.0, > pag. > 28) that the read-only mount option (-o ro)? is not explicitly cited. > This option is supported in version 5.*. > > However we tried, as a test, the -o ro mount option on two clients > with > the following SO and kernels > > client 1 > > CentOS Linux release 7.3.1611 (Core) > uname -r > 3.10.0-514.26.2.el7.x86_64 > > > client 2 > > CentOS release 6.4 (Final) > uname -r > 2.6.32-358.23.2.el6.x86_64 > > and it worked fine. > > The -o ro option is non permanent in the sense that unmounting and > remounting the file system on these clients it turned in r/w > original state. > > Now, we have the necessity of setting read-only the file system on > all > clients of our cluster. > > The question is: could we safely use the -o ro option for all clients > even if this option is not cited in the official (v. 4 release 2.0) > documentation? > > Thank you very much. > > Best regards, > > Agostino Funel > > > > > -- > Agostino Funel > DTE-ICT-HPC > ENEA > P.le E. Fermi 1 > 80055 Portici (Napoli) Italy > Phone: (+39) 081-7723575 > Fax: (+39) 081-7723344 > E-mail: agostino.funel at enea.it > WWW: http://www.afs.enea.it/funel > > > > ================================================== > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente > alle persone indicate e la casella di posta elettronica da cui e' > stata inviata e' da qualificarsi quale strumento aziendale. > La diffusione, copia o qualsiasi altra azione derivante dalla > conoscenza di queste informazioni sono rigorosamente vietate (art. > 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). > Qualora abbiate ricevuto questo documento per errore siete > cortesemente pregati di darne immediata comunicazione al mittente > e di provvedere alla sua distruzione. Grazie. > > This e-mail and any attachments is confidential and may contain > privileged information intended for the addressee(s) only. > Dissemination, copying, printing or use by anybody else is > unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent > amendments and GDPR UE 2016/679). > If you are not the intended recipient, please delete this message > and any attachments and advise the sender by return e-mail. Thanks. > > ================================================== > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Mar 7 00:52:06 2020 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 6 Mar 2020 18:52:06 -0600 Subject: [gpfsug-discuss] =?utf-8?q?Read-only_mount_option_for_GPFS_versio?= =?utf-8?b?bgk0LjIuMy4xOQ==?= In-Reply-To: <9e43ea92-870a-9aa0-c92a-15f553d0fbc1@qsplace.co.uk> References: <87216c24-a638-64d2-f2ed-704985820661@enea.it> <9e43ea92-870a-9aa0-c92a-15f553d0fbc1@qsplace.co.uk> Message-ID: With regard to your question: "The question is: could we safely use the -o ro option for all clients even if this option is not cited in the official (v. 4 release 2.0) documentation?" The answer is yes, because the '-o ro' option comes from the OS mount command and is not specific to Spectrum Scale's mmmount options (thus not documented that way). So you should be fine there. Next, consider setting that option with mmchfs to make it permanent vs on the mmmount command: mmchfs ?[-o MountOptions] or specifically mmchfs -o ro Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Laurence Horrocks-Barlow To: gpfsug-discuss at spectrumscale.org Date: 03/06/2020 01:03 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] Read-only mount option for GPFS version 4.2.3.19 Sent by: gpfsug-discuss-bounces at spectrumscale.org Exporting the filesystem to the remove cluster as RO is an excellent idea and works if all remote clients only need RO. Alternatively (and I believe this is still the case) you can set the "ro" filesystem mount option in the file "/var/mmfs/etc/localMountOptions" or "/var/mmfs/etc/localMountOptions." on all nodes. The issue with changing /etc/fstab is that in my experience GPFS has been known to override it. -- Lauz On 04/03/2020 11:24, Jan-Frode Myklebust wrote: I don?t know the answer ? but as an alternative solution, have you considered splitting the read only clients out into a separate cluster. Then you could enforce the read-only setting using ?mmauth grant ... -a ro?. That should be supported. ? -jf ons. 4. mar. 2020 kl. 12:05 skrev Agostino Funel < agostino.funel at enea.it>: Hi, we have a GPFS cluster version 4.2.3.19. We have seen in the official "Administration and Programming Reference" (version 4 Release 2.0, pag. 28) that the read-only mount option (-o ro)? is not explicitly cited. This option is supported in version 5.*. However we tried, as a test, the -o ro mount option on two clients with the following SO and kernels client 1 CentOS Linux release 7.3.1611 (Core) uname -r 3.10.0-514.26.2.el7.x86_64 client 2 CentOS release 6.4 (Final) uname -r 2.6.32-358.23.2.el6.x86_64 and it worked fine. The -o ro option is non permanent in the sense that unmounting and remounting the file system on these clients it turned in r/w original state. Now, we have the necessity of setting read-only the file system on all clients of our cluster. The question is: could we safely use the -o ro option for all clients even if this option is not cited in the official (v. 4 release 2.0) documentation? Thank you very much. Best regards, Agostino Funel -- Agostino Funel DTE-ICT-HPC ENEA P.le E. Fermi 1 80055 Portici (Napoli) Italy Phone: (+39) 081-7723575 Fax: (+39) 081-7723344 E-mail: agostino.funel at enea.it WWW: http://www.afs.enea.it/funel ================================================== Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui e' stata inviata e' da qualificarsi quale strumento aziendale. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679). If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks. ================================================== _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=Fx-WRuJQxi23Sd8N-mxMO4kCN6P0YBMQYWMxiLZhOEw&s=ggxOhPp6D3BSWttqakUi8m_gp61HBPQgrag0WRQ5_oc&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From cabrillo at ifca.unican.es Mon Mar 9 11:48:45 2020 From: cabrillo at ifca.unican.es (Iban Cabrillo) Date: Mon, 9 Mar 2020 12:48:45 +0100 (CET) Subject: [gpfsug-discuss] Gpfs Standar and JBOD enclosures Message-ID: <855324483.13939921.1583754525009.JavaMail.zimbra@ifca.unican.es> Dear, We have a cluster with more than 1PB running Spectrum Scale Standar 4.2.3-9 version. The filesystem that is running now is configured using a DAS systems with NSDs over a NetApp DDP (distributed raid 6). These days have arrived to our hands a pait of JBOB (D3284) enclosures and 4 new servers, the idea is connect it in pairs by two 2 x (1 enclosure + 2 SAS servers). Is there any way to add this enclosures to the actual Cluster using GNR (something similar) o this option is only available for GSS/ESS? If GNR is not an option, Is there any other way to add this enclosures to the cluster keeping the redundacy (path/server)? Maybe one failure group by enclosure with data replica set to 2 (we will lost the 50% of the sapace)? Regards, I -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Mon Mar 9 12:09:48 2020 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Mon, 9 Mar 2020 12:09:48 +0000 Subject: [gpfsug-discuss] Gpfs Standar and JBOD enclosures In-Reply-To: <855324483.13939921.1583754525009.JavaMail.zimbra@ifca.unican.es> Message-ID: Iban, Spectrum scale native raid will not be an option for your scenario. Scale erasure code edition does not support JBOD enclosures at this point. And Scale Native RAID is only certified for specific hardware ( IBM ESS or Lenovo GSS ) This means you either need to use Hardware raid controllers or a shared nothing disk architecture ( I would recommend the raid controllers ) Regards, Andrew Sent from my iPhone > On 9 Mar 2020, at 21:59, Iban Cabrillo wrote: > > ? > Dear, > We have a cluster with more than 1PB running Spectrum Scale Standar 4.2.3-9 version. > > The filesystem that is running now is configured using a DAS systems with NSDs over a NetApp DDP (distributed raid 6). > These days have arrived to our hands a pait of JBOB (D3284) enclosures and 4 new servers, the idea is connect it in pairs by two 2 x (1 enclosure + 2 SAS servers). > Is there any way to add this enclosures to the actual Cluster using GNR (something similar) o this option is only available for GSS/ESS? > If GNR is not an option, Is there any other way to add this enclosures to the cluster keeping the redundacy (path/server)? Maybe one failure group by enclosure with data replica set to 2 (we will lost the 50% of the sapace)? > > Regards, I > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phgrau at zedat.fu-berlin.de Fri Mar 13 12:41:55 2020 From: phgrau at zedat.fu-berlin.de (Philipp Grau) Date: Fri, 13 Mar 2020 13:41:55 +0100 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip "gpfs_maxstatcache_low". Message-ID: <20200313124155.GA94452@CIS.FU-Berlin.DE> Hello, we have a two node NSD cluster based on a DDN system. Currently we run Spectrum Scale 5.0.4.1 in an HPC environment. Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our current settings are: # mmdiag --config | grep -i cache ! maxFilesToCache 3000000 maxStatCache 10000 maxFilesToCache was tuned during installion and maxStatCache is the according default value. After discussing this issue on the german spectumscale meeting, I understand that it is difficult to give a formula on howto calulate this values. But I learnt that a FilesToCache entry costs about 10 kbytes of memory and a StatCache entry about 500 bytes. And typically maxStatCache should (obviously) be greater than maxFilesToCache. There is a average 100 GB memory usage on our systems (with a total of 265 GB RAM). So setting maxStatCache to at least 3000000 should be no problem. But is that correct or to high/low? Has anyone some hints or thoughts on this topic? Help is welcome. Regards, Philipp -- Philipp Grau | Freie Universitaet Berlin phgrau at ZEDAT.FU-Berlin.DE | Zentraleinrichtung fuer Datenverarbeitung Tel: +49 (30) 838 56583 | Fabeckstr. 32 Fax: +49 (30) 838 56721 | 14195 Berlin From Achim.Rehor at de.ibm.com Fri Mar 13 14:00:48 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Fri, 13 Mar 2020 15:00:48 +0100 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: <20200313124155.GA94452@CIS.FU-Berlin.DE> References: <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Fri Mar 13 14:01:18 2020 From: stockf at us.ibm.com (Frederick Stock) Date: Fri, 13 Mar 2020 14:01:18 +0000 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: <20200313124155.GA94452@CIS.FU-Berlin.DE> References: <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: An HTML attachment was scrubbed... URL: From ckrafft at de.ibm.com Fri Mar 13 14:59:52 2020 From: ckrafft at de.ibm.com (Christoph Krafft) Date: Fri, 13 Mar 2020 15:59:52 +0100 Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node Message-ID: Dear Spectrum Scale User Group, can somebody please share with me answers to the following setup / architectural questions: - What size is typically required for a descOnly NSD, that will only hold a copy of a filesystem descriptor area? - Having a 2-node "core-cluster" and a quorum node at a third site with locally attached descOnly NSDs to that q-node. What are the network latency / bandwidth requirements for traffic flowing between core-cluster-nodes and the q-node? (from my understanding the traffic contains primarily mmfsd communication - including FS descOnly updates - but no data/metadata I/O) Thank you in advance for answers - have a nice weekend. Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany IBM Data Privacy Statement IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 4F403830.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From Paul.Sanchez at deshaw.com Fri Mar 13 15:06:14 2020 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 13 Mar 2020 15:06:14 +0000 Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node In-Reply-To: References: Message-ID: <86503c8905ea4db7b4f9e2f5c9c11716@deshaw.com> Per https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_nsdfail.htm and confirmed by several instances in a live environment... > This disk would exist solely to contain a replica of the file system descriptor (that is, it would not contain any file system metadata or data). This disk should be at least 128MB in size. From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Christoph Krafft Sent: Friday, March 13, 2020 11:00 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node This message was sent by an external party. Dear Spectrum Scale User Group, can somebody please share with me answers to the following setup / architectural questions: - What size is typically required for a descOnly NSD, that will only hold a copy of a filesystem descriptor area? - Having a 2-node "core-cluster" and a quorum node at a third site with locally attached descOnly NSDs to that q-node. What are the network latency / bandwidth requirements for traffic flowing between core-cluster-nodes and the q-node? (from my understanding the traffic contains primarily mmfsd communication - including FS descOnly updates - but no data/metadata I/O) Thank you in advance for answers - have a nice weekend. Mit freundlichen Gr??en / Sincerely Christoph Krafft Client Technical Specialist - Power Systems, IBM Systems Certified IT Specialist @ The Open Group ________________________________ Phone: +49 (0) 7034 643 2171 IBM Deutschland GmbH [cid:image002.gif at 01D5F927.69089AB0] Mobile: +49 (0) 160 97 81 86 12 Am Weiher 24 Email: ckrafft at de.ibm.com 65451 Kelsterbach Germany ________________________________ IBM Data Privacy Statement IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 166 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 1851 bytes Desc: image002.gif URL: From knop at us.ibm.com Fri Mar 13 15:21:22 2020 From: knop at us.ibm.com (Felipe Knop) Date: Fri, 13 Mar 2020 15:21:22 +0000 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: References: , <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: An HTML attachment was scrubbed... URL: From kevindjo at us.ibm.com Fri Mar 13 17:31:43 2020 From: kevindjo at us.ibm.com (Kevin D Johnson) Date: Fri, 13 Mar 2020 17:31:43 +0000 Subject: [gpfsug-discuss] Spectrum Scale Sizing Questions -> Typical size of descOnly NSD & Network latency/throughput requirements quorum-only node In-Reply-To: <86503c8905ea4db7b4f9e2f5c9c11716@deshaw.com> References: <86503c8905ea4db7b4f9e2f5c9c11716@deshaw.com>, Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.image001.png at 01D5F927.69089AB0.png Type: image/png Size: 166 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.image002.gif at 01D5F927.69089AB0.gif Type: image/gif Size: 1851 bytes Desc: not available URL: From phgrau at zedat.fu-berlin.de Sat Mar 14 13:12:11 2020 From: phgrau at zedat.fu-berlin.de (Philipp Grau) Date: Sat, 14 Mar 2020 14:12:11 +0100 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: References: <20200313124155.GA94452@CIS.FU-Berlin.DE> Message-ID: <20200314131211.GA717391@CIS.FU-Berlin.DE> Hello all, thank you all a lot for the feedback to my question. I think that I now understand the situation better. I will talk with my coworkers and we will find a better setting for the values... Many thanks, Philipp * Felipe Knop [13.03.20 16:22]: > All, > > Looks to me that the demands of the workload will dictate how many files we > should be cache, that is: maxStatCache + maxFilesToCache . > > The "mix" between maxStatCache and maxFilesToCache depends on how much memory > can be made available. Accessing files from maxFilesToCache is more efficient, > but stat cache entries use much less space. > > With the > > ! maxFilesToCache 3000000 > maxStatCache 10000 > > combination, the stat cache is not providing any significant help, since only > 0.3% of the files that are cached can fit in the stat cache. If enough memory > is available then maxStatCache could be increased to (say) 3000000, at a cost > of 1.4GB. But maxFilesToCache = 3000000 uses up to 27GB. The next questions > are then > > 1) Can such memory become available on the node, given the pagepool size ? > > 2) Does the workload require caching that many files? > > > Felipe > > ---- > Felipe Knop knop at us.ibm.com > GPFS Development and Security > IBM Systems > IBM Building 008 > 2455 South Rd, Poughkeepsie, NY 12601 > (845) 433-9314 T/L 293-9314 > > > > > ----- Original message ----- > From: "Frederick Stock" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip"gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 10:01 AM > > As you have learned there is no simple formula for setting the > maxStatToCache, or for that matter the maxFilesToCache, configuration > values. Memory is certainly one consideration but another is directory > listing operations. The information kept in the stat cache is sufficient > for fulfilling directory listings. If your users are doing directory > listings regularly then a larger stat cache could be helpful. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > stockf at us.ibm.com > > > > ----- Original message ----- > From: Philipp Grau > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: [EXTERNAL] [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip "gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 8:49 AM > > Hello, > > we have a two node NSD cluster based on a DDN system. Currently we > run Spectrum Scale 5.0.4.1 in an HPC environment. > > Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our current > settings are: > > # mmdiag --config | grep -i cache > ! maxFilesToCache 3000000 > maxStatCache 10000 > > maxFilesToCache was tuned during installion and maxStatCache is the > according default value. > > After discussing this issue on the german spectumscale meeting, I > understand that it is difficult to give a formula on howto calulate > this values. > > But I learnt that a FilesToCache entry costs about 10 kbytes of memory > and a StatCache entry about 500 bytes. And typically maxStatCache > should (obviously) be greater than maxFilesToCache. There is a average > 100 GB memory usage on our systems (with a total of 265 GB RAM). > > So setting maxStatCache to at least 3000000 should be no problem. But > is that correct or to high/low? > > Has anyone some hints or thoughts on this topic? Help is welcome. > > Regards, > > Philipp > From TOMP at il.ibm.com Mon Mar 16 08:49:00 2020 From: TOMP at il.ibm.com (Tomer Perry) Date: Mon, 16 Mar 2020 10:49:00 +0200 Subject: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". In-Reply-To: <20200314131211.GA717391@CIS.FU-Berlin.DE> References: <20200313124155.GA94452@CIS.FU-Berlin.DE> <20200314131211.GA717391@CIS.FU-Berlin.DE> Message-ID: Hi, Just remember to take into account the token memory aspect of the chosen values, especially for large clusters ( check https://www.spectrumscaleug.org/wp-content/uploads/2019/05/SSSD19DE-Day-2-B04-Spectrum-Scale-Memory-Usage.pdf for more details). Regards, Tomer Perry From: Philipp Grau To: gpfsug main discussion list Date: 14/03/2020 15:12 Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low". Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all, thank you all a lot for the feedback to my question. I think that I now understand the situation better. I will talk with my coworkers and we will find a better setting for the values... Many thanks, Philipp * Felipe Knop [13.03.20 16:22]: > All, > > Looks to me that the demands of the workload will dictate how many files we > should be cache, that is: maxStatCache + maxFilesToCache . > > The "mix" between maxStatCache and maxFilesToCache depends on how much memory > can be made available. Accessing files from maxFilesToCache is more efficient, > but stat cache entries use much less space. > > With the > > ! maxFilesToCache 3000000 > maxStatCache 10000 > > combination, the stat cache is not providing any significant help, since only > 0.3% of the files that are cached can fit in the stat cache. If enough memory > is available then maxStatCache could be increased to (say) 3000000, at a cost > of 1.4GB. But maxFilesToCache = 3000000 uses up to 27GB. The next questions > are then > > 1) Can such memory become available on the node, given the pagepool size ? > > 2) Does the workload require caching that many files? > > > Felipe > > ---- > Felipe Knop knop at us.ibm.com > GPFS Development and Security > IBM Systems > IBM Building 008 > 2455 South Rd, Poughkeepsie, NY 12601 > (845) 433-9314 T/L 293-9314 > > > > > ----- Original message ----- > From: "Frederick Stock" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip"gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 10:01 AM > > As you have learned there is no simple formula for setting the > maxStatToCache, or for that matter the maxFilesToCache, configuration > values. Memory is certainly one consideration but another is directory > listing operations. The information kept in the stat cache is sufficient > for fulfilling directory listings. If your users are doing directory > listings regularly then a larger stat cache could be helpful. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > stockf at us.ibm.com > > > > ----- Original message ----- > From: Philipp Grau > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: [EXTERNAL] [gpfsug-discuss] maxStatCache and maxFilesToCache: > Tip "gpfs_maxstatcache_low". > Date: Fri, Mar 13, 2020 8:49 AM > > Hello, > > we have a two node NSD cluster based on a DDN system. Currently we > run Spectrum Scale 5.0.4.1 in an HPC environment. > > Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our current > settings are: > > # mmdiag --config | grep -i cache > ! maxFilesToCache 3000000 > maxStatCache 10000 > > maxFilesToCache was tuned during installion and maxStatCache is the > according default value. > > After discussing this issue on the german spectumscale meeting, I > understand that it is difficult to give a formula on howto calulate > this values. > > But I learnt that a FilesToCache entry costs about 10 kbytes of memory > and a StatCache entry about 500 bytes. And typically maxStatCache > should (obviously) be greater than maxFilesToCache. There is a average > 100 GB memory usage on our systems (with a total of 265 GB RAM). > > So setting maxStatCache to at least 3000000 should be no problem. But > is that correct or to high/low? > > Has anyone some hints or thoughts on this topic? Help is welcome. > > Regards, > > Philipp > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=mLPyKeOa1gNDrORvEXBgMw&m=OfAPrgGhpUsG419cks6FynnKaWgb84-BkzQXkaxraJw&s=HFKr-6vELN-qgWlEQsLH74gFoLShJnfLN89xNRNGocQ&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Mon Mar 16 22:10:11 2020 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Mon, 16 Mar 2020 22:10:11 +0000 Subject: [gpfsug-discuss] SSUG Events 2020 update In-Reply-To: References: Message-ID: <7C0DB0DD-EEE5-412D-864E-E1048E46E71E@bham.ac.uk> Unfortunately, due to the escalating COVID-19 situation, this year we?re having to cancel the London user group meeting. We?re aware that IBM has cancelled a number of events through to the end of Q2, and whilst we?re independent of IBM, we rely on them to support the group with speakers and so we?ve taken the decision to cancel. Having spoken briefly with Ulf, Kristy and Bob, we?re hoping to bring a digital event of some sort though we need to work out the logistics of doing this and how we make sure it works for the community. We all recognise that the in-person meetings are better, but this year it looks like we?re going to struggle to run one for the foreseeable future! Thanks go to the sponsors who?d signed up to support us for this year?s UK event ? OCF, Lenovo, StarFish and NVIDIA. Simon UK Group Chair From: on behalf of "chair at spectrumscale.org" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Thursday, 5 March 2020 at 22:35 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] SSUG Events 2020 update Registration is open for the May UK meeting. We plan to review 3 weeks before the event in w/c 20th April: * current situation * number of registrations * any travel restrictions in place for speakers/IBM/attendees https://www.spectrumscaleug.org/event/worldwide-uk-2020-spectrum-scale-user-group/ There is also a new user day open for registration at: https://www.spectrumscaleug.org/event/worldwide-uk-spectrum-scale-2020-new-user-day/ Simon From: on behalf of "chair at spectrumscale.org" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Friday, 28 February 2020 at 08:55 To: "gpfsug-discuss at spectrumscale.org" Subject: [gpfsug-discuss] SSUG Events 2020 update Hi All, I thought it might be giving a little bit of an update on where we are with events this year. * As you may know, SCAsia was cancelled in its entirety due to Covid-19 in Singapore and so there was no SSUG meeting. * In the US, we struggled to find a venue to host the spring meeting and now time is a little short to arrange something for the end of March planned date. * The IBM Spectrum Scale Strategy Days in Germany in March are currently still planned to happen next week. * For the UK meeting (May), we haven?t yet opened registration but are planning to do so next week. We currently believe that as an event with 120-130 attendees, this is probably very low risk, but we?ll keep the current government advice under review as we approach the date. I would suggest that if you are planning to travel internationally to the UK event that you delay booking flights/book refundable transport and ensure you have adequate insurance in place in the event we have to cancel the event. * For ISC in June, we currently don?t have a date, nor any firm plans to run an event this year. Simon Thompson UK group chair -------------- next part -------------- An HTML attachment was scrubbed... URL: From sandeep.patil at in.ibm.com Tue Mar 17 08:07:58 2020 From: sandeep.patil at in.ibm.com (Sandeep Ramesh) Date: Tue, 17 Mar 2020 08:07:58 +0000 Subject: [gpfsug-discuss] Latest Technical Blogs/Papers on IBM Spectrum Scale (Q3 2019 - Q1 2020) In-Reply-To: References: Message-ID: Dear User Group Members, In continuation to this email thread, here are list of development blogs/Redpaper in the past 2 quarters . We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to this list. Redpaper HIPAA Compliance for Healthcare Workloads on IBM Spectrum Scale http://www.redbooks.ibm.com/abstracts/redp5591.html?Open IBM Spectrum Scale CSI Driver For Container Persistent Storage http://www.redbooks.ibm.com/redpieces/abstracts/redp5589.html?Open Cyber Resiliency Solution for IBM Spectrum Scale , Blueprint http://www.redbooks.ibm.com/abstracts/redp5559.html?Open Enhanced Cyber Security with IBM Spectrum Scale and IBM QRadar http://www.redbooks.ibm.com/abstracts/redp5560.html?Open Monitoring and Managing the IBM Elastic Storage Server Using the GUI http://www.redbooks.ibm.com/abstracts/redp5471.html?Open IBM Hybrid Solution for Scalable Data Solutions using IBM Spectrum Scale http://www.redbooks.ibm.com/abstracts/redp5549.html?Open IBM Spectrum Discover: Metadata Management for Deep Insight of Unstructured Storage http://www.redbooks.ibm.com/abstracts/redp5550.html?Open Monitoring and Managing IBM Spectrum Scale Using the GUI http://www.redbooks.ibm.com/abstracts/redp5458.html?Open IBM Reference Architecture for High Performance Data and AI in Healthcare and Life Sciences, http://www.redbooks.ibm.com/abstracts/redp5481.html?Open Blogs: Why Storage and HIPAA Compliance for AI & Analytics Workloads for Healthcare https://developer.ibm.com/storage/2020/03/17/why-storage-and-hipaa-compliance-for-ai-analytics-workloads-for-healthcare/ Innovation via Integration ? Proactively Securing Your Unstructured Data from Cyber Threats & Attacks --> This was done based on your inputs (as a part of Security Survey) last year on need for Spectrum Scale integrayion with IDS a https://developer.ibm.com/storage/2020/02/24/innovation-via-integration-proactively-securing-your-unstructured-data-from-cyber-threats-attacks/ IBM Spectrum Scale CES HDFS Transparency support https://developer.ibm.com/storage/2020/02/03/ces-hdfs-transparency-support/ How to set up a remote cluster with IBM Spectrum Scale ? steps, limitations and troubleshooting https://developer.ibm.com/storage/2020/01/27/how-to-set-up-a-remote-cluster-with-ibm-spectrum-scale-steps-limitations-and-troubleshooting/ How to use IBM Spectrum Scale with CSI Operator 1.0 on Openshift 4.2 ? sample usage scenario with Tensorflow deployment https://developer.ibm.com/storage/2020/01/20/how-to-use-ibm-spectrum-scale-with-csi-operator-1-0-on-openshift-4-2-sample-usage-scenario-with-tensorflow-deployment/ Achieving WORM like functionality from NFS/SMB clients for data on Spectrum Scale https://developer.ibm.com/storage/2020/01/10/achieving-worm-like-functionality-from-nfs-smb-clients-for-data-on-spectrum-scale/ IBM Spectrum Scale CSI driver video blogs, https://developer.ibm.com/storage/2019/12/26/ibm-spectrum-scale-csi-driver-video-blogs/ IBM Spectrum Scale CSI Driver v1.0.0 released https://developer.ibm.com/storage/2019/12/10/ibm-spectrum-scale-csi-driver-v1-0-0-released/ Now configure IBM? Spectrum Scale with Overlapping UNIXMAP ranges https://developer.ibm.com/storage/2019/11/12/now-configure-ibm-spectrum-scale-with-overlapping-unixmap-ranges/ ?mmadquery?, a Powerful tool helps check AD settings from Spectrum Scale https://developer.ibm.com/storage/2019/11/11/mmadquery-a-powerful-tool-helps-check-ad-settings-from-spectrum-scale/ Spectrum Scale Data Security Modes, https://developer.ibm.com/storage/2019/10/31/spectrum-scale-data-security-modes/ IBM Spectrum Scale for Linux on IBM Z ? What?s new in IBM Spectrum Scale 5.0.4 ? https://developer.ibm.com/storage/2019/10/25/ibm-spectrum-scale-for-linux-on-ibm-z-whats-new-in-ibm-spectrum-scale-5-0-4/ IBM Spectrum Scale installation toolkit ? enhancements over releases ? 5.0.4.0 https://developer.ibm.com/storage/2019/10/18/ibm-spectrum-scale-installation-toolkit-enhancements-over-releases-5-0-4-0/ IBM Spectrum Scale CSI driver beta on GitHub, https://developer.ibm.com/storage/2019/09/26/ibm-spectrum-scale-csi-driver-on-github/ Help Article: Care to be taken when configuring AD with RFC2307 https://developer.ibm.com/storage/2019/09/18/help-article-care-to-be-taken-when-configuring-ad-with-rfc2307/ IBM Spectrum Scale Erasure Code Edition (ECE): Installation Demonstration https://developer.ibm.com/storage/2019/09/10/ibm-spectrum-scale-erasure-code-edition-ece-installation-demonstration/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 09/03/2019 10:58 AM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q2 2019) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q2 2019). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. Redpaper : IBM Power Systems Enterprise AI Solutions (W/ SPECTRUM SCALE) http://www.redbooks.ibm.com/redpieces/abstracts/redp5556.html?Open IBM Spectrum Scale Erasure Code Edition (ECE): Installation Demonstration https://www.youtube.com/watch?v=6If50EvgP-U Blogs: Using IBM Spectrum Scale as platform storage for running containerized Hadoop/Spark workloads https://developer.ibm.com/storage/2019/08/27/using-ibm-spectrum-scale-as-platform-storage-for-running-containerized-hadoop-spark-workloads/ Useful Tools for Spectrum Scale CES NFS https://developer.ibm.com/storage/2019/07/22/useful-tools-for-spectrum-scale-ces-nfs/ How to ensure NFS uses strong encryption algorithms for secure data in motion ? https://developer.ibm.com/storage/2019/07/19/how-to-ensure-nfs-uses-strong-encryption-algorithms-for-secure-data-in-motion/ Introducing IBM Spectrum Scale Erasure Code Edition https://developer.ibm.com/storage/2019/07/07/introducing-ibm-spectrum-scale-erasure-code-edition/ Spectrum Scale: Which Filesystem Encryption Algo to Consider ? https://developer.ibm.com/storage/2019/07/01/spectrum-scale-which-filesystem-encryption-algo-to-consider/ IBM Spectrum Scale HDFS Transparency Apache Hadoop 3.1.x Support https://developer.ibm.com/storage/2019/06/24/ibm-spectrum-scale-hdfs-transparency-apache-hadoop-3-0-x-support/ Enhanced features in Elastic Storage Server (ESS) 5.3.4 https://developer.ibm.com/storage/2019/06/19/enhanced-features-in-elastic-storage-server-ess-5-3-4/ Upgrading IBM Spectrum Scale Erasure Code Edition using installation toolkit https://developer.ibm.com/storage/2019/06/09/upgrading-ibm-spectrum-scale-erasure-code-edition-using-installation-toolkit/ Upgrading IBM Spectrum Scale sync replication / stretch cluster setup in PureApp https://developer.ibm.com/storage/2019/06/06/upgrading-ibm-spectrum-scale-sync-replication-stretch-cluster-setup/ GPFS config remote access with multiple network definitions https://developer.ibm.com/storage/2019/05/30/gpfs-config-remote-access-with-multiple-network-definitions/ IBM Spectrum Scale Erasure Code Edition Fault Tolerance https://developer.ibm.com/storage/2019/05/30/ibm-spectrum-scale-erasure-code-edition-fault-tolerance/ IBM Spectrum Scale for Linux on IBM Z ? What?s new in IBM Spectrum Scale 5.0.3 ? https://developer.ibm.com/storage/2019/05/02/ibm-spectrum-scale-for-linux-on-ibm-z-whats-new-in-ibm-spectrum-scale-5-0-3/ Understanding and Solving WBC_ERR_DOMAIN_NOT_FOUND error with Spectrum Scale https://crk10.wordpress.com/2019/07/21/solving-the-wbc-err-domain-not-found-nt-status-none-mapped-glitch-in-ibm-spectrum-scale/ Understanding and Solving NT_STATUS_INVALID_SID issue for SMB access with Spectrum Scale https://crk10.wordpress.com/2019/07/24/solving-nt_status_invalid_sid-for-smb-share-access-in-ibm-spectrum-scale/ mmadquery primer (apparatus to query Active Directory from IBM Spectrum Scale) https://crk10.wordpress.com/2019/07/27/mmadquery-primer-apparatus-to-query-active-directory-from-ibm-spectrum-scale/ How to configure RHEL host as Active Directory Client using SSSD https://crk10.wordpress.com/2019/07/28/configure-rhel-machine-as-active-directory-client-using-sssd/ How to configure RHEL host as LDAP client using nslcd https://crk10.wordpress.com/2019/07/28/configure-rhel-machine-as-ldap-client-using-nslcd/ Solving NFSv4 AUTH_SYS nobody ownership issue https://crk10.wordpress.com/2019/07/29/nfsv4-auth_sys-nobody-ownership-and-idmapd/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ Consolidation list of all blogs and collaterals. https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Blogs%2C%20White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 04/29/2019 12:12 PM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q1 2019) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q1 2019). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. Spectrum Scale 5.0.3 https://developer.ibm.com/storage/2019/04/24/spectrum-scale-5-0-3/ IBM Spectrum Scale HDFS Transparency Ranger Support https://developer.ibm.com/storage/2019/04/01/ibm-spectrum-scale-hdfs-transparency-ranger-support/ Integration of IBM Aspera Sync with IBM Spectrum Scale: Protecting and Sharing Files Globally, http://www.redbooks.ibm.com/abstracts/redp5527.html?Open Spectrum Scale user group in Singapore, 2019 https://developer.ibm.com/storage/2019/03/14/spectrum-scale-user-group-in-singapore-2019/ 7 traits to use Spectrum Scale to run container workload https://developer.ibm.com/storage/2019/02/26/7-traits-to-use-spectrum-scale-to-run-container-workload/ Health Monitoring of IBM Spectrum Scale Cluster via External Monitoring Framework https://developer.ibm.com/storage/2019/01/22/health-monitoring-of-ibm-spectrum-scale-cluster-via-external-monitoring-framework/ Migrating data from native HDFS to IBM Spectrum Scale based shared storage https://developer.ibm.com/storage/2019/01/18/migrating-data-from-native-hdfs-to-ibm-spectrum-scale-based-shared-storage/ Bulk File Creation useful for Test on Filesystems https://developer.ibm.com/storage/2019/01/16/bulk-file-creation-useful-for-test-on-filesystems/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Blogs%2C%20White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 01/14/2019 06:24 PM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q4 2018) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q4 2018). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. Redpaper: IBM Spectrum Scale and IBM StoredIQ: Identifying and securing your business data to support regulatory requirements http://www.redbooks.ibm.com/abstracts/redp5525.html?Open IBM Spectrum Scale Memory Usage https://www.slideshare.net/tomerperry/ibm-spectrum-scale-memory-usage?qid=50a1dfda-3102-484f-b9d0-14b69fc4800b&v=&b=&from_search=2 Spectrum Scale and Containers https://developer.ibm.com/storage/2018/12/20/spectrum-scale-and-containers/ IBM Elastic Storage Server Performance Graphical Visualization with Grafana https://developer.ibm.com/storage/2018/12/18/ibm-elastic-storage-server-performance-graphical-visualization-with-grafana/ Hadoop Performance for disaggregated compute and storage configurations based on IBM Spectrum Scale Storage https://developer.ibm.com/storage/2018/12/13/hadoop-performance-for-disaggregated-compute-and-storage-configurations-based-on-ibm-spectrum-scale-storage/ EMS HA in ESS LE (Little Endian) environment https://developer.ibm.com/storage/2018/12/07/ems-ha-in-ess-le-little-endian-environment/ What?s new in ESS 5.3.2 https://developer.ibm.com/storage/2018/12/04/whats-new-in-ess-5-3-2/ Administer your Spectrum Scale cluster easily https://developer.ibm.com/storage/2018/11/13/administer-your-spectrum-scale-cluster-easily/ Disaster Recovery using Spectrum Scale?s Active File Management https://developer.ibm.com/storage/2018/11/13/disaster-recovery-using-spectrum-scales-active-file-management/ Recovery Group Failover Procedure of IBM Elastic Storage Server (ESS) https://developer.ibm.com/storage/2018/10/08/recovery-group-failover-procedure-ibm-elastic-storage-server-ess/ Whats new in IBM Elastic Storage Server (ESS) Version 5.3.1 and 5.3.1.1 https://developer.ibm.com/storage/2018/10/04/whats-new-ibm-elastic-storage-server-ess-version-5-3-1-5-3-1-1/ For more : Search /browse here: https://developer.ibm.com/storage/blog User Group Presentations: https://www.spectrumscale.org/presentations/ Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Blogs%2C%20White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 10/03/2018 08:48 PM Subject: Latest Technical Blogs on IBM Spectrum Scale (Q3 2018) Dear User Group Members, In continuation, here are list of development blogs in the this quarter (Q3 2018). We now have over 100+ developer blogs on Spectrum Scale/ESS. As discussed in User Groups, passing it along to the emailing list. How NFS exports became more dynamic with Spectrum Scale 5.0.2 https://developer.ibm.com/storage/2018/10/02/nfs-exports-became-dynamic-spectrum-scale-5-0-2/ HPC storage on AWS (IBM Spectrum Scale) https://developer.ibm.com/storage/2018/10/02/hpc-storage-aws-ibm-spectrum-scale/ Upgrade with Excluding the node(s) using Install-toolkit https://developer.ibm.com/storage/2018/09/30/upgrade-excluding-nodes-using-install-toolkit/ Offline upgrade using Install-toolkit https://developer.ibm.com/storage/2018/09/30/offline-upgrade-using-install-toolkit/ IBM Spectrum Scale for Linux on IBM Z ? What?s new in IBM Spectrum Scale 5.0.2 ? https://developer.ibm.com/storage/2018/09/21/ibm-spectrum-scale-for-linux-on-ibm-z-whats-new-in-ibm-spectrum-scale-5-0-2/ What?s New in IBM Spectrum Scale 5.0.2 ? https://developer.ibm.com/storage/2018/09/15/whats-new-ibm-spectrum-scale-5-0-2/ Starting IBM Spectrum Scale 5.0.2 release, the installation toolkit supports upgrade rerun if fresh upgrade fails. https://developer.ibm.com/storage/2018/09/15/starting-ibm-spectrum-scale-5-0-2-release-installation-toolkit-supports-upgrade-rerun-fresh-upgrade-fails/ IBM Spectrum Scale installation toolkit ? enhancements over releases ? 5.0.2.0 https://developer.ibm.com/storage/2018/09/15/ibm-spectrum-scale-installation-toolkit-enhancements-releases-5-0-2-0/ Announcing HDP 3.0 support with IBM Spectrum Scale https://developer.ibm.com/storage/2018/08/31/announcing-hdp-3-0-support-ibm-spectrum-scale/ IBM Spectrum Scale Tuning Overview for Hadoop Workload https://developer.ibm.com/storage/2018/08/20/ibm-spectrum-scale-tuning-overview-hadoop-workload/ Making the Most of Multicloud Storage https://developer.ibm.com/storage/2018/08/13/making-multicloud-storage/ Disaster Recovery for Transparent Cloud Tiering using SOBAR https://developer.ibm.com/storage/2018/08/13/disaster-recovery-transparent-cloud-tiering-using-sobar/ Your Optimal Choice of AI Storage for Today and Tomorrow https://developer.ibm.com/storage/2018/08/10/spectrum-scale-ai-workloads/ Analyze IBM Spectrum Scale File Access Audit with ELK Stack https://developer.ibm.com/storage/2018/07/30/analyze-ibm-spectrum-scale-file-access-audit-elk-stack/ Mellanox SX1710 40G switch MLAG configuration for IBM ESS https://developer.ibm.com/storage/2018/07/12/mellanox-sx1710-40g-switcher-mlag-configuration/ Protocol Problem Determination Guide for IBM Spectrum Scale? ? SMB and NFS Access issues https://developer.ibm.com/storage/2018/07/10/protocol-problem-determination-guide-ibm-spectrum-scale-smb-nfs-access-issues/ Access Control in IBM Spectrum Scale Object https://developer.ibm.com/storage/2018/07/06/access-control-ibm-spectrum-scale-object/ IBM Spectrum Scale HDFS Transparency Docker support https://developer.ibm.com/storage/2018/07/06/ibm-spectrum-scale-hdfs-transparency-docker-support/ Protocol Problem Determination Guide for IBM Spectrum Scale? ? Log Collection https://developer.ibm.com/storage/2018/07/04/protocol-problem-determination-guide-ibm-spectrum-scale-log-collection/ Redpapers IBM Spectrum Scale Immutability Introduction, Configuration Guidance, and Use Cases http://www.redbooks.ibm.com/abstracts/redp5507.html?Open Certifications Assessment of the immutability function of IBM Spectrum Scale Version 5.0 in accordance to US SEC17a-4f, EU GDPR Article 21 Section 1, German and Swiss laws and regulations in collaboration with KPMG. Certificate: http://www.kpmg.de/bescheinigungen/RequestReport.aspx?DE968667B47544FF83F6CCDCF37E5FB5 Full assessment report: http://www.kpmg.de/bescheinigungen/RequestReport.aspx?B290411BE1224F5A9B4D24663BCD3C5D For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 07/03/2018 12:13 AM Subject: Re: Latest Technical Blogs on Spectrum Scale (Q2 2018) Dear User Group Members, In continuation , here are list of development blogs in the this quarter (Q2 2018). We now have over 100+ developer blogs. As discussed in User Groups, passing it along: IBM Spectrum Scale 5.0.1 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2018/06/15/6494/ IBM Spectrum Scale ILM Policies https://developer.ibm.com/storage/2018/06/02/ibm-spectrum-scale-ilm-policies/ IBM Spectrum Scale 5.0.1 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2018/06/15/6494/ Management GUI enhancements in IBM Spectrum Scale release 5.0.1 https://developer.ibm.com/storage/2018/05/18/management-gui-enhancements-in-ibm-spectrum-scale-release-5-0-1/ Managing IBM Spectrum Scale services through GUI https://developer.ibm.com/storage/2018/05/18/managing-ibm-spectrum-scale-services-through-gui/ Use AWS CLI with IBM Spectrum Scale? object storage https://developer.ibm.com/storage/2018/05/16/use-awscli-with-ibm-spectrum-scale-object-storage/ Hadoop Storage Tiering with IBM Spectrum Scale https://developer.ibm.com/storage/2018/05/09/hadoop-storage-tiering-ibm-spectrum-scale/ How many Files on my Filesystem? https://developer.ibm.com/storage/2018/05/07/many-files-filesystem/ Recording Spectrum Scale Object Stats for Potential Billing like Purpose using Elasticsearch https://developer.ibm.com/storage/2018/05/04/spectrum-scale-object-stats-for-billing-using-elasticsearch/ New features in IBM Elastic Storage Server (ESS) Version 5.3 https://developer.ibm.com/storage/2018/04/09/new-features-ibm-elastic-storage-server-ess-version-5-3/ Using IBM Spectrum Scale for storage in IBM Cloud Private (Missed to send earlier) https://medium.com/ibm-cloud/ibm-spectrum-scale-with-ibm-cloud-private-8bf801796f19 Redpapers Hortonworks Data Platform with IBM Spectrum Scale: Reference Guide for Building an Integrated Solution http://www.redbooks.ibm.com/redpieces/abstracts/redp5448.html, Enabling Hybrid Cloud Storage for IBM Spectrum Scale Using Transparent Cloud Tiering http://www.redbooks.ibm.com/abstracts/redp5411.html?Open SAP HANA and ESS: A Winning Combination (Update) http://www.redbooks.ibm.com/abstracts/redp5436.html?Open Others IBM Spectrum Scale Software Version Recommendation Preventive Service Planning (Updated) http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009703, IDC Infobrief: A Modular Approach to Genomics Infrastructure at Scale in HCLS https://www.ibm.com/common/ssi/cgi-bin/ssialias?htmlfid=37016937USEN& For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Date: 03/27/2018 05:23 PM Subject: Re: Latest Technical Blogs on Spectrum Scale Dear User Group Members, In continuation , here are list of development blogs in the this quarter (Q1 2018). As discussed in User Groups, passing it along: GDPR Compliance and Unstructured Data Storage https://developer.ibm.com/storage/2018/03/27/gdpr-compliance-unstructure-data-storage/ IBM Spectrum Scale for Linux on IBM Z ? Release 5.0 features and highlights https://developer.ibm.com/storage/2018/03/09/ibm-spectrum-scale-linux-ibm-z-release-5-0-features-highlights/ Management GUI enhancements in IBM Spectrum Scale release 5.0.0 https://developer.ibm.com/storage/2018/01/18/gui-enhancements-in-spectrum-scale-release-5-0-0/ IBM Spectrum Scale 5.0.0 ? What?s new in NFS? https://developer.ibm.com/storage/2018/01/18/ibm-spectrum-scale-5-0-0-whats-new-nfs/ Benefits and implementation of Spectrum Scale sudo wrappers https://developer.ibm.com/storage/2018/01/15/benefits-implementation-spectrum-scale-sudo-wrappers/ IBM Spectrum Scale: Big Data and Analytics Solution Brief https://developer.ibm.com/storage/2018/01/15/ibm-spectrum-scale-big-data-analytics-solution-brief/ Variant Sub-blocks in Spectrum Scale 5.0 https://developer.ibm.com/storage/2018/01/11/spectrum-scale-variant-sub-blocks/ Compression support in Spectrum Scale 5.0.0 https://developer.ibm.com/storage/2018/01/11/compression-support-spectrum-scale-5-0-0/ IBM Spectrum Scale Versus Apache Hadoop HDFS https://developer.ibm.com/storage/2018/01/10/spectrumscale_vs_hdfs/ ESS Fault Tolerance https://developer.ibm.com/storage/2018/01/09/ess-fault-tolerance/ Genomic Workloads ? How To Get it Right From Infrastructure Point Of View. https://developer.ibm.com/storage/2018/01/06/genomic-workloads-get-right-infrastructure-point-view/ IBM Spectrum Scale On AWS Cloud : This video explains how to deploy IBM Spectrum Scale on AWS. This solution helps the users who require highly available access to a shared name space across multiple instances with good performance, without requiring an in-depth knowledge of IBM Spectrum Scale. Detailed Demo : https://www.youtube.com/watch?v=6j5Xj_d0bh4 Brief Demo : https://www.youtube.com/watch?v=-aMQKPW_RfY. For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Cc: Doris Conti/Poughkeepsie/IBM at IBMUS Date: 01/10/2018 12:13 PM Subject: Re: Latest Technical Blogs on Spectrum Scale Dear User Group Members, Here are list of development blogs in the last quarter. Passing it to this email group as Doris had got a feedback in the UG meetings to notify the members with the latest updates periodically. Genomic Workloads ? How To Get it Right From Infrastructure Point Of View. https://developer.ibm.com/storage/2018/01/06/genomic-workloads-get-right-infrastructure-point-view/ IBM Spectrum Scale Versus Apache Hadoop HDFS https://developer.ibm.com/storage/2018/01/10/spectrumscale_vs_hdfs/ ESS Fault Tolerance https://developer.ibm.com/storage/2018/01/09/ess-fault-tolerance/ IBM Spectrum Scale MMFSCK ? Savvy Enhancements https://developer.ibm.com/storage/2018/01/05/ibm-spectrum-scale-mmfsck-savvy-enhancements/ ESS Disk Management https://developer.ibm.com/storage/2018/01/02/ess-disk-management/ IBM Spectrum Scale Object Protocol On Ubuntu https://developer.ibm.com/storage/2018/01/01/ibm-spectrum-scale-object-protocol-ubuntu/ IBM Spectrum Scale 5.0 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2017/12/20/ibm-spectrum-scale-5-0-whats-new-object/ A Complete Guide to ? Protocol Problem Determination Guide for IBM Spectrum Scale? ? Part 1 https://developer.ibm.com/storage/2017/12/19/complete-guide-protocol-problem-determination-guide-ibm-spectrum-scale-1/ IBM Spectrum Scale installation toolkit ? enhancements over releases https://developer.ibm.com/storage/2017/12/15/ibm-spectrum-scale-installation-toolkit-enhancements-releases/ Network requirements in an Elastic Storage Server Setup https://developer.ibm.com/storage/2017/12/13/network-requirements-in-an-elastic-storage-server-setup/ Co-resident migration with Transparent cloud tierin https://developer.ibm.com/storage/2017/12/05/co-resident-migration-transparent-cloud-tierin/ IBM Spectrum Scale on Hortonworks HDP Hadoop clusters : A Complete Big Data Solution https://developer.ibm.com/storage/2017/12/05/ibm-spectrum-scale-hortonworks-hdp-hadoop-clusters-complete-big-data-solution/ Big data analytics with Spectrum Scale using remote cluster mount & multi-filesystem support https://developer.ibm.com/storage/2017/11/28/big-data-analytics-spectrum-scale-using-remote-cluster-mount-multi-filesystem-support/ IBM Spectrum Scale HDFS Transparency Short Circuit Write Support https://developer.ibm.com/storage/2017/11/28/ibm-spectrum-scale-hdfs-transparency-short-circuit-write-support/ IBM Spectrum Scale HDFS Transparency Federation Support https://developer.ibm.com/storage/2017/11/27/ibm-spectrum-scale-hdfs-transparency-federation-support/ How to configure and performance tuning different system workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-different-system-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Spark workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-spark-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning database workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-database-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Hadoop workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/24/configure-performance-tuning-hadoop-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ IBM Spectrum Scale Sharing Nothing Cluster Performance Tuning https://developer.ibm.com/storage/2017/11/24/ibm-spectrum-scale-sharing-nothing-cluster-performance-tuning/ How to Configure IBM Spectrum Scale? with NIS based Authentication. https://developer.ibm.com/storage/2017/11/21/configure-ibm-spectrum-scale-nis-based-authentication/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Cc: Doris Conti/Poughkeepsie/IBM at IBMUS Date: 11/16/2017 08:15 PM Subject: Latest Technical Blogs on Spectrum Scale Dear User Group members, Here are the Development Blogs in last 3 months on Spectrum Scale Technical Topics. Spectrum Scale Monitoring ? Know More ? https://developer.ibm.com/storage/2017/11/16/spectrum-scale-monitoring-know/ IBM Spectrum Scale 5.0 Release ? What?s coming ! https://developer.ibm.com/storage/2017/11/14/ibm-spectrum-scale-5-0-release-whats-coming/ Four Essentials things to know for managing data ACLs on IBM Spectrum Scale? from Windows https://developer.ibm.com/storage/2017/11/13/four-essentials-things-know-managing-data-acls-ibm-spectrum-scale-windows/ GSSUTILS: A new way of running SSR, Deploying or Upgrading ESS Server https://developer.ibm.com/storage/2017/11/13/gssutils/ IBM Spectrum Scale Object Authentication https://developer.ibm.com/storage/2017/11/02/spectrum-scale-object-authentication/ Video Surveillance ? Choosing the right storage https://developer.ibm.com/storage/2017/11/02/video-surveillance-choosing-right-storage/ IBM Spectrum scale object deep dive training with problem determination https://www.slideshare.net/SmitaRaut/ibm-spectrum-scale-object-deep-dive-training Spectrum Scale as preferred software defined storage for Ubuntu OpenStack https://developer.ibm.com/storage/2017/09/29/spectrum-scale-preferred-software-defined-storage-ubuntu-openstack/ IBM Elastic Storage Server 2U24 Storage ? an All-Flash offering, a performance workhorse https://developer.ibm.com/storage/2017/10/06/ess-5-2-flash-storage/ A Complete Guide to Configure LDAP-based authentication with IBM Spectrum Scale? for File Access https://developer.ibm.com/storage/2017/09/21/complete-guide-configure-ldap-based-authentication-ibm-spectrum-scale-file-access/ Deploying IBM Spectrum Scale on AWS Quick Start https://developer.ibm.com/storage/2017/09/18/deploy-ibm-spectrum-scale-on-aws-quick-start/ Monitoring Spectrum Scale Object metrics https://developer.ibm.com/storage/2017/09/14/monitoring-spectrum-scale-object-metrics/ Tier your data with ease to Spectrum Scale Private Cloud(s) using Moonwalk Universal https://developer.ibm.com/storage/2017/09/14/tier-data-ease-spectrum-scale-private-clouds-using-moonwalk-universal/ Why do I see owner as ?Nobody? for my export mounted using NFSV4 Protocol on IBM Spectrum Scale?? https://developer.ibm.com/storage/2017/09/08/see-owner-nobody-export-mounted-using-nfsv4-protocol-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory and LDAP https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-ldap/ IBM Spectrum Scale? Authentication using Active Directory and RFC2307 https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-rfc2307/ High Availability Implementation with IBM Spectrum Virtualize and IBM Spectrum Scale https://developer.ibm.com/storage/2017/08/30/high-availability-implementation-ibm-spectrum-virtualize-ibm-spectrum-scale/ 10 Frequently asked Questions on configuring Authentication using AD + AUTO ID mapping on IBM Spectrum Scale?. https://developer.ibm.com/storage/2017/08/04/10-frequently-asked-questions-configuring-authentication-using-ad-auto-id-mapping-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory https://developer.ibm.com/storage/2017/07/30/ibm-spectrum-scale-auth-using-active-directory/ Five cool things that you didn?t know Transparent Cloud Tiering on Spectrum Scale can do https://developer.ibm.com/storage/2017/07/29/five-cool-things-didnt-know-transparent-cloud-tiering-spectrum-scale-can/ IBM Spectrum Scale GUI videos https://developer.ibm.com/storage/2017/07/25/ibm-spectrum-scale-gui-videos/ IBM Spectrum Scale? Authentication ? Planning for NFS Access https://developer.ibm.com/storage/2017/07/24/ibm-spectrum-scale-planning-nfs-access/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media -------------- next part -------------- An HTML attachment was scrubbed... URL: From valleru at cbio.mskcc.org Fri Mar 20 21:18:42 2020 From: valleru at cbio.mskcc.org (Valleru, Lohit/Information Systems) Date: Fri, 20 Mar 2020 17:18:42 -0400 Subject: [gpfsug-discuss] Network switches/architecture for GPFS Message-ID: <35ADE287-4355-430B-83A0-A00C76525822@cbio.mskcc.org> Hello All, I would like to discuss or understand on which ethernet networking switches/architecture seems to work best with GPFS. We had thought about infiniband, but are not yet ready to move to infiniband because of the complexity/upgrade and debugging issues that come with it. Current hardware: We are currently using Arista 7328x 100G core switch for networking among the GPFS clusters and the compute nodes. It is heterogeneous network, with some of the servers on 10G/25G/100G with LACP and without LACP. For example: GPFS storage clusters either have 25G LACP, or 10G LACP, or a single 100G network port. Compute nodes range from 10G to 100G. Login nodes/transfer servers etc have 25G bonded. Most of the servers have Mellanox ConnectX-4 or ConnectX-5 adapters. But we also have few older Intel,Broadcom and Chelsio network cards in the clusters. Most of the transceivers that we use are Mellanox,Finisar,Intel. Issue: We had upgraded to the above switch recently, and we had seen that it is not able to handle the network traffic because of higher NSD servers bandwidth vs lower compute node bandwidth. One issue that we did see was a lot of network discards on the switch side and network congestion with slow IO performance on respective compute nodes. Once we enabled ECN - we did see that it had reduced the network congestion. We do see expels once in a while, but that is mostly related to the network errors or the host not responding. We observed that bonding/LACP does make expels much more trickier, so we have decided to go with no LACP until GPFS code gets better at handling LACP - which I think they are working on. We have heard that our current switch is a shallow buffer switch, and we would need a higher/deep buffer Arista switch to perform better with no congestion/lesser latency and more throughput. On the other side, Mellanox promises to use better ASIC design and buffer architecture with spine leaf design, instead of one deep buffer core switch to get better performance than Arista. Most of the applications that run on the clusters are either genomic applications on CPUs and deep learning applications on GPUs. All of our GPFS storage cluster versions are above 5.0.2 with the compute filesystems at 16M block size on near line rotating disks, and Flash storage at 512K block size. May I know if could feedback from anyone who is using Arista or Mellanox switches on the clusters to understand the pros and cons, stability and the performance numbers of the same? Thank you, Lohit -------------- next part -------------- An HTML attachment was scrubbed... URL: From cblack at nygenome.org Sun Mar 22 00:58:42 2020 From: cblack at nygenome.org (Christopher Black) Date: Sun, 22 Mar 2020 00:58:42 +0000 Subject: [gpfsug-discuss] Network switches/architecture for GPFS In-Reply-To: <35ADE287-4355-430B-83A0-A00C76525822@cbio.mskcc.org> References: <35ADE287-4355-430B-83A0-A00C76525822@cbio.mskcc.org> Message-ID: <74B39595-5718-4B88-B7B8-513942E1983D@nygenome.org> We?ve had good luck moving from older Mellanox 1710 ethernet switches to newer Arista ethernet switches. Our core is a pair of Arista 7508s primarily with 100G cards. Leaf switches are Arista 7280QR for racks with 40Gb-connected servers and 7280SR for racks w/ 10Gb-connected servers. Uplinks from leaf switches to core are multiple 100G connections. Our nsd servers are connected with dual-40Gb connections, each connection a separate Mellanox ConnectX-3 card to spread load and failure across separate pcie slots. Our compute nodes are primarily connected with dual-10Gb connections on Intel x520 or x710 nics (dual-port on a single nic). We also have some Cisco UCS nodes going through Cisco FI?s, these do not perform nearly as well and we?ve had some trouble with them and high bandwidth network storage, especially with defaults. We have some data transfer nodes connected at 2x40Gb, but other than that our only 40Gb-connected nodes are nsd servers. Any server, nsd or compute, uses lacp to bond and has mtu set to 9000. We also set: BONDING_OPTS="mode=4 miimon=100 xmit_hash_policy=layer3+4" For ecn, we have sysctl net.ipv4.tcp_ecn = 2. We also run primarily genomics applications. We have no experience with more recent Mellanox switches, but the ethernet software implementation on their older switches gave us plenty of problems. I?m not the network expert at our site, but they seem to like the Arista software much more than Mellanox. We run some non-default tcp/ip and ethernet settings, primarily from fasterdata.es.net recommendations. IBM?s older wiki notes about linux sysctls sometimes does not match es.net recommendation, and in those cases we generally go w/ es.net, especially as some of the IBM docs were written for older Linux kernels. However, there are some sysctl recommendations from IBM docs that are unique to gpfs (net.core.somaxconn). Regarding non-net tuning to improve gpfs stability, we?ve found the following are also important: vm.min_free_kbytes vm.dirty_bytes vm.dirty_background_bytes It took us a long time to figure out that on systems with lots of memory, many dirty pages could be buffered before being flushed out to network, resulting in a storm of heavy traffic that could cause problems for gpfs disk lease renewals and other control traffic to get through quick enough to avoid expels. For client NIC tuning, we set txqueuelen 10000 but I?ve read that this may not be necessary on newer kernels. On older nics, or even current intel nics with older firmware, we found turning some offload optimizations OFF made things better (gro, lro, gso, lso). I hope this helps you or others running gpfs on ethernet! -Chris From: on behalf of "Valleru, Lohit/Information Systems" Reply-To: gpfsug main discussion list Date: Friday, March 20, 2020 at 5:18 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Network switches/architecture for GPFS Hello All, I would like to discuss or understand on which ethernet networking switches/architecture seems to work best with GPFS. We had thought about infiniband, but are not yet ready to move to infiniband because of the complexity/upgrade and debugging issues that come with it. Current hardware: We are currently using Arista 7328x 100G core switch for networking among the GPFS clusters and the compute nodes. It is heterogeneous network, with some of the servers on 10G/25G/100G with LACP and without LACP. For example: GPFS storage clusters either have 25G LACP, or 10G LACP, or a single 100G network port. Compute nodes range from 10G to 100G. Login nodes/transfer servers etc have 25G bonded. Most of the servers have Mellanox ConnectX-4 or ConnectX-5 adapters. But we also have few older Intel,Broadcom and Chelsio network cards in the clusters. Most of the transceivers that we use are Mellanox,Finisar,Intel. Issue: We had upgraded to the above switch recently, and we had seen that it is not able to handle the network traffic because of higher NSD servers bandwidth vs lower compute node bandwidth. One issue that we did see was a lot of network discards on the switch side and network congestion with slow IO performance on respective compute nodes. Once we enabled ECN - we did see that it had reduced the network congestion. We do see expels once in a while, but that is mostly related to the network errors or the host not responding. We observed that bonding/LACP does make expels much more trickier, so we have decided to go with no LACP until GPFS code gets better at handling LACP - which I think they are working on. We have heard that our current switch is a shallow buffer switch, and we would need a higher/deep buffer Arista switch to perform better with no congestion/lesser latency and more throughput. On the other side, Mellanox promises to use better ASIC design and buffer architecture with spine leaf design, instead of one deep buffer core switch to get better performance than Arista. Most of the applications that run on the clusters are either genomic applications on CPUs and deep learning applications on GPUs. All of our GPFS storage cluster versions are above 5.0.2 with the compute filesystems at 16M block size on near line rotating disks, and Flash storage at 512K block size. May I know if could feedback from anyone who is using Arista or Mellanox switches on the clusters to understand the pros and cons, stability and the performance numbers of the same? Thank you, Lohit ________________________________ This message is for the recipient?s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kraemerf at de.ibm.com Mon Mar 23 07:46:54 2020 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Mon, 23 Mar 2020 08:46:54 +0100 Subject: [gpfsug-discuss] Network switches/architecture for Spectrum Scale (GPFS) In-Reply-To: References: Message-ID: Hi, > May I know if could feedback from anyone who is using Arista or Mellanox switches on the > clusters to understand the pros and cons, stability and the performance numbers of the same? Most current network switches are using Merchant Silicon ASICs instead of custom ASICs as in the past. (This is even true for some Cisco devices.) "Merchant silicon is a term used to described chips, usually ASICs, that are designed and made by an entity other than the company selling the switches in which they are used. I might be tempted to say such switches use off-the-shelf ASICs, though that might imply that I could buy these chips from a retail store." https://www.datacenterknowledge.com/networks/why-merchant-silicon-taking-over-data-center-network-market All of the Merchant Silicon ASICs lines have different family members from entry to high-end features. Switch vendors will pick and choose these ASICs by price and target market for the specific switch models. You can't tell from the "outside" which switch does offer which kind of performance. You need to do more research on the specific switch and use case. Most of the time Mellanox switches does offer very attractive features and performance levels for parallel I/O workloads but there are more options in the market. The NOS (Network OS) which is running on these new networking switches is also not a fixed option. You have some choice here. Some switches can be ordered with different NOS versions. Which NOS is the best fit for Spectrum Scale or other pFS's is still a topic to be researched. Last but not least the question of which NIC card is the best fit in this job is important. SmartNIC: A network interface card (network adapter) that offloads processing tasks that the system CPU would normally handle. Using its own on-board processor, the SmartNIC may be able to perform any combination of encryption/decryption, firewall, TCP/IP and HTTP processing. SmartNICs can be ASIC, FPGA, and System-on-a-Chip (SOC) based. Naturally vendors who make just one kind of NIC seem to insist that only the type of NIC they make should qualify as a SmartNIC. Mellanox is a good choice here but there is wide range of choices and features already in the market. -frank- Appendix: 1 - A Network-centric View of Scalable Storage by Andy Bechtolsheim, Chief Development Officer and Co-Founder, Arista Networks https://youtu.be/__LomgSguSc 2 - Broadcom Ships Tomahawk 4, Industry?s Highest Bandwidth Ethernet Switch Chip at 25.6 Terabits per Second ? Broadcom https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm56990-series https://packetpushers.net/podcast/network-break-264-broadcoms-new-tomahawk-4-hits-25-6tbps-juniper-announces-sd-lan-for-ex-switches/ https://www.telecompaper.com/news/broadcom-ships-tomahawk-4-ethernet-switch-chip--1319338 Broadcom delivered the StrataXGS Tomahawk 4 switch series, demonstrating an unprecedented 25.6 Tbps of Ethernet switching performance in a single device. 3 - Mellanox Introduces Revolutionary ConnectX-6 Dx and BlueField-2 Secure Cloud SmartNICs and I/O Processing Unit Solutions ConnectX-6 Dx and BlueField-2 Provide 200 Gb/s Ethernet and InfiniBand Connectivity, Enabling Next Generation of Clouds, Secure Data Centers and Storage Platforms https://www.mellanox.com/page/press_release_item?id=2195 Frank Kraemer IBM Consulting IT Specialist / Client Technical Architect Am Weiher 24, 65451 Kelsterbach, Germany mailto:kraemerf at de.ibm.com Mobile +49171-3043699 IBM Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Mon Mar 23 18:09:16 2020 From: mweil at wustl.edu (Matt Weil) Date: Mon, 23 Mar 2020 13:09:16 -0500 Subject: [gpfsug-discuss] AFM gateway node scaling Message-ID: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Hello all, Is there any guide and or recommendation as to how to scale this. filesets per gateway node?? Is it necessary to separate NSD server and gateway roles.? Are dedicated gateway nodes licensed as clients? Thanks for any guidance. Matt From peter.chase at metoffice.gov.uk Tue Mar 24 11:08:15 2020 From: peter.chase at metoffice.gov.uk (Chase, Peter) Date: Tue, 24 Mar 2020 11:08:15 +0000 Subject: [gpfsug-discuss] tctwatcher command Message-ID: Hello, The scale 5.0.4 documentation makes reference to a tctwatcher command, but I can't find the executable in any of the 5.0.4-3 RPMs I've deployed (including gpfs.tct.server, gpfs.tct.client, gpfs.kafka and gpfs.librdkafka). I'd wondered if it needed compiling from somewhere in /usr/lpp/mmfs/samples/ or maybe /opt/ibm/MCStore/samples/ but I can't find anything referencing it there either! Does anyone know where it is? Documentation: https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/b1ladm_integratingwfolderintocloud.htm It's also in the 5.0.3 doc. Regards, Pete Chase Mainframe, Virtualization and Storage team Met Office -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Wed Mar 25 10:01:04 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 10:01:04 +0000 Subject: [gpfsug-discuss] mmbackup monitoring Message-ID: What is the best way of monitoring whether or not mmbackup has managed to complete a backup successfully? Traditionally one use a TSM monitoring solution of your choice to make sure nodes where backing up (I am assuming mmbackup is being used in conjunction with TSM here). However mmbackup does not update the backup_end column in the filespaceview table (at least in 4.2) which makes things rather more complicated. The best I can come up with is querying the events table to see if the client schedule completed, but that gives a false sense of security as the schedule completing does not mean the backup completed as far as I know. What solutions are you all using, or does mmbackup in 5.x update the filespaceview table? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From jonathan.buzzard at strath.ac.uk Wed Mar 25 10:09:12 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 10:09:12 +0000 Subject: [gpfsug-discuss] GPFS 5 and supported rhel OS In-Reply-To: References: Message-ID: <91d02fd3-2af7-5880-e1f2-aaf9b1f8040a@strath.ac.uk> On 19/02/2020 23:34, Renata Maria Dart wrote: > Hi, I understand gpfs 4.2.3 is end of support this coming September. A planning question at this stage. Do IBM intend to hold to this date or is/could there be a relaxation due to COVID-19? Basically I was planning to do the upgrade this summer, but what with working from home I am less keen to do a a 4.2.3 to 5.x upgrade while not on hand to the actual hardware. Obviously if we have to we have to, just want to know where we stand. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From carlz at us.ibm.com Wed Mar 25 12:12:38 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Wed, 25 Mar 2020 12:12:38 +0000 Subject: [gpfsug-discuss] GPFS 5 and supported rhel OS Message-ID: <70ECB57D-C012-43FB-89F1-AF5810984DAF@us.ibm.com> So far we have not revisited the EOS date for 4.2.3, but I would not rule it out entirely if the lockdown continues well into the summer. If we did, the next likely EOS date would be April 30th. Even if we do postpone the date for 4.2.3, keep two other dates in mind for planning: - RHEL 6 support is coming to an end in November. We won't support Scale with RHEL 6 once Red Hat stops supporting RHEL 6 - RHEL 7 will be supported with 5.0.5, but not "5.next", the release scheduled for the second half of 2020. So you'll need to plan to adopt RHEL 8 before upgrading to Scale "5.next" As much as possible we are going to try to stick to our release cadence of twice a year even through these difficulties, including designating 5.0.5 for Extended Updates. "Keep Calm and Scale Out". Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com Message: 2 Date: Wed, 25 Mar 2020 10:09:12 +0000 From: Jonathan Buzzard To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] GPFS 5 and supported rhel OS Message-ID: <91d02fd3-2af7-5880-e1f2-aaf9b1f8040a at strath.ac.uk> Content-Type: text/plain; charset=utf-8; format=flowed On 19/02/2020 23:34, Renata Maria Dart wrote: > Hi, I understand gpfs 4.2.3 is end of support this coming September. A planning question at this stage. Do IBM intend to hold to this date or is/could there be a relaxation due to COVID-19? Basically I was planning to do the upgrade this summer, but what with working from home I am less keen to do a a 4.2.3 to 5.x upgrade while not on hand to the actual hardware. Obviously if we have to we have to, just want to know where we stand. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG ------------------------------ From skylar2 at u.washington.edu Wed Mar 25 14:15:59 2020 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Wed, 25 Mar 2020 07:15:59 -0700 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: References: Message-ID: <20200325141558.2iwnalq3hwo3r5ha@illiuin> We execute mmbackup via a regular TSM client schedule with an incremental action, with a virtualmountpoint set to an empty, local "canary" directory. mmbackup runs as a preschedule command, and the client -domain parameter is set only to backup the canary directory. dsmc will backup the canary directory as a filespace only if mmbackup succeeds (exits with 0). We can then monitor the canary and infer the status of the associated GPFS filespace or fileset. On Wed, Mar 25, 2020 at 10:01:04AM +0000, Jonathan Buzzard wrote: > > What is the best way of monitoring whether or not mmbackup has managed to > complete a backup successfully? > > Traditionally one use a TSM monitoring solution of your choice to make sure > nodes where backing up (I am assuming mmbackup is being used in conjunction > with TSM here). > > However mmbackup does not update the backup_end column in the filespaceview > table (at least in 4.2) which makes things rather more complicated. > > The best I can come up with is querying the events table to see if the > client schedule completed, but that gives a false sense of security as the > schedule completing does not mean the backup completed as far as I know. > > What solutions are you all using, or does mmbackup in 5.x update the > filespaceview table? -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From pinto at scinet.utoronto.ca Wed Mar 25 14:38:55 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Wed, 25 Mar 2020 10:38:55 -0400 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <20200325141558.2iwnalq3hwo3r5ha@illiuin> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> Message-ID: Additionally, mmbackup creates by default a .mmbackupCfg directory on the root of the fileset where it dumps several files and directories with the progress of the backup. For instance: expiredFiles/, prepFiles/, updatedFiles/, dsminstr.log, ... You may then create a script to search these directories for logs/lists of what has happened, and generate a more detailed report of what happened during the backup. In our case I generate a daily report of how many files and how much data have been sent to the TSM server and deleted for each user, including their paths. You can do more tricks if you want. Jaime On 3/25/2020 10:15:59, Skylar Thompson wrote: > We execute mmbackup via a regular TSM client schedule with an incremental > action, with a virtualmountpoint set to an empty, local "canary" directory. > mmbackup runs as a preschedule command, and the client -domain parameter is > set only to backup the canary directory. dsmc will backup the canary > directory as a filespace only if mmbackup succeeds (exits with 0). We can > then monitor the canary and infer the status of the associated GPFS > filespace or fileset. > > On Wed, Mar 25, 2020 at 10:01:04AM +0000, Jonathan Buzzard wrote: >> >> What is the best way of monitoring whether or not mmbackup has managed to >> complete a backup successfully? >> >> Traditionally one use a TSM monitoring solution of your choice to make sure >> nodes where backing up (I am assuming mmbackup is being used in conjunction >> with TSM here). >> >> However mmbackup does not update the backup_end column in the filespaceview >> table (at least in 4.2) which makes things rather more complicated. >> >> The best I can come up with is querying the events table to see if the >> client schedule completed, but that gives a false sense of security as the >> schedule completing does not mean the backup completed as far as I know. >> >> What solutions are you all using, or does mmbackup in 5.x update the >> filespaceview table? > . . . ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From skylar2 at u.washington.edu Wed Mar 25 15:25:15 2020 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Wed, 25 Mar 2020 08:25:15 -0700 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> Message-ID: <20200325152515.rd2uyh5fckkwdx4p@illiuin> IIRC, I think you need to set 2 in the bit field of the DEBUGmmbackup environment variable. I had a long-term task to see what I could get out of that, but this just reminded me of it and current events might actually let me have time to look into it now... On Wed, Mar 25, 2020 at 10:38:55AM -0400, Jaime Pinto wrote: > Additionally, mmbackup creates by default a .mmbackupCfg directory on the root of the fileset where it dumps several files and directories with the progress of the backup. For instance: expiredFiles/, prepFiles/, updatedFiles/, dsminstr.log, ... > > You may then create a script to search these directories for logs/lists of what has happened, and generate a more detailed report of what happened during the backup. In our case I generate a daily report of how many files and how much data have been sent to the TSM server and deleted for each user, including their paths. You can do more tricks if you want. > > Jaime > > > On 3/25/2020 10:15:59, Skylar Thompson wrote: > > We execute mmbackup via a regular TSM client schedule with an incremental > > action, with a virtualmountpoint set to an empty, local "canary" directory. > > mmbackup runs as a preschedule command, and the client -domain parameter is > > set only to backup the canary directory. dsmc will backup the canary > > directory as a filespace only if mmbackup succeeds (exits with 0). We can > > then monitor the canary and infer the status of the associated GPFS > > filespace or fileset. > > > > On Wed, Mar 25, 2020 at 10:01:04AM +0000, Jonathan Buzzard wrote: > > > > > > What is the best way of monitoring whether or not mmbackup has managed to > > > complete a backup successfully? > > > > > > Traditionally one use a TSM monitoring solution of your choice to make sure > > > nodes where backing up (I am assuming mmbackup is being used in conjunction > > > with TSM here). > > > > > > However mmbackup does not update the backup_end column in the filespaceview > > > table (at least in 4.2) which makes things rather more complicated. > > > > > > The best I can come up with is querying the events table to see if the > > > client schedule completed, but that gives a false sense of security as the > > > schedule completing does not mean the backup completed as far as I know. > > > > > > What solutions are you all using, or does mmbackup in 5.x update the > > > filespaceview table? > > > > . > . > . ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto - Storage Analyst > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From heinrich.billich at id.ethz.ch Wed Mar 25 15:35:55 2020 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Wed, 25 Mar 2020 15:35:55 +0000 Subject: [gpfsug-discuss] GUI timeout when running HW_INVENTORY on little endian ESS server Message-ID: Hello, I did ask about this timeouts when the gui runs HW_INVENTORY before. Now I would like to know what the exact timeout value in the gui code is and if we can change it. I want to argue: If a xCat command takes X seconds but the GUI code timeouts after Y we know the command will fail if X > Y, hence we need to increase Y unless we can reduce X ... It's this function which raises the timeout: at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory If we can't fix the long execution time for the time being, can we raise the timeout value? I know this most likely is a Firmware issue with little endian power systems, but we won't update some more time. Thank you, Heiner debug: Running 'xcat.sh rinv '10.250.***' '*****' 'all' ' on node localhost err: com.ibm.fscc.common.exceptions.FsccException: Error executing rinv command. Exit code = 1; Command output = ; Command error = *****: [****]: Error: timeout at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory(InventoryAndStateHelper.java:92) at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.buildHardwareInventory(InventoryAndStateHelper.java:175) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.inner_run(InventoryRefreshTask.java:94) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.run(InventoryRefreshTask.java:72) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:227) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:199) at com.ibm.fscc.common.newscheduler.RefreshTaskIds.execute(RefreshTaskIds.java:482) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:80) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) ... debug: Running 'mmsysmonc event 'gui' 'xcat_nodelist_ok' ****-i' ***-i' ' on node localhost err: ***-i: Error executing rinv command. Exit code = 1; Command output = ; Command error = nas12io04b: [***]: Error: timeout ,*** -i: Error executing rinv command. Exit code = 1; Command output = ; Command error =***: [***]: Error: timeout err: com.ibm.fscc.cli.CommandException: EFSSG1150C Running specified task was unsuccessful. at com.ibm.fscc.cli.CommandException.createCommandException(CommandException.java:117) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:84) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) EFSSG1150C Running specified task was unsuccessful. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== From heinrich.billich at id.ethz.ch Wed Mar 25 15:53:04 2020 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Wed, 25 Mar 2020 15:53:04 +0000 Subject: [gpfsug-discuss] GUI timeout when running HW_INVENTORY on little endian ESS server In-Reply-To: References: Message-ID: <3C52599F-DC19-47F2-9CF2-90B905D43A1B@id.ethz.ch> Hello, Sorry, I was wrong. Looks like the timeout already happens in xCAT/rinv and the gui just reports it. What to some respect is good - now this is a purely xCAT/hardware issue. The GUI isn't involved any more. Kind regards Heiner /var/log/xcat/command.log: ==================================================== [Date] 2020-03-25 15:03:46 [ClientType] cli [Request] rinv ***** all [Response] ***: Error: timeout [NumberNodes] 1 [ElapsedTime] 97.085 s ==================================================== GUI: HW_INVENTORY * 2020-03-25 15:03:26 681436ms failed CmdRunTask.doExecute nas12io04b-i: Error executing rinv command. Exit code = 1; Command output = ; Command error =***: [**]: Error: timeout ?On 25.03.20, 16:35, "Billich Heinrich Rainer (ID SD)" wrote: Hello, I did ask about this timeouts when the gui runs HW_INVENTORY before. Now I would like to know what the exact timeout value in the gui code is and if we can change it. I want to argue: If a xCat command takes X seconds but the GUI code timeouts after Y we know the command will fail if X > Y, hence we need to increase Y unless we can reduce X ... It's this function which raises the timeout: at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory If we can't fix the long execution time for the time being, can we raise the timeout value? I know this most likely is a Firmware issue with little endian power systems, but we won't update some more time. Thank you, Heiner debug: Running 'xcat.sh rinv '10.250.***' '*****' 'all' ' on node localhost err: com.ibm.fscc.common.exceptions.FsccException: Error executing rinv command. Exit code = 1; Command output = ; Command error = *****: [****]: Error: timeout at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.runRemoteInventory(InventoryAndStateHelper.java:92) at com.ibm.fscc.ras.xcat.InventoryAndStateHelper.buildHardwareInventory(InventoryAndStateHelper.java:175) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.inner_run(InventoryRefreshTask.java:94) at com.ibm.fscc.ras.xcat.InventoryRefreshTask.run(InventoryRefreshTask.java:72) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:227) at com.ibm.fscc.common.newscheduler.RefreshTaskExecutor.executeRefreshTask(RefreshTaskExecutor.java:199) at com.ibm.fscc.common.newscheduler.RefreshTaskIds.execute(RefreshTaskIds.java:482) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:80) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) ... debug: Running 'mmsysmonc event 'gui' 'xcat_nodelist_ok' ****-i' ***-i' ' on node localhost err: ***-i: Error executing rinv command. Exit code = 1; Command output = ; Command error = nas12io04b: [***]: Error: timeout ,*** -i: Error executing rinv command. Exit code = 1; Command output = ; Command error =***: [***]: Error: timeout err: com.ibm.fscc.cli.CommandException: EFSSG1150C Running specified task was unsuccessful. at com.ibm.fscc.cli.CommandException.createCommandException(CommandException.java:117) at com.ibm.fscc.newcli.commands.task.CmdRunTask.doExecute(CmdRunTask.java:84) at com.ibm.fscc.newcli.internal.AbstractCliCommand.execute(AbstractCliCommand.java:156) at com.ibm.fscc.cli.CliProtocol.processNewStyleCommand(CliProtocol.java:470) at com.ibm.fscc.cli.CliProtocol.processRequest(CliProtocol.java:456) at com.ibm.fscc.cli.CliServer$CliClientServer.run(CliServer.java:97) EFSSG1150C Running specified task was unsuccessful. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== From jonathan.buzzard at strath.ac.uk Wed Mar 25 16:27:27 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 16:27:27 +0000 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <20200325141558.2iwnalq3hwo3r5ha@illiuin> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> Message-ID: <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> On 25/03/2020 14:15, Skylar Thompson wrote: > We execute mmbackup via a regular TSM client schedule with an incremental > action, with a virtualmountpoint set to an empty, local "canary" directory. > mmbackup runs as a preschedule command, and the client -domain parameter is > set only to backup the canary directory. dsmc will backup the canary > directory as a filespace only if mmbackup succeeds (exits with 0). We can > then monitor the canary and infer the status of the associated GPFS > filespace or fileset. > I prefer this approach I think than grovelling around in log files that could easily break on an update. Though there is a better approach which in my view IBM should be using already in mmbackup. It came to me this afternoon that one could use the TSM API for this. After a bit of Googling I find there is an API call dsmUpdateFS, which allows you to update the filespace information on the TSM server. Fields that you can update include DSM_FSUPD_OCCUPANCY DSM_FSUPD_CAPACITY DSM_FSUPD_BACKSTARTDATE DSM_FSUPD_BACKCOMPLETEDATE Information on the API call here https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/api/r_cmd_dsmupdatefs.html How do we submit this as a feature request again? That said in my view it's a bug in mmbackup. The latest in a very long line stretching back well over a decade that make mmbackup less than production ready rather than a feature request :-) I feel a breakout of a text editor and some C code coming on in the meantime. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From skylar2 at u.washington.edu Wed Mar 25 16:32:24 2020 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Wed, 25 Mar 2020 09:32:24 -0700 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> Message-ID: <20200325163224.jbibka2nukfhmc7u@illiuin> On Wed, Mar 25, 2020 at 04:27:27PM +0000, Jonathan Buzzard wrote: > On 25/03/2020 14:15, Skylar Thompson wrote: > > We execute mmbackup via a regular TSM client schedule with an incremental > > action, with a virtualmountpoint set to an empty, local "canary" directory. > > mmbackup runs as a preschedule command, and the client -domain parameter is > > set only to backup the canary directory. dsmc will backup the canary > > directory as a filespace only if mmbackup succeeds (exits with 0). We can > > then monitor the canary and infer the status of the associated GPFS > > filespace or fileset. > > > > I prefer this approach I think than grovelling around in log files that > could easily break on an update. Though there is a better approach which in > my view IBM should be using already in mmbackup. > > It came to me this afternoon that one could use the TSM API for this. After > a bit of Googling I find there is an API call dsmUpdateFS, which allows you > to update the filespace information on the TSM server. > > Fields that you can update include > > DSM_FSUPD_OCCUPANCY > DSM_FSUPD_CAPACITY > DSM_FSUPD_BACKSTARTDATE > DSM_FSUPD_BACKCOMPLETEDATE > > Information on the API call here > > https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/api/r_cmd_dsmupdatefs.html > > How do we submit this as a feature request again? That said in my view it's > a bug in mmbackup. The latest in a very long line stretching back well over > a decade that make mmbackup less than production ready rather than a feature > request :-) > > I feel a breakout of a text editor and some C code coming on in the > meantime. I actually tried using the API years ago to try to do some custom queries, and ran into the problem that custom API clients can only see data from custom API clients; they can't see data from the standard BA client. I contacted IBM about this, and they said it was a safety feature to prevent a rogue/poorly-written client from trashing regular backup/archive data, which makes some sense. Unfortunately, it does mean that IBM would have to be the source of the fix. -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From jonathan.buzzard at strath.ac.uk Wed Mar 25 16:45:54 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 25 Mar 2020 16:45:54 +0000 Subject: [gpfsug-discuss] mmbackup monitoring In-Reply-To: <20200325163224.jbibka2nukfhmc7u@illiuin> References: <20200325141558.2iwnalq3hwo3r5ha@illiuin> <53076ba7-5452-347b-21ae-dc10e17ede0e@strath.ac.uk> <20200325163224.jbibka2nukfhmc7u@illiuin> Message-ID: On 25/03/2020 16:32, Skylar Thompson wrote: > On Wed, Mar 25, 2020 at 04:27:27PM +0000, Jonathan Buzzard wrote: >> On 25/03/2020 14:15, Skylar Thompson wrote: >>> We execute mmbackup via a regular TSM client schedule with an incremental >>> action, with a virtualmountpoint set to an empty, local "canary" directory. >>> mmbackup runs as a preschedule command, and the client -domain parameter is >>> set only to backup the canary directory. dsmc will backup the canary >>> directory as a filespace only if mmbackup succeeds (exits with 0). We can >>> then monitor the canary and infer the status of the associated GPFS >>> filespace or fileset. >>> >> >> I prefer this approach I think than grovelling around in log files that >> could easily break on an update. Though there is a better approach which in >> my view IBM should be using already in mmbackup. >> >> It came to me this afternoon that one could use the TSM API for this. After >> a bit of Googling I find there is an API call dsmUpdateFS, which allows you >> to update the filespace information on the TSM server. >> >> Fields that you can update include >> >> DSM_FSUPD_OCCUPANCY >> DSM_FSUPD_CAPACITY >> DSM_FSUPD_BACKSTARTDATE >> DSM_FSUPD_BACKCOMPLETEDATE >> >> Information on the API call here >> >> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fsupport%2Fknowledgecenter%2FSSEQVQ_8.1.9%2Fapi%2Fr_cmd_dsmupdatefs.html&data=02%7C01%7Cjonathan.buzzard%40strath.ac.uk%7C8c7605146223442e8a2708d7d0dab99d%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C637207510145541248&sdata=QV9U0WxL5BTtR5%2Fasv1X202d9PqnLXZNg5bzX8KpUOo%3D&reserved=0 >> >> How do we submit this as a feature request again? That said in my view it's >> a bug in mmbackup. The latest in a very long line stretching back well over >> a decade that make mmbackup less than production ready rather than a feature >> request :-) >> >> I feel a breakout of a text editor and some C code coming on in the >> meantime. > > I actually tried using the API years ago to try to do some custom queries, > and ran into the problem that custom API clients can only see data from > custom API clients; they can't see data from the standard BA client. I > contacted IBM about this, and they said it was a safety feature to prevent > a rogue/poorly-written client from trashing regular backup/archive data, > which makes some sense. Unfortunately, it does mean that IBM would have to > be the source of the fix. > Grrr, I had forgotten that. Well then IBM need to fix this. Bug mmbackup does not update the occupancy, capacity, backup start date and backup end date when doing a backup. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From vpuvvada at in.ibm.com Wed Mar 25 16:50:41 2020 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Wed, 25 Mar 2020 22:20:41 +0530 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: Matt, It is recommended to have dedicated AFM gateway nodes. Memory and CPU requirements for AFM gateway node depends on the number of filesets handled by the node and the inode usage of those filesets. Since AFM keeps track of changes in the memory, any network disturbance can cause the memory utilization to go high and which eventually leads to in-memory queue to be dropped. After the queue is dropped, AFM runs recovery to recover the lost operations which is expensive as it involves creating the snapshot, running policy scan, doing readdir from home/secondary and build the list of lost operations. When the gateway node goes down, all the filesets handled by that node distributed to the remaining active gateway nodes. After the gateway node comes back, filesets are transferred back to the original gateway node. When designing the gateway node, make sure that it have enough memory , CPU resources for handling the incoming and outgoing data based on the bandwidth. Limit the filesets per gateway(ex. less than 20 filesets per gateway) so that number of AFM recoveries triggered will be minimal when the queues are lost. Also limit the total number of inodes handled by the gateway node across all the filesets (ex. less than 400 million inodes per gateway). AFM gateway nodes are licensed as server nodes. ~Venkat (vpuvvada at in.ibm.com) From: Matt Weil To: gpfsug-discuss at spectrumscale.org Date: 03/23/2020 11:39 PM Subject: [EXTERNAL] [gpfsug-discuss] AFM gateway node scaling Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all, Is there any guide and or recommendation as to how to scale this. filesets per gateway node? Is it necessary to separate NSD server and gateway roles. Are dedicated gateway nodes licensed as clients? Thanks for any guidance. Matt _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=BosatlBIMbvMZJYB2C0VAcEW4Dr9ApcpPbM9zYSCz7A&s=dmS3n52oSxBzBmWt0E1YgfkPxxwttyfkDBt_sW60f6I&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From mweil at wustl.edu Wed Mar 25 17:01:33 2020 From: mweil at wustl.edu (Matt Weil) Date: Wed, 25 Mar 2020 12:01:33 -0500 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: thank you thank you... I would like to see that in IBM documentation somewhere. On 3/25/20 11:50 AM, Venkateswara R Puvvada wrote: > Matt, > > It is recommended to have dedicated AFM gateway nodes. Memory and CPU > requirements for AFM gateway node depends on the number of filesets > handled by the node and the inode usage of those filesets. Since AFM > keeps track of changes in the memory, any network disturbance can > cause the memory utilization to go high and which eventually leads to > in-memory queue to be dropped. After the queue is dropped, AFM runs > recovery to recover the lost operations which is expensive as it > involves creating the snapshot, running policy scan, doing readdir > from home/secondary and build the list of ?lost operations. When the > gateway node goes down, all the filesets handled by that node > distributed to the remaining active gateway nodes. After the gateway > node comes back, filesets are transferred back to the original gateway > node. When designing the gateway node, make sure that it have enough > memory , CPU resources for handling the incoming and outgoing data > based on the bandwidth. Limit the filesets per gateway(ex. less than > 20 filesets per gateway) ?so that number of AFM recoveries triggered > will be minimal when the queues are lost. Also limit the total number > of inodes handled by the gateway node across all the filesets (ex. > less than 400 million inodes per gateway). ?AFM gateway nodes are > licensed as server nodes. > > > ~Venkat (vpuvvada at in.ibm.com) > > > > From: ? ? ? ?Matt Weil > To: ? ? ? ?gpfsug-discuss at spectrumscale.org > Date: ? ? ? ?03/23/2020 11:39 PM > Subject: ? ? ? ?[EXTERNAL] [gpfsug-discuss] AFM gateway node scaling > Sent by: ? ? ? ?gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > Hello all, > > Is there any guide and or recommendation as to how to scale this. > > filesets per gateway node?? Is it necessary to separate NSD server and > gateway roles.? Are dedicated gateway nodes licensed as clients? > > Thanks for any guidance. > > Matt > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From vpuvvada at in.ibm.com Thu Mar 26 09:03:45 2020 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Thu, 26 Mar 2020 14:33:45 +0530 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: Most of these recommendations documented in KC, we will add missing information on number of filesets and inodes per gateway in the next release. https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_gatewaynodefailureafm.htm ~Venkat (vpuvvada at in.ibm.com) From: Matt Weil To: gpfsug-discuss at spectrumscale.org Date: 03/25/2020 10:34 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] AFM gateway node scaling Sent by: gpfsug-discuss-bounces at spectrumscale.org thank you thank you... I would like to see that in IBM documentation somewhere. On 3/25/20 11:50 AM, Venkateswara R Puvvada wrote: Matt, It is recommended to have dedicated AFM gateway nodes. Memory and CPU requirements for AFM gateway node depends on the number of filesets handled by the node and the inode usage of those filesets. Since AFM keeps track of changes in the memory, any network disturbance can cause the memory utilization to go high and which eventually leads to in-memory queue to be dropped. After the queue is dropped, AFM runs recovery to recover the lost operations which is expensive as it involves creating the snapshot, running policy scan, doing readdir from home/secondary and build the list of lost operations. When the gateway node goes down, all the filesets handled by that node distributed to the remaining active gateway nodes. After the gateway node comes back, filesets are transferred back to the original gateway node. When designing the gateway node, make sure that it have enough memory , CPU resources for handling the incoming and outgoing data based on the bandwidth. Limit the filesets per gateway(ex. less than 20 filesets per gateway) so that number of AFM recoveries triggered will be minimal when the queues are lost. Also limit the total number of inodes handled by the gateway node across all the filesets (ex. less than 400 million inodes per gateway). AFM gateway nodes are licensed as server nodes. ~Venkat (vpuvvada at in.ibm.com) From: Matt Weil To: gpfsug-discuss at spectrumscale.org Date: 03/23/2020 11:39 PM Subject: [EXTERNAL] [gpfsug-discuss] AFM gateway node scaling Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello all, Is there any guide and or recommendation as to how to scale this. filesets per gateway node? Is it necessary to separate NSD server and gateway roles. Are dedicated gateway nodes licensed as clients? Thanks for any guidance. Matt _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=7I3h0KRloboZtWXLSSteXUiOKrzQPEohIjQqN8YdDec&s=WiXsp6jbHf0g1paCMaCkruQpEklQOYDQQ4yQaGbksqA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From willi.engeli at id.ethz.ch Thu Mar 26 09:27:07 2020 From: willi.engeli at id.ethz.ch (Engeli Willi (ID SD)) Date: Thu, 26 Mar 2020 09:27:07 +0000 Subject: [gpfsug-discuss] AFM gateway node scaling Message-ID: <4ea3a7b7d5504ceca414b489152cbdeb@id.ethz.ch> We are using AFM as well on a relatively small cluster. But if I see the recommended scaling factors, the cluster will grow in Gateway nodes quicker then everywhere else. I have had a recovery during a multiple week period. I understand the reasons for the limits. I would wish that the communication between home and cash system get realized in a way that latencies and number of requests are handled more efficiently. That would allow to set the limitations to Bandwidth and Volume. Kind regards Willi -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5803 bytes Desc: not available URL: From mweil at wustl.edu Thu Mar 26 19:26:14 2020 From: mweil at wustl.edu (Matt Weil) Date: Thu, 26 Mar 2020 14:26:14 -0500 Subject: [gpfsug-discuss] AFM gateway node scaling In-Reply-To: References: <3c41cbe2-f6f8-0db5-4aa5-baab6e4b4e29@wustl.edu> Message-ID: Also I found no documentation that advised against having the gateway role on a nsd server.? There was advise to not run the gateway role on a CES node.? What is the recommendation there.? Would a SAN client or shared disk be preferred to keep the latency down. Thanks Matt On 3/26/20 4:03 AM, Venkateswara R Puvvada wrote: > Most of these recommendations documented in KC, we will add missing > information on number of filesets and inodes per gateway in the next > release. > > https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_gatewaynodefailureafm.htm > > ~Venkat (vpuvvada at in.ibm.com) > > > > From: ? ? ? ?Matt Weil > To: ? ? ? ?gpfsug-discuss at spectrumscale.org > Date: ? ? ? ?03/25/2020 10:34 PM > Subject: ? ? ? ?[EXTERNAL] Re: [gpfsug-discuss] AFM gateway node scaling > Sent by: ? ? ? ?gpfsug-discuss-bounces at spectrumscale.org > ------------------------------------------------------------------------ > > > > thank you thank you... I would like to see that in IBM documentation > somewhere. > > On 3/25/20 11:50 AM, Venkateswara R Puvvada wrote: > Matt, > > It is recommended to have dedicated AFM gateway nodes. Memory and CPU > requirements for AFM gateway node depends on the number of filesets > handled by the node and the inode usage of those filesets. Since AFM > keeps track of changes in the memory, any network disturbance can > cause the memory utilization to go high and which eventually leads to > in-memory queue to be dropped. After the queue is dropped, AFM runs > recovery to recover the lost operations which is expensive as it > involves creating the snapshot, running policy scan, doing readdir > from home/secondary and build the list of ?lost operations. When the > gateway node goes down, all the filesets handled by that node > distributed to the remaining active gateway nodes. After the gateway > node comes back, filesets are transferred back to the original gateway > node. When designing the gateway node, make sure that it have enough > memory , CPU resources for handling the incoming and outgoing data > based on the bandwidth. Limit the filesets per gateway(ex. less than > 20 filesets per gateway) ?so that number of AFM recoveries triggered > will be minimal when the queues are lost. Also limit the total number > of inodes handled by the gateway node across all the filesets (ex. > less than 400 million inodes per gateway). ?AFM gateway nodes are > licensed as server nodes. > > > ~Venkat (_vpuvvada at in.ibm.com_ ) > > > > From: ? ? ? ?Matt Weil __ > To: ? ? ? ?_gpfsug-discuss at spectrumscale.org_ > > Date: ? ? ? ?03/23/2020 11:39 PM > Subject: ? ? ? ?[EXTERNAL] [gpfsug-discuss] AFM gateway node scaling > Sent by: ? ? ? ?_gpfsug-discuss-bounces at spectrumscale.org_ > > > ------------------------------------------------------------------------ > > > > Hello all, > > Is there any guide and or recommendation as to how to scale this. > > filesets per gateway node? ?Is it necessary to separate NSD server and > gateway roles. ?Are dedicated gateway nodes licensed as clients? > > Thanks for any guidance. > > Matt > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org_ > __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_ > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_ > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: