From kenneth.waegeman at ugent.be Mon May 4 10:11:50 2020 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Mon, 4 May 2020 11:11:50 +0200 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? Message-ID: Hi all, I didn't see any updates in the faq yet about the new 5.0.4.4 release. Does this release support rhel 7.8 ? Thank you! Kenneth From jonathan.buzzard at strath.ac.uk Mon May 4 11:13:00 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 4 May 2020 11:13:00 +0100 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? In-Reply-To: References: Message-ID: <94ec87a7-64df-1623-37ef-81f2055c70db@strath.ac.uk> On 04/05/2020 10:11, Kenneth Waegeman wrote: > Hi all, > > I didn't see any updates in the faq yet about the new 5.0.4.4 release. > > Does this release support rhel 7.8 ? > Has the fix for 7.7 with a kernel >= 3.10.0-1062.18.1.el7 been released yet? If it hasn't then there is no chance of 7.8 working... JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From kenneth.waegeman at ugent.be Mon May 4 12:58:49 2020 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Mon, 4 May 2020 13:58:49 +0200 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? In-Reply-To: <94ec87a7-64df-1623-37ef-81f2055c70db@strath.ac.uk> References: <94ec87a7-64df-1623-37ef-81f2055c70db@strath.ac.uk> Message-ID: <531e4116-c196-a7b9-61bd-2a7adb98f376@ugent.be> On 04/05/2020 12:13, Jonathan Buzzard wrote: > On 04/05/2020 10:11, Kenneth Waegeman wrote: >> Hi all, >> >> I didn't see any updates in the faq yet about the new 5.0.4.4 release. >> >> Does this release support rhel 7.8 ? >> > > Has the fix for 7.7 with a kernel >= 3.10.0-1062.18.1.el7 been > released yet? If it hasn't then there is no chance of 7.8 working... Yes, this fix is in the release notes of 5.0.4.4: * Item: IJ24294 * Problem description: On RHEL 7.7 node, with supported GPFS versions 4.2.3.18 or higher and 5.0.4.0 or higher, when the kernel is upgraded to a version 3.10.0-1062.18.1 or higher, the node may encounter a kernel crash when accessing a deleted directory * Work around: None * Problem trigger: Accessing a directory which has been deleted * Symptom: Abend/Crash * Platforms affected: All RHEL 7.7 OS environments with kernel version equal or higher than 3.10.0-1062.18.1 * Functional Area affected: All * Customer Impact: High K > > JAB. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Mon May 4 13:19:59 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Mon, 4 May 2020 12:19:59 +0000 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? Message-ID: <497CD318-97FF-40D9-ACE4-D50D83D2B1CF@us.ibm.com> Right now the dev/test team is working flat-out to try to get RHEL 7.8 supported with the upcoming 5.0.5 release, expected this month. We haven't yet made a decision about 5.0.4.4. It will depend on what we learn about the changes required to support RHEL 7.8, and whether those are small enough to be worth backporting to 5.0.4.4. Bear in mind that 5.0.4.4 was the last PTF in the 5.0.4 stream, and customers who can update to 5.0.5 are advised to do so. 5.0.5 will be our first Extended Updates release, meaning you can stay there for up to two years while still receiving PTFs including security fixes. There is no additional charge for this: it's included in your existing S&S. As soon as I know more, I'll share it here. Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com From knop at us.ibm.com Mon May 4 14:16:22 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 4 May 2020 13:16:22 +0000 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? In-Reply-To: <497CD318-97FF-40D9-ACE4-D50D83D2B1CF@us.ibm.com> References: <497CD318-97FF-40D9-ACE4-D50D83D2B1CF@us.ibm.com> Message-ID: An HTML attachment was scrubbed... URL: From petr.plodik at mcomputers.cz Tue May 5 00:13:44 2020 From: petr.plodik at mcomputers.cz (=?iso-8859-1?Q?Petr_Plod=EDk?=) Date: Mon, 4 May 2020 23:13:44 +0000 Subject: [gpfsug-discuss] GPFS ECE IOPS scaling Message-ID: <8ec7eb20765b472aa5d900da9437983c@MCOMPEXCH01.mcomputers.local> Hi all, do you have any experiences / references with GPFS (ECE) scaling in terms of IOPS performance? We are looking for HPC scratch storage with 5M IOPS (4KiB block size, 80%/20% read/write) with two drives failure redundancy. Thank you! Petr Plodik From christian.vieser at 1und1.de Thu May 7 11:01:49 2020 From: christian.vieser at 1und1.de (Christian Vieser) Date: Thu, 7 May 2020 12:01:49 +0200 Subject: [gpfsug-discuss] Enabling SSL/HTTPS/ on Object S3. Message-ID: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> Hi Andi, up to now there are no instructions available on how to enable SSL on the Swift/S3 endpoints. The only thing is that you can enable SSL on the authentication path. So your connection to Swift authentication on port 35357 will be secured and the S3 authentication arriving at http port 8080 will internally take the SSL path, if configured properly. We have successfully done that in a test environment. Be sure to use the --pwd-file option with the "mmuserauth service create ..." and verify the proxy settings afterwards. It should look like this: # mmobj config list --ccrfile proxy-server.conf --section filter:s3token [filter:s3token] auth_uri = https://127.0.0.1:35357/ use = egg:swift3#s3token insecure = true You can correct wrong settings with # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property insecure --value true # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property auth_uri --value 'https://127.0.0.1:35357/' Regards, Christian > i have tried what you suggested. mmobj swift base ran fine. but after i have > deleted the userauth and try to set it up again with ks-ssl enabled it just > hangs: > > # mmuserauth service create --data-access-method object --type local > --enable-ks-ssl > > still waiting for it to finish, 15 mins now.. :) >> Basically all i need is this: >> >> https://s3.something.com:8080 https://s3.something.com:8080 which points >> to the WAN ip of the CES cluster (already configured and ready) >> >> and endpoints like this: >> >> None | keystone | identity | True | public | https://cluster_domain:5000/ >> https://cluster_domain:5000/ >> RegionOne | swift | object-store | True | public | >> https://cluster_domain:443/v1/AUTH_%(tenant_id)s >> RegionOne | swift | object-store | True | public | >> https://cluster_domain:8080/v1/AUTH_%(tenant_id)s -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3378 bytes Desc: S/MIME Cryptographic Signature URL: From andi at christiansen.xxx Thu May 7 13:59:43 2020 From: andi at christiansen.xxx (Andi Christiansen) Date: Thu, 7 May 2020 14:59:43 +0200 Subject: [gpfsug-discuss] Enabling SSL/HTTPS/ on Object S3. In-Reply-To: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> References: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> Message-ID: Hi Christian, Thanks for answering! We solved this with lab services a while back now, and ended up setting up haproxys I front of the ces nodes and then they handle the ssl encryption to the S3 API Thanks Andi Christiansen Sendt fra min iPhone > Den 7. maj 2020 kl. 12.08 skrev Christian Vieser : > > ? > Hi Andi, > up to now there are no instructions available on how to enable SSL on the Swift/S3 endpoints. > The only thing is that you can enable SSL on the authentication path. So your connection to Swift authentication on port 35357 will be secured and the S3 authentication arriving at http port 8080 will internally take the SSL path, if configured properly. We have successfully done that in a test environment. Be sure to use the --pwd-file option with the "mmuserauth service create ..." and verify the proxy settings afterwards. It should look like this: > # mmobj config list --ccrfile proxy-server.conf --section filter:s3token > > [filter:s3token] > auth_uri = https://127.0.0.1:35357/ > use = egg:swift3#s3token > insecure = true > > You can correct wrong settings with > # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property insecure --value true > # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property auth_uri --value 'https://127.0.0.1:35357/' > > Regards, > Christian > > > i have tried what you suggested. mmobj swift base ran fine. but after i have > > deleted the userauth and try to set it up again with ks-ssl enabled it just > > hangs: > > > > # mmuserauth service create --data-access-method object --type local > > --enable-ks-ssl > > > > still waiting for it to finish, 15 mins now.. :) > > > >> Basically all i need is this: > >> > >> https://s3.something.com:8080 https://s3.something.com:8080 which points > >> to the WAN ip of the CES cluster (already configured and ready) > >> > >> and endpoints like this: > >> > >> None | keystone | identity | True | public | https://cluster_domain:5000/ > >> https://cluster_domain:5000/ > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:443/v1/AUTH_%(tenant_id)s > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:8080/v1/AUTH_%(tenant_id)s > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu May 7 21:02:12 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 7 May 2020 22:02:12 +0200 Subject: [gpfsug-discuss] Enabling SSL/HTTPS/ on Object S3. In-Reply-To: References: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> Message-ID: (almost verbatim copy of my previous email ? in case anybody else needs it, or has ideas for improvements :-) The way I would do this is to install "haproxy" on all these nodes, and have haproxy terminate SSL and balance incoming requests over the 3 CES-addresses. For S3 -- we only need to provide access to the swift port at 8080. First install haproxy: # yum install haproxy Put your cert and key into /etc/haproxy/ssl.pem: # cat server.key server.crt cert-chain.crt > /etc/haproxy/ssl.pem # chmod 400 /etc/haproxy/ssl.pem Then create a /etc/haproxy/haproxy.cfg: # cat <<'EOF' > /etc/haproxy/haproxy,cfg #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon tune.ssl.default-dh-param 2048 # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 listen stats *:80 maxconn 10 timeout client 100s timeout server 100s timeout connect 100s timeout queue 100s stats enable stats hide-version stats refresh 30s stats show-node stats auth admin:password stats uri /haproxy?stats frontend s3-frontend bind *:443 ssl crt /etc/haproxy/ssl.pem default_backend s3-backend backend s3-backend balance roundrobin server ces1 10.33.23.167:8080 check server ces2 10.33.23.168:8080 check server ces3 10.33.23.169:8080 check EOF # systemctl enable haproxy # systemctl start haproxy You only need to modify the IP-addresses in the s3-backend (make sure they point at your floating CES addresses, not the static ones), and maybe make a better username/password for the stats page at " *http://hostname/haproxy?stats* ". This setup does two layers of loadbalancing. First DNS round-robin, then haproxy roundrobin -- I don't think this is a negative thing, and it does make it possible to tune the loadbalancing further with more advanced algorithms for selecting backends. F.ex. we can do weighting, if some are more powerful than others. Or "leastconn", to send new requests to backends with the least number of active connections. -jf tor. 7. mai 2020 kl. 14:59 skrev Andi Christiansen : > Hi Christian, > > Thanks for answering! > > We solved this with lab services a while back now, and ended up setting up > haproxys I front of the ces nodes and then they handle the ssl encryption > to the S3 API > > Thanks > Andi Christiansen > > Sendt fra min iPhone > > Den 7. maj 2020 kl. 12.08 skrev Christian Vieser < > christian.vieser at 1und1.de>: > > ? > > Hi Andi, > > up to now there are no instructions available on how to enable SSL on the Swift/S3 endpoints. > > The only thing is that you can enable SSL on the authentication path. So your connection to Swift authentication on port 35357 will be secured and the S3 authentication arriving at http port 8080 will internally take the SSL path, if configured properly. We have successfully done that in a test environment. Be sure to use the --pwd-file option with the "mmuserauth service create ..." and verify the proxy settings afterwards. It should look like this: > > # mmobj config list --ccrfile proxy-server.conf --section filter:s3token > > [filter:s3token] > auth_uri = https://127.0.0.1:35357/use = egg:swift3#s3token > insecure = true > > You can correct wrong settings with# mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property insecure --value true > # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property auth_uri --value 'https://127.0.0.1:35357/' > > Regards, > Christian > > > > i have tried what you suggested. mmobj swift base ran fine. but after i have > > deleted the userauth and try to set it up again with ks-ssl enabled it just > > hangs: > > > > # mmuserauth service create --data-access-method object --type local > > --enable-ks-ssl > > > > still waiting for it to finish, 15 mins now.. :) > > > >> Basically all i need is this: > >> > >> https://s3.something.com:8080 https://s3.something.com:8080 which points > >> to the WAN ip of the CES cluster (already configured and ready) > >> > >> and endpoints like this: > >> > >> None | keystone | identity | True | public | https://cluster_domain:5000/ > >> https://cluster_domain:5000/ > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:443/v1/AUTH_%(tenant_id)s > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:8080/v1/AUTH_%(tenant_id)s > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.turner at ed.ac.uk Sat May 9 11:25:28 2020 From: aaron.turner at ed.ac.uk (TURNER Aaron) Date: Sat, 9 May 2020 10:25:28 +0000 Subject: [gpfsug-discuss] Odd networking/name resolution issue Message-ID: Dear All, We are getting, on an intermittent basis with currently no obvious pattern, an issue with GPFS nodes reporting rejecting nodes of the form: nodename.domain.domain.domain.... DNS resolution using the standard command-line tools of the IP address present in the logs does not repeat the domain, and so far it seems isolated to GPFS. Ultimately the nodes are rejected as not responding on the network. Has anyone seen this sort of behaviour before? Regards Aaron Turner The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Sat May 9 12:06:44 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Sat, 9 May 2020 07:06:44 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: DNS shouldn't be relied upon on a GPFS cluster for internal communication/management or data. As a starting point, make sure the IP's and names of all managers/quorum nodes and clients have *unique* entries in the hosts files of all other nodes in the clusters, being the same as how they where joined and licensed in the first place. If you issue a 'mmlscluster' on the cluster manager for the servers and clients, those results should be used to build the common hosts file for all nodes involved. Also, all nodes should have a common ntp configuration, pointing to the same *internal* ntp server, easily accessible via name/IP also on the hosts file. And obviously, you need a stable network, eth or IB. Have a good monitoring tool in place, to rule out network as a possible culprit. In the particular case of IB, check that the fabric managers are doing their jobs properly. And keep one eye on the 'tail -f /var/mmfs/gen/mmfslog' output of the managers and the nodes being expelled for other clues. Jaime On 5/9/2020 06:25:28, TURNER Aaron wrote: > Dear All, > > We are getting, on an intermittent basis with currently no obvious pattern, an issue with GPFS nodes reporting rejecting nodes of the form: > > nodename.domain.domain.domain.... > > DNS resolution using the standard command-line tools of the IP address present in the logs does not repeat the domain, and so far it seems isolated to GPFS. > > Ultimately the nodes are rejected as not responding on the network. > > Has anyone seen this sort of behaviour before? > > Regards > > Aaron Turner > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > . . . ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From jonathan.buzzard at strath.ac.uk Sat May 9 23:22:15 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sat, 9 May 2020 23:22:15 +0100 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: On 09/05/2020 12:06, Jaime Pinto wrote: > DNS shouldn't be relied upon on a GPFS cluster for internal > communication/management or data. > The 1980's have called and want their lack of IP resolution protocols back :-) I would kindly disagree. If your DNS is not working then your cluster is fubar anyway and a zillion other things will also break very rapidly. For us at least half of the running jobs would be dead in a few minutes as failure to contact license servers would cause the software to stop. All authentication and account lookup is also going to fail as well. You could distribute a hosts file but frankly outside of a storage only cluster (as opposed to one with hundreds if not thousands of compute nodes) that is frankly madness and will inevitably come to bite you in the ass because they *will* get out of sync. The only hosts entry we have is for the Salt Stack host because it tries to do things before the DNS resolvers have been setup and consequently breaks otherwise. Which IMHO is duff on it's behalf. I would add I can't think of a time in the last 16 years where internal DNS at any University I have worked at has stopped working for even one millisecond. If DNS is that flaky at your institution then I suggest sacking the people responsible for it's maintenance as being incompetent twits. It is just such a vanishingly remote possibility that it's not worth bothering about. Frankly a aircraft falling out the sky and squishing your data centre seems more likely to me. Finally in a world of IPv6 then anything other than DNS is a utter madness IMHO. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From aaron.turner at ed.ac.uk Sun May 10 08:35:29 2020 From: aaron.turner at ed.ac.uk (TURNER Aaron) Date: Sun, 10 May 2020 07:35:29 +0000 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: , Message-ID: Following on from Jonathan Buzzards comments, I'd also like to point out that I've never known a central DNS failure in a UK HEI for as long as I can remember, and it was certainly not my intention to suggest that as I think a central DNS issue is highly unlikely. And indeed, as I originally noted, the standard command-line tools on the nodes resolve the names as expected, so whatever is going on looks like it affects GPFS only. It may even be that the repetition of the domain names in the logs is just a function of something it is doing when logging when a node is failing to connect for some other reason entirely. It's just not something I recall having seen before and wanted to see if anyone else had seen it. ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard Sent: 09 May 2020 23:22 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Odd networking/name resolution issue On 09/05/2020 12:06, Jaime Pinto wrote: > DNS shouldn't be relied upon on a GPFS cluster for internal > communication/management or data. > The 1980's have called and want their lack of IP resolution protocols back :-) I would kindly disagree. If your DNS is not working then your cluster is fubar anyway and a zillion other things will also break very rapidly. For us at least half of the running jobs would be dead in a few minutes as failure to contact license servers would cause the software to stop. All authentication and account lookup is also going to fail as well. You could distribute a hosts file but frankly outside of a storage only cluster (as opposed to one with hundreds if not thousands of compute nodes) that is frankly madness and will inevitably come to bite you in the ass because they *will* get out of sync. The only hosts entry we have is for the Salt Stack host because it tries to do things before the DNS resolvers have been setup and consequently breaks otherwise. Which IMHO is duff on it's behalf. I would add I can't think of a time in the last 16 years where internal DNS at any University I have worked at has stopped working for even one millisecond. If DNS is that flaky at your institution then I suggest sacking the people responsible for it's maintenance as being incompetent twits. It is just such a vanishingly remote possibility that it's not worth bothering about. Frankly a aircraft falling out the sky and squishing your data centre seems more likely to me. Finally in a world of IPv6 then anything other than DNS is a utter madness IMHO. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Sun May 10 14:28:15 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Sun, 10 May 2020 09:28:15 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: The rationale for my suggestion doesn't have much to do with the central DNS server, but everything to do with the DNS client side of the service. If you have a very busy cluster at times, and a number of nodes really busy with 1000+ IOPs for instance, so much that the OS on the client can't barely spare a cycle to query the DSN server on what the IP associated with the name of interface leading to the GPFS infrastructure is, or even process that response when it returns, on the same interface where it's having contentions and trying to process all the gpfs data transactions, you can have temporary catch 22 situations. This can generate a backlog of waiters, and eventual expelling of some nodes when the cluster managers don't hear from them in reasonable time. It's doesn't really matter if you have a central DNS server in steroids. Jaime On 5/10/2020 03:35:29, TURNER Aaron wrote: > Following on from Jonathan Buzzards comments, I'd also like to point out that I've never known a central DNS failure in a UK HEI for as long as I can remember, and it was certainly not my intention to suggest that as I think a central DNS issue is highly unlikely. And indeed, as I originally noted, the standard command-line tools on the nodes resolve the names as expected, so whatever is going on looks like it affects GPFS only. It may even be that the repetition of the domain names in the logs is just a function of something it is doing when logging when a node is failing to connect for some other reason entirely. It's just not something I recall having seen before and wanted to see if anyone else had seen it. > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard > *Sent:* 09 May 2020 23:22 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* Re: [gpfsug-discuss] Odd networking/name resolution issue > On 09/05/2020 12:06, Jaime Pinto wrote: >> DNS shouldn't be relied upon on a GPFS cluster for internal >> communication/management or data. >> > > The 1980's have called and want their lack of IP resolution protocols > back :-) > > I would kindly disagree. If your DNS is not working then your cluster is > fubar anyway and a zillion other things will also break very rapidly. > For us at least half of the running jobs would be dead in a few minutes > as failure to contact license servers would cause the software to stop. > All authentication and account lookup is also going to fail as well. > > You could distribute a hosts file but frankly outside of a storage only > cluster (as opposed to one with hundreds if not thousands of compute > nodes) that is frankly madness and will inevitably come to bite you in > the ass because they *will* get out of sync. The only hosts entry we > have is for the Salt Stack host because it tries to do things before the > DNS resolvers have been setup and consequently breaks otherwise. Which > IMHO is duff on it's behalf. > > I would add I can't think of a time in the last 16 years where internal > DNS at any University I have worked at has stopped working for even one > millisecond. If DNS is that flaky at your institution then I suggest > sacking the people responsible for it's maintenance as being incompetent > twits. It is just such a vanishingly remote possibility that it's not > worth bothering about. Frankly a aircraft falling out the sky and > squishing your data centre seems more likely to me. > > Finally in a world of IPv6 then anything other than DNS is a utter > madness IMHO. > > > JAB. > > -- > Jonathan A. Buzzard???????????????????????? Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From richard.rupp at us.ibm.com Sun May 10 19:31:39 2020 From: richard.rupp at us.ibm.com (RICHARD RUPP) Date: Sun, 10 May 2020 14:31:39 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: Normally the DNS server publishes a TTL and the client side caches the info until the TTL expires. Could the server side be mis-configured for a very short TTL? Regards, Richard Rupp, Sales Specialist, Phone: 1-347-510-6746 From: Jaime Pinto To: gpfsug main discussion list , TURNER Aaron Date: 05/10/2020 09:28 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] Odd networking/name resolution issue Sent by: gpfsug-discuss-bounces at spectrumscale.org The rationale for my suggestion doesn't have much to do with the central DNS server, but everything to do with the DNS client side of the service. If you have a very busy cluster at times, and a number of nodes really busy with 1000+ IOPs for instance, so much that the OS on the client can't barely spare a cycle to query the DSN server on what the IP associated with the name of interface leading to the GPFS infrastructure is, or even process that response when it returns, on the same interface where it's having contentions and trying to process all the gpfs data transactions, you can have temporary catch 22 situations. This can generate a backlog of waiters, and eventual expelling of some nodes when the cluster managers don't hear from them in reasonable time. It's doesn't really matter if you have a central DNS server in steroids. Jaime On 5/10/2020 03:35:29, TURNER Aaron wrote: > Following on from Jonathan Buzzards comments, I'd also like to point out that I've never known a central DNS failure in a UK HEI for as long as I can remember, and it was certainly not my intention to suggest that as I think a central DNS issue is highly unlikely. And indeed, as I originally noted, the standard command-line tools on the nodes resolve the names as expected, so whatever is going on looks like it affects GPFS only. It may even be that the repetition of the domain names in the logs is just a function of something it is doing when logging when a node is failing to connect for some other reason entirely. It's just not something I recall having seen before and wanted to see if anyone else had seen it. > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard > *Sent:* 09 May 2020 23:22 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* Re: [gpfsug-discuss] Odd networking/name resolution issue > On 09/05/2020 12:06, Jaime Pinto wrote: >> DNS shouldn't be relied upon on a GPFS cluster for internal >> communication/management or data. >> > > The 1980's have called and want their lack of IP resolution protocols > back :-) > > I would kindly disagree. If your DNS is not working then your cluster is > fubar anyway and a zillion other things will also break very rapidly. > For us at least half of the running jobs would be dead in a few minutes > as failure to contact license servers would cause the software to stop. > All authentication and account lookup is also going to fail as well. > > You could distribute a hosts file but frankly outside of a storage only > cluster (as opposed to one with hundreds if not thousands of compute > nodes) that is frankly madness and will inevitably come to bite you in > the ass because they *will* get out of sync. The only hosts entry we > have is for the Salt Stack host because it tries to do things before the > DNS resolvers have been setup and consequently breaks otherwise. Which > IMHO is duff on it's behalf. > > I would add I can't think of a time in the last 16 years where internal > DNS at any University I have worked at has stopped working for even one > millisecond. If DNS is that flaky at your institution then I suggest > sacking the people responsible for it's maintenance as being incompetent > twits. It is just such a vanishingly remote possibility that it's not > worth bothering about. Frankly a aircraft falling out the sky and > squishing your data centre seems more likely to me. > > Finally in a world of IPv6 then anything other than DNS is a utter > madness IMHO. > > > JAB. > > -- > Jonathan A. Buzzard???????????????????????? Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=EXL-jEd1jmdzvOIhT87C7SIqmAS9uhVQ6J3kObct4OY&m=celJq_mDruJ5qISvmV3mGvLXYvIVcOQlhK3uohc8ZWI&s=31S9Rqbtlv7T0AqAEsLWo8BMPRuHOi9z29DLUYW_PAs&e= > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=EXL-jEd1jmdzvOIhT87C7SIqmAS9uhVQ6J3kObct4OY&m=celJq_mDruJ5qISvmV3mGvLXYvIVcOQlhK3uohc8ZWI&s=31S9Rqbtlv7T0AqAEsLWo8BMPRuHOi9z29DLUYW_PAs&e= > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=EXL-jEd1jmdzvOIhT87C7SIqmAS9uhVQ6J3kObct4OY&m=celJq_mDruJ5qISvmV3mGvLXYvIVcOQlhK3uohc8ZWI&s=31S9Rqbtlv7T0AqAEsLWo8BMPRuHOi9z29DLUYW_PAs&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From jcatana at gmail.com Sun May 10 20:09:39 2020 From: jcatana at gmail.com (Josh Catana) Date: Sun, 10 May 2020 15:09:39 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: I've seen odd behavior like this before and it is to do with name resolution. It might be from your local /etc/hosts entries or potentially the names used to add the nodes to the cluster or even if you are using DNS aliases that are configured improperly. In my case someone added DNS aliases to use in our cluster with fqdn instead of shortname which caused the triple append to appear in the logs you mentioned. I don't think it hurts anything since GPFS has its own name-to-ip table, but you probably want to track it down and fix it to be safe. On Sun, May 10, 2020, 2:31 PM RICHARD RUPP wrote: > *Normally the DNS server publishes a TTL and the client side caches the > info until the TTL expires. Could the server side be mis-configured for a > very short TTL?* > > > Regards, > > *Richard Rupp*, Sales Specialist, *Phone:* *1-347-510-6746* > > > [image: Inactive hide details for Jaime Pinto ---05/10/2020 09:28:46 > AM---The rationale for my suggestion doesn't have much to do with]Jaime > Pinto ---05/10/2020 09:28:46 AM---The rationale for my suggestion doesn't > have much to do with the central DNS server, but everything > > From: Jaime Pinto > To: gpfsug main discussion list , > TURNER Aaron > Date: 05/10/2020 09:28 AM > Subject: [EXTERNAL] Re: [gpfsug-discuss] Odd networking/name resolution > issue > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > The rationale for my suggestion doesn't have much to do with the central > DNS server, but everything to do with the DNS client side of the service. > If you have a very busy cluster at times, and a number of nodes really > busy with 1000+ IOPs for instance, so much that the OS on the client can't > barely spare a cycle to query the DSN server on what the IP associated with > the name of interface leading to the GPFS infrastructure is, or even > process that response when it returns, on the same interface where it's > having contentions and trying to process all the gpfs data transactions, > you can have temporary catch 22 situations. This can generate a backlog of > waiters, and eventual expelling of some nodes when the cluster managers > don't hear from them in reasonable time. > > It's doesn't really matter if you have a central DNS server in steroids. > > Jaime > > On 5/10/2020 03:35:29, TURNER Aaron wrote: > > Following on from Jonathan Buzzards comments, I'd also like to point out > that I've never known a central DNS failure in a UK HEI for as long as I > can remember, and it was certainly not my intention to suggest that as I > think a central DNS issue is highly unlikely. And indeed, as I originally > noted, the standard command-line tools on the nodes resolve the names as > expected, so whatever is going on looks like it affects GPFS only. It may > even be that the repetition of the domain names in the logs is just a > function of something it is doing when logging when a node is failing to > connect for some other reason entirely. It's just not something I recall > having seen before and wanted to see if anyone else had seen it. > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > *From:* gpfsug-discuss-bounces at spectrumscale.org < > gpfsug-discuss-bounces at spectrumscale.org> on behalf of Jonathan Buzzard < > jonathan.buzzard at strath.ac.uk> > > *Sent:* 09 May 2020 23:22 > > *To:* gpfsug-discuss at spectrumscale.org > > > *Subject:* Re: [gpfsug-discuss] Odd networking/name resolution issue > > On 09/05/2020 12:06, Jaime Pinto wrote: > >> DNS shouldn't be relied upon on a GPFS cluster for internal > >> communication/management or data. > >> > > > > The 1980's have called and want their lack of IP resolution protocols > > back :-) > > > > I would kindly disagree. If your DNS is not working then your cluster is > > fubar anyway and a zillion other things will also break very rapidly. > > For us at least half of the running jobs would be dead in a few minutes > > as failure to contact license servers would cause the software to stop. > > All authentication and account lookup is also going to fail as well. > > > > You could distribute a hosts file but frankly outside of a storage only > > cluster (as opposed to one with hundreds if not thousands of compute > > nodes) that is frankly madness and will inevitably come to bite you in > > the ass because they *will* get out of sync. The only hosts entry we > > have is for the Salt Stack host because it tries to do things before the > > DNS resolvers have been setup and consequently breaks otherwise. Which > > IMHO is duff on it's behalf. > > > > I would add I can't think of a time in the last 16 years where internal > > DNS at any University I have worked at has stopped working for even one > > millisecond. If DNS is that flaky at your institution then I suggest > > sacking the people responsible for it's maintenance as being incompetent > > twits. It is just such a vanishingly remote possibility that it's not > > worth bothering about. Frankly a aircraft falling out the sky and > > squishing your data centre seems more likely to me. > > > > Finally in a world of IPv6 then anything other than DNS is a utter > > madness IMHO. > > > > > > JAB. > > > > -- > > Jonathan A. Buzzard Tel: +44141-5483420 > > HPC System Administrator, ARCHIE-WeSt. > > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From knop at us.ibm.com Mon May 11 03:54:13 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 11 May 2020 02:54:13 +0000 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon May 11 11:01:49 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 11 May 2020 11:01:49 +0100 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: <1bc652e7-9c81-82ed-a962-23c24a5c83b2@strath.ac.uk> On 10/05/2020 14:28, Jaime Pinto wrote: > The rationale for my suggestion doesn't have much to do with the central > DNS server, but everything to do with the DNS client side of the service. > If you have a very busy cluster at times, and a number of nodes really > busy with 1000+ IOPs for instance, so much that the OS on the client > can't barely spare a cycle to query the DSN server on what the IP > associated with the name of interface leading to the GPFS infrastructure > is, or even process that response when it returns, on the same interface > where it's having contentions and trying to process all the gpfs data > transactions, you can have temporary catch 22 situations. This can > generate a backlog of waiters, and eventual expelling of some nodes when > the cluster managers don't hear from them in reasonable time. > > It's doesn't really matter if you have a central DNS server in steroids. > If that is the scenario it will struggle to look up the DNS to IP lists in /etc/hosts. GPFS itself will struggle to get scheduled. Besides which systemd is caching it all locally anyways which will get hit *before* /etc/hosts does. In an none systemd install then most likely nscd is running which provides similar service. You don't appear to a full grasp of how it hostname to IP resolution works. It is a outdated notion to suggest using a system that was deprecated over 30 years ago that needs stomping on because it comes from IMHO a lack of seeing the whole picture. Finally as I understand it GPFS maintains it own hostname to IP lookup table which makes it a completely moot point. You can see this from the fact you cannot change the IP address of a node in a cluster. You must remove it and then rejoin with the new IP address. If it was storing client information as hostnames and using hostname to IP resolution to work out the IP addresses that would not be the case. As such your suggestion is dreaming up a solution to a none existent problem, which makes it an even worse idea IMHO. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From TROPPENS at de.ibm.com Tue May 12 13:41:00 2020 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Tue, 12 May 2020 12:41:00 +0000 Subject: [gpfsug-discuss] THINK Virtual User Group Day In-Reply-To: <5BE5B210-5FEE-45E0-AC0D-1B184B5B8E45@spectrumscale.org> References: <5BE5B210-5FEE-45E0-AC0D-1B184B5B8E45@spectrumscale.org> Message-ID: An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Thu May 14 13:20:57 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Thu, 14 May 2020 12:20:57 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Message-ID: Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_2020152979] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From dean.flanders at fmi.ch Thu May 14 13:31:06 2020 From: dean.flanders at fmi.ch (Flanders, Dean) Date: Thu, 14 May 2020 12:31:06 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: References: Message-ID: Hello, It is great, that RHEL 7.8 is supported on SS 4.2.3.22, when will RHEL 8.x be supported on GPFS SS 4.2.3.X?? Thanks, Dean From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Carl Zetie - carlz at us.ibm.com Sent: Thursday, May 14, 2020 2:21 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_2020152979] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 13822 bytes Desc: image002.png URL: From Dugan.Witherick at warwick.ac.uk Thu May 14 14:03:17 2020 From: Dugan.Witherick at warwick.ac.uk (Witherick, Dugan) Date: Thu, 14 May 2020 13:03:17 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: References: Message-ID: <69508fba4e506dfed64703801f3756cfd29e8f1c.camel@warwick.ac.uk> Thanks Carl! Best regards, Dugan On Thu, 2020-05-14 at 12:20 +0000, Carl Zetie - carlz at us.ibm.com wrote: Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_2020152979] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From jonathan.buzzard at strath.ac.uk Thu May 14 15:22:25 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 14 May 2020 15:22:25 +0100 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: References: Message-ID: <0a7278de-33c6-ad20-d818-f3bbf5d19bcb@strath.ac.uk> On 14/05/2020 13:31, Flanders, Dean wrote: > Hello, It is great, that RHEL 7.8 is supported on SS 4.2.3.22, when will > RHEL 8.x be supported on GPFS SS 4.2.3.X?? Thanks, Dean That would be never, 4.2.3 goes out of support in September. Is 5.x supported in 7.8 yet? Some urgent upgrading of systems being looked into right now and I would prefer to jump to 5.x than 4.2.3.22 and have to upgrade again. However needs must. Oh and if you haven't yet I would recommend any HPC site revoking all your users SSH keys immediately. To many users have been creating SSH keys without passphrases and jumping from system to system. Several sites in the UK are currently down and I understand it has effected Germany too :-( There are zero guesses to the origin of the hacks. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From stockf at us.ibm.com Thu May 14 15:55:04 2020 From: stockf at us.ibm.com (Frederick Stock) Date: Thu, 14 May 2020 14:55:04 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: <0a7278de-33c6-ad20-d818-f3bbf5d19bcb@strath.ac.uk> References: <0a7278de-33c6-ad20-d818-f3bbf5d19bcb@strath.ac.uk>, Message-ID: An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Fri May 15 13:07:27 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Fri, 15 May 2020 12:07:27 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Message-ID: <04A3D7D0-749D-4B5C-A6EC-117CA02B003E@us.ibm.com> JAB> Is 5.x supported in 7.8 yet? Support is planned for 5.0.5 later this month, subject to final regression testing now in progress Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_1898525102] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From carlz at us.ibm.com Mon May 18 13:06:55 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Mon, 18 May 2020 12:06:55 +0000 Subject: [gpfsug-discuss] Scale and Linux distros FAQ improvement Message-ID: Inspired by the RHEL 7.8 compatibility issues (and some earlier incidents too), we have adopted a new practice for the Scale FAQ where we will list new distro releases explicitly with a note indicating if they are not yet supported. In the past we should say things like ?7.7.xxx or later? for the supported release level, and not say anything at all about new OS releases until they were supported. That would cause confusion when 7.8 came along, since 7.8 is definitely later than 7.7.xxx... Hopefully this small change will help you with your admin. Regards, Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_565114115] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From b.cregan at imperial.ac.uk Mon May 18 13:59:09 2020 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Mon, 18 May 2020 12:59:09 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues Message-ID: Hi, At Imperial we have been experiencing an issue with SS 5.0.x and quotas. The main symptom is a slow decay in the accuracy of reported quota usage when compared to the actual usage as reported by "du". This discrepancy can be as little as a few percent and as much as many X100% . We also sometimes see bizarre effects such negative file number counts being reported. We have been working with IBM and have put in the latest 5.0.4-4 (that has a fix for mmcheckquota) that we have been pinning our hopes on, but this has not worked. Is anyone else experiencing similar issues? We need to try and get an idea if this is an issue peculiar to our set up or a more general SS problem. We are using user and group quotas in a fileset context. Thanks Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From pinto at scinet.utoronto.ca Mon May 18 14:55:47 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 18 May 2020 09:55:47 -0400 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: References: Message-ID: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> So Bob, Yes, we too have observed an uncharacteristic lag on the correction of the internal quota accounting on GPFS since we updated from version 3.3 to version 4.x some 7-8 years ago. That lag remains through version 5.0.x as well. And it persisted through several appliances (DDN, G200, GSS, ESS and now DSS-G). In our university environment there is also a lot of data churning, in particular small files. The workaround has always been to periodically run mmcheckquota on the top independent fileset to expedite that correction (I have a crontab script that measures the relative size of the in-dought columns on the mmrepquota report, size and inodes, and if either exceeds 2% for any USR/GRP/FILESET I run mmrepquota) We have opened supports calls with IBM about this issue in the past, and we never got a solution, to possibly adjust some GPFS configuration parameter, and have this correction done automatically. We gave up. And that begs the question: what do you mean by "... 5.0.4-4 ... that has a fix for mmcheckquota"? Isn't mmcheckquota zeroing the in-doubt columns when you run it? The fix should be for gpfs (something buried in the code over many versions). As far as I can tell there has never been anything wrong with mmcheckquota. Thanks Jaime On 5/18/2020 08:59:09, Cregan, Bob wrote: > Hi, > ? ? ? At Imperial we? have been experiencing an issue with SS 5.0.x and quotas. The main symptom is a slow decay in the accuracy of reported quota usage when compared to the actual usage as reported by "du". This discrepancy can be as little as a few percent and as much as many? X100% . We also sometimes see bizarre effects such negative file number counts being reported. > > We have been working with IBM? and have put in the latest 5.0.4-4 (that has a fix for mmcheckquota) that we have been pinning our hopes on, but this has not worked. > > Is anyone else experiencing similar issues? We need to try and get an idea if this is an issue peculiar to our set up or a more general SS problem. > > We are using user and group quotas in a fileset context. > > Thanks > > > *Bob Cregan* > HPC Systems Analyst > > Information & Communication Technologies > > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > > W: www.imperial.ac.uk/ict/rcs > > _1505984389175_twitter.png @imperialRCS @imperialRSE > > 1505983882959_Imperial-RCS.png > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > . . . ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From knop at us.ibm.com Mon May 18 17:58:06 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 18 May 2020 16:58:06 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> References: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca>, Message-ID: An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Mon May 18 18:34:06 2020 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Mon, 18 May 2020 17:34:06 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: References: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> Message-ID: Is there a simple way to tell if we?re currently being affected by this bug? We run 5.0.4-1 on the client side and 5.0.3-2.3 on the server side (DSS-G 2.4b I believe it is). -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' > On May 18, 2020, at 12:58 PM, Felipe Knop wrote: > > All, > > Likely not an answer to these situations, but a fix was introduced in 5.0.4.4 (APAR IJ22894) and 4.2.3.22 (APAR IJ24661) to address a (long-standing) problem where mmcheckquota would compute quotas incorrectly in file systems where metadata pool and data pool subblocksizes are different. > > Felipe > > ---- > Felipe Knop knop at us.ibm.com > GPFS Development and Security > IBM Systems > IBM Building 008 > 2455 South Rd, Poughkeepsie, NY 12601 > (845) 433-9314 T/L 293-9314 > > > > ----- Original message ----- > From: Jaime Pinto > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] SS 5.0.x and quota issues > Date: Mon, May 18, 2020 9:56 AM > > So Bob, > Yes, we too have observed an uncharacteristic lag on the correction of the internal quota accounting on GPFS since we updated from version 3.3 to version 4.x some 7-8 years ago. That lag remains through version 5.0.x as well. And it persisted through several appliances (DDN, G200, GSS, ESS and now DSS-G). In our university environment there is also a lot of data churning, in particular small files. > > The workaround has always been to periodically run mmcheckquota on the top independent fileset to expedite that correction (I have a crontab script that measures the relative size of the in-dought columns on the mmrepquota report, size and inodes, and if either exceeds 2% for any USR/GRP/FILESET I run mmrepquota) > > We have opened supports calls with IBM about this issue in the past, and we never got a solution, to possibly adjust some GPFS configuration parameter, and have this correction done automatically. We gave up. > > And that begs the question: what do you mean by "... 5.0.4-4 ... that has a fix for mmcheckquota"? Isn't mmcheckquota zeroing the in-doubt columns when you run it? > > The fix should be for gpfs (something buried in the code over many versions). As far as I can tell there has never been anything wrong with mmcheckquota. > > Thanks > Jaime > > > On 5/18/2020 08:59:09, Cregan, Bob wrote: > > Hi, > > At Imperial we have been experiencing an issue with SS 5.0.x and quotas. The main symptom is a slow decay in the accuracy of reported quota usage when compared to the actual usage as reported by "du". This discrepancy can be as little as a few percent and as much as many X100% . We also sometimes see bizarre effects such negative file number counts being reported. > > > > We have been working with IBM and have put in the latest 5.0.4-4 (that has a fix for mmcheckquota) that we have been pinning our hopes on, but this has not worked. > > > > Is anyone else experiencing similar issues? We need to try and get an idea if this is an issue peculiar to our set up or a more general SS problem. > > > > We are using user and group quotas in a fileset context. > > > > Thanks > > > > > > *Bob Cregan* > > HPC Systems Analyst > > > > Information & Communication Technologies > > > > Imperial College London, > > South Kensington Campus London, SW7 2AZ > > T: 07712388129 > > E: b.cregan at imperial.ac.uk > > > > W: www.imperial.ac.uk/ict/rcs > > > > _1505984389175_twitter.png @imperialRCS @imperialRSE > > > > 1505983882959_Imperial-RCS.png > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > . > . > . ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto - Storage Analyst > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From knop at us.ibm.com Mon May 18 21:05:07 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 18 May 2020 20:05:07 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: References: , <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> Message-ID: An HTML attachment was scrubbed... URL: From juergen.hannappel at desy.de Tue May 19 15:30:34 2020 From: juergen.hannappel at desy.de (Hannappel, Juergen) Date: Tue, 19 May 2020 16:30:34 +0200 (CEST) Subject: [gpfsug-discuss] gpfs_fcntl fails on read-only-fs Message-ID: <1008080204.14431165.1589898634575.JavaMail.zimbra@desy.de> Hi, I tried to do gpfs_fcntl with the GPFS_ACCESS_RANGE hint, with the isWrite field set to 0 and start = 0, lenght = size of the file to indicate that I want to read the entire file now, but on a readonly remote file system the gpfs_fcntl() call returns -1 and sets errno to "Read-only file system" whic is true but should not matter in my Opinion. Why is announcing a range of the file for reading not allowed on a read onoy file system? -- Dr. J?rgen Hannappel DESY/IT Tel. : +49 40 8998-4616 From carlz at us.ibm.com Wed May 20 14:05:11 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Wed, 20 May 2020 13:05:11 +0000 Subject: [gpfsug-discuss] Are you using rsync? Message-ID: Folks, I?m looking for a handful of volunteers who are using rsync with Scale and willing to jump on a 15 minute call (separate calls for each volunteer, to be clear) to discuss how you are using rsync, what features are essential to you, what its shortcomings are, and anything else you want to share. And if you?re *not* using rsync because of some specific limitation but would like to, I?m also interested in hearing from you. And finally: if you have thoughts on this topic but don?t want to get on the phone, please feel free to email me. Please respond OFF LIST to the email address below. Thanks, Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_74924333] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From Ivano.Talamo at psi.ch Fri May 22 08:47:45 2020 From: Ivano.Talamo at psi.ch (Talamo Ivano Giuseppe (PSI)) Date: Fri, 22 May 2020 07:47:45 +0000 Subject: [gpfsug-discuss] selinux context Message-ID: Hi all, I?m configuring a set of login nodes with home directories in GPFS (but not on /home), with SElinux in enforcing mode and auto creation of home directory (via PAM). I?ve been able to partially achieve my target, by basically running the two following commands: semanage fcontext -a -e /home /das/home restorecon -v /das/home After having done this on one node, the context on the directory is the expected one (system_u:object_r:home_root_t:s0). And everything works as expected (a new user logs in and his directory is created). But on all the other nodes of the cluster still the old context is shown (system_u:object_r:unlabeled_t:s0). Unless I run the restorecon on them too. Furthermore, since the filesystem is a remote-cluster mount, on all the nodes on the central (storage) cluster, the corrent (home_root_t) context is shown. I was expecting the SElinux context to be stored in the inodes, but now the situation looks mixed and I?m puzzled. In case it can help, the login nodes are RHEL 7.7 with Spectrum Scale 5.0.4. The storage is RHEL 7.6 with 5.0.3. Does someone have any experience/idea? Thanks, __________________________________________ Paul Scherrer Institut Ivano Talamo WHGA/038 Forschungsstrasse 111 5232 Villigen PSI Schweiz Telefon: +41 56 310 47 71 E-Mail: ivano.talamo at psi.ch From lists at esquad.de Fri May 22 19:22:51 2020 From: lists at esquad.de (Dieter Mosbach) Date: Fri, 22 May 2020 20:22:51 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) Message-ID: From valdis.kletnieks at vt.edu Sun May 24 09:42:13 2020 From: valdis.kletnieks at vt.edu (Valdis Kl=?utf-8?Q?=c4=93?=tnieks) Date: Sun, 24 May 2020 04:42:13 -0400 Subject: [gpfsug-discuss] selinux context In-Reply-To: References: Message-ID: <34924.1590309733@turing-police> On Fri, 22 May 2020 07:47:45 -0000, "Talamo Ivano Giuseppe (PSI)" said: > After having done this on one node, the context on the directory is the expec > expected one (system_u:object_r:home_root_t:s0). And everything works as expected (a > new user logs in and his directory is created). > But on all the other nodes of the cluster still the old context is shown > (system_u:object_r:unlabeled_t:s0). Unless I run the restorecon on them too. > Furthermore, since the filesystem is a remote-cluster mount, on all the nodes > on the central (storage) cluster, the corrent (home_root_t) context is shown. > I was expecting the SElinux context to be stored in the inodes, but now the > situation looks mixed and I???m puzzled. I suspect the issue is that the other nodes have that inode cached already, and they don't find out that that the SELinux context has been changed. I can't tell from here from whether GPFS is failing to realize that a context change means the old inode is stale just like any other inode change, or if there's something else that has gone astray. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From scottg at emailhosting.com Sun May 24 14:57:57 2020 From: scottg at emailhosting.com (scott) Date: Sun, 24 May 2020 09:57:57 -0400 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: References: Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Is there a bug-fix list? we cant seem to find it - we can just find the new/change features. On 5/22/20 2:22 PM, Dieter Mosbach wrote: > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Achim.Rehor at de.ibm.com Sun May 24 16:58:35 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Sun, 24 May 2020 17:58:35 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Message-ID: An HTML attachment was scrubbed... URL: From Achim.Rehor at de.ibm.com Mon May 25 08:48:42 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Mon, 25 May 2020 09:48:42 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Message-ID: no, as this is a .0 package .. there are no 'bug-fixes' ;) you will see this again, when PTF 1 is announced. Mit freundlichen Gr??en / Kind regards Achim Rehor gpfsug-discuss-bounces at spectrumscale.org wrote on 24/05/2020 15:57:57: > From: scott > To: gpfsug-discuss at spectrumscale.org > Date: 24/05/2020 16:04 > Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is > available on FixCentral (n/t) > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Is there a bug-fix list? we cant seem to find it - we can just find the > new/change features. > > On 5/22/20 2:22 PM, Dieter Mosbach wrote: > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=RGTETs2tk0Kz_VOpznDVDkqChhnfLapOTkxLvgmR2- > M&m=P0Br9hWVFau3jYGAI4PCmhGihPTI0UNbIc- > S68rfjy0&s=FeqsNGo3OsZ9n7DfC_Rrx5MyiXeG3hO7oR3YwhWuU8s&e= > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=RGTETs2tk0Kz_VOpznDVDkqChhnfLapOTkxLvgmR2- > M&m=P0Br9hWVFau3jYGAI4PCmhGihPTI0UNbIc- > S68rfjy0&s=FeqsNGo3OsZ9n7DfC_Rrx5MyiXeG3hO7oR3YwhWuU8s&e= > From jonathan.buzzard at strath.ac.uk Mon May 25 19:23:07 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 25 May 2020 19:23:07 +0100 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Message-ID: <06948041-50e5-4150-083d-cdcf52f27515@strath.ac.uk> On 24/05/2020 16:58, Achim Rehor wrote: > > no, as this is a .0 package .. there are no 'bug-fixes' ;) ?you will see > this again, when PTF 1 is announced. So you are saying that no bugs that existed in 5.0.4.x have been fixed in 5.0.5.0? That has the credibility of Dominic Cummings driving to Barnard Castle to "test his eyesight". JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From S.J.Thompson at bham.ac.uk Tue May 26 12:59:30 2020 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Tue, 26 May 2020 11:59:30 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: <06948041-50e5-4150-083d-cdcf52f27515@strath.ac.uk> References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> <06948041-50e5-4150-083d-cdcf52f27515@strath.ac.uk> Message-ID: <35762876-A525-4CD9-8C91-A1CE5DC05135@bham.ac.uk> Well there is an interesting question there ... Where would you start counting the "bugs" from? 5.0.4.0? or 0.0? Or 5.0.4.4? So I think there is a valid point somewhere in there __ Simon ?On 25/05/2020, 19:23, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard" wrote: On 24/05/2020 16:58, Achim Rehor wrote: > > no, as this is a .0 package .. there are no 'bug-fixes' ;) you will see > this again, when PTF 1 is announced. So you are saying that no bugs that existed in 5.0.4.x have been fixed in 5.0.5.0? That has the credibility of Dominic Cummings driving to Barnard Castle to "test his eyesight". JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From carlz at us.ibm.com Tue May 26 14:44:45 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Tue, 26 May 2020 13:44:45 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral Message-ID: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Achim, I think the request here (lost in translation?) is for a list of the bugs that 5.0.5.0 addresses. And we're currently looking to see if we can generate that list. Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com ?On 5/25/20, 7:00 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=obB2s7QQTgU9QMn1708Vpg&m=vmAWHys4rqj2tGjdDeGEv0DQVLBf8oN-bBlHJc23SQ0&s=G30K6mkoWpJkWBqltaiUM49Bn_RWnK2Ke9VTThHnUeo&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) (scott) 2. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) (Achim Rehor) 3. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) (Achim Rehor) ---------------------------------------------------------------------- Message: 1 Date: Sun, 24 May 2020 09:57:57 -0400 From: scott To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b at emailhosting.com> Content-Type: text/plain; charset=utf-8; format=flowed Is there a bug-fix list? we cant seem to find it - we can just find the new/change features. From carlz at us.ibm.com Tue May 26 14:47:05 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Tue, 26 May 2020 13:47:05 +0000 Subject: [gpfsug-discuss] Follow up on Scale and rsync Message-ID: Folks, I want to thank everybody who responded to my request about rsync usage with Scale. I got many more replies than I expected, and several of you went ahead and provided information without even waiting for a phone call. Consequently I have a lot of information to read through and digest, and will probably be rewriting my questions as a result, so it?s going to take me a little longer to get back to those of you who kindly offered your time to discuss the topic. However, the first takeaway is that clearly, a lot of you are heavily dependent on rsync. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_1890439710] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From janfrode at tanso.net Tue May 26 15:51:27 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Tue, 26 May 2020 16:51:27 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral In-Reply-To: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> References: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Message-ID: Seeing that as a %changelog in the RPMs would be fantastic.. :-) -jf tir. 26. mai 2020 kl. 15:44 skrev Carl Zetie - carlz at us.ibm.com < carlz at us.ibm.com>: > Achim, I think the request here (lost in translation?) is for a list of > the bugs that 5.0.5.0 addresses. And we're currently looking to see if we > can generate that list. > > > > Carl Zetie > Program Director > Offering Management > Spectrum Scale > ---- > (919) 473 3318 ][ Research Triangle Park > carlz at us.ibm.com > > > ?On 5/25/20, 7:00 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of gpfsug-discuss-request at spectrumscale.org" < > gpfsug-discuss-bounces at spectrumscale.org on behalf of > gpfsug-discuss-request at spectrumscale.org> wrote: > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=obB2s7QQTgU9QMn1708Vpg&m=vmAWHys4rqj2tGjdDeGEv0DQVLBf8oN-bBlHJc23SQ0&s=G30K6mkoWpJkWBqltaiUM49Bn_RWnK2Ke9VTThHnUeo&e= > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (scott) > 2. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > 3. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 24 May 2020 09:57:57 -0400 > From: scott > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on > FixCentral (n/t) > Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b at emailhosting.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > Is there a bug-fix list? we cant seem to find it - we can just find > the > new/change features. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Achim.Rehor at de.ibm.com Tue May 26 16:04:17 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Tue, 26 May 2020 17:04:17 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral In-Reply-To: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> References: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Message-ID: Yes, Carl, i guess that's what he is looking for ... but Simon has a valid point. Against what would we count bug fixes, as it is a .0 release ? ;) I would guess that's why we never did list "Problems fixed in ..." in the README of the .0 releases. Changes are documented, but these are feature changes new to 5.0.5 (as opposed to 5.0.4) Mit freundlichen Gr??en / Kind regards Achim Rehor Technical Support Specialist Spectrum Scale and ESS (SME) Advisory Product Services Professional IBM Systems Storage Support - EMEA gpfsug-discuss-bounces at spectrumscale.org wrote on 26/05/2020 15:44:45: > From: "Carl Zetie - carlz at us.ibm.com" > To: "gpfsug-discuss at spectrumscale.org" > Date: 26/05/2020 15:44 > Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is > available on FixCentral > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Achim, I think the request here (lost in translation?) is for a list > of the bugs that 5.0.5.0 addresses. And we're currently looking to > see if we can generate that list. > > > > Carl Zetie > Program Director > Offering Management > Spectrum Scale > ---- > (919) 473 3318 ][ Research Triangle Park > carlz at us.ibm.com > > > On 5/25/20, 7:00 AM, "gpfsug-discuss-bounces at spectrumscale.org on > behalf of gpfsug-discuss-request at spectrumscale.org" bounces at spectrumscale.org on behalf of gpfsug-discuss- > request at spectrumscale.org> wrote: > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=obB2s7QQTgU9QMn1708Vpg&m=vmAWHys4rqj2tGjdDeGEv0DQVLBf8oN- > bBlHJc23SQ0&s=G30K6mkoWpJkWBqltaiUM49Bn_RWnK2Ke9VTThHnUeo&e= > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (scott) > 2. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > 3. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 24 May 2020 09:57:57 -0400 > From: scott > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on > FixCentral (n/t) > Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b at emailhosting.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > Is there a bug-fix list? we cant seem to find it - we can just find the > new/change features. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=RGTETs2tk0Kz_VOpznDVDkqChhnfLapOTkxLvgmR2- > M&m=9yPfjB7JuhNahhwFwELNpS_PTi9-2RKvKXsoavY3mgs&s=- > JzPycQHt6vWBH_qWUEBBn8WISeK4Xbfxh4HRJ2zbsE&e= > From christian.vieser at 1und1.de Wed May 27 08:18:44 2020 From: christian.vieser at 1und1.de (Christian Vieser) Date: Wed, 27 May 2020 09:18:44 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral In-Reply-To: References: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Message-ID: <77257593-9ae5-8bf6-4aff-cf6a4ec3aca5@1und1.de> While these all are valid points, I had several support cases in the last months where I got feedback that the issue will be fixed in 5.0.5 or 5.1.0 . So there has to be a list of fixes even in a .0 release, that didn't made it to the last PTF of the previous release. Christian Achim Rehor wrote: > Yes, Carl, > > i guess that's what he is looking for ... but Simon has a valid point. > Against what would we count bug fixes, as it is a .0 release ? ;) > I would guess that's why we never did list "Problems fixed in ..." in the > README of the .0 releases. > Changes are documented, but these are feature changes new to 5.0.5 (as > opposed to 5.0.4) > > > Mit freundlichen Gr??en / Kind regards > > Achim Rehor > Simon Thompson wrote: > Well there is an interesting question there ... > > Where would you start counting the "bugs" from? > > 5.0.4.0? or 0.0? Or 5.0.4.4? > > So I think there is a valid point somewhere in there __ > > Simon -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3378 bytes Desc: S/MIME Cryptographic Signature URL: From heinrich.billich at id.ethz.ch Wed May 27 09:41:11 2020 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Wed, 27 May 2020 08:41:11 +0000 Subject: [gpfsug-discuss] Parse -Y command output Message-ID: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> Hello, I wonder if any python or bash functions exist to parse the output of mm-command's -Y format, i.e. colon-separated with HEADER rows. I would be nice to convert the output to a python list of named tuples or even better pandas dataframe. I would like to access the values by column name and row index. This would allow to do quick scripts or reports easily. Imagine using a jupyter notebook to dig into mmrepquota or mmlsdisk output, even doing charts, ..., easily. I know the ReST interface does exisit, which I could call to get the data in Json format, but to parse the Y-format seems much less cumbersome and allows to collect the data at one time and process it later. If you have other suggestions please let me know. What I would like to get: A list of NSDs which shows the connection to vdisk-set. Vdisk, recovery group and ESS system for each NSD. To get this I need to merge at least two tables. A list of user of fileset quota values highlighting those having non-default vaulues or close to the limit. The ability to get the data quickly in an uniform way into some tool, preferably python, to do analysis or formatting. Cheers, Heiner From juergen.hannappel at desy.de Wed May 27 10:04:00 2020 From: juergen.hannappel at desy.de (Hannappel, Juergen) Date: Wed, 27 May 2020 11:04:00 +0200 (CEST) Subject: [gpfsug-discuss] Parse -Y command output In-Reply-To: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> References: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> Message-ID: <1635082966.18343904.1590570240877.JavaMail.zimbra@desy.de> Hi, an example in python 2.7 (for Python 3 you need to add e.g. errors='ignore' to the parameters of the Popen call to get the proper kind of text stream as output. import subprocess import csv mmlsfileset = subprocess.Popen(['/usr/lpp/mmfs/bin/mmlsfileset', args.filesystem, '-Y'], stdout=subprocess.PIPE) reader = csv.DictReader(mmlsfileset.stdout, delimiter=':') filesets=[] for row in reader: if row['status'] == 'Linked': filesets.append({'name':row['filesetName'], 'linkdir':urllib.unquote(row['path'])}) mmlsfileset.wait() ----- Original Message ----- > From: "Billich Heinrich Rainer (ID SD)" > To: "gpfsug main discussion list" > Sent: Wednesday, 27 May, 2020 10:41:11 > Subject: [gpfsug-discuss] Parse -Y command output > Hello, > > I wonder if any python or bash functions exist to parse the output of > mm-command's -Y format, i.e. colon-separated with HEADER rows. I would be nice > to convert the output to a python list of named tuples or even better pandas > dataframe. I would like to access the values by column name and row index. > This would allow to do quick scripts or reports easily. Imagine using a > jupyter notebook to dig into mmrepquota or mmlsdisk output, even doing charts, > ..., easily. > > I know the ReST interface does exisit, which I could call to get the data in > Json format, but to parse the Y-format seems much less cumbersome and allows to > collect the data at one time and process it later. > > If you have other suggestions please let me know. What I would like to get: > A list of NSDs which shows the connection to vdisk-set. Vdisk, recovery group > and ESS system for each NSD. To get this I need to merge at least two tables. > A list of user of fileset quota values highlighting those having non-default > vaulues or close to the limit. > The ability to get the data quickly in an uniform way into some tool, preferably > python, to do analysis or formatting. > > Cheers, > > Heiner > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From achim.christ at de.ibm.com Wed May 27 10:08:26 2020 From: achim.christ at de.ibm.com (Achim Christ) Date: Wed, 27 May 2020 09:08:26 +0000 Subject: [gpfsug-discuss] Parse -Y command output In-Reply-To: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> References: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> Message-ID: An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Wed May 27 13:39:49 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Wed, 27 May 2020 12:39:49 +0000 Subject: [gpfsug-discuss] Scale release changelog (was: 5.0.5 available on Fix Central) Message-ID: <07847A5D-ED06-43B4-A69A-98A2B02787A3@us.ibm.com> > i guess that's what he is looking for ... but Simon has a valid point. > Against what would we count bug fixes, as it is a .0 release ? ;) The previous Mod level? But really, I'd like to know what the users think would be most useful to them. So I've created an RFE and invite people to submit their comments and votes there: https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=142683 Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com ? From prasad.surampudi at theatsgroup.com Thu May 28 23:31:04 2020 From: prasad.surampudi at theatsgroup.com (Prasad Surampudi) Date: Thu, 28 May 2020 22:31:04 +0000 Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> We have two scale clusters, cluster-A running version Scale 4.2.3 and RHEL6/7 and Cluster-B running Spectrum Scale 5.0.4 and RHEL 8.1. All the nodes in both Cluster-A and Cluster-B are direct attached and no NSD servers. We have our current filesystem gpfs_4 in Cluster-A and new filesystem gpfs_5 in Cluster-B. We want to copy all our data from gpfs_4 filesystem into gpfs_5 which has variable block size. So, can we map NSDs of gpfs_4 to Cluster-B nodes and do a mmexportfs of gpfs_4 from Cluster-A and mmimportfs into Cluster-B so that we have both filesystems available on same node in Cluster-B for copying data across fiber channel? If mmexportfs/mmimportfs works, can we delete nodes from Cluster-A and add them to Cluster-B without upgrading RHEL or GPFS versions for now and plan upgrading them at a later time? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjdoherty at yahoo.com Fri May 29 02:35:13 2020 From: jjdoherty at yahoo.com (Jim Doherty) Date: Fri, 29 May 2020 01:35:13 +0000 (UTC) Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster In-Reply-To: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> References: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> Message-ID: <102892274.973726.1590716113411@mail.yahoo.com> What is the minimum release level of the Spectrum Scale 5.0.4 cluster???? Is it 4.2.3.X?? Jim Doherty On Thursday, May 28, 2020, 6:31:21 PM EDT, Prasad Surampudi wrote: We have two scale clusters, cluster-A running version Scale 4.2.3 and RHEL6/7 and Cluster-B running Spectrum Scale ?5.0.4 and RHEL 8.1. All the nodes in both Cluster-A and Cluster-B are direct attached and no NSD servers. We have our current filesystem gpfs_4 in Cluster-A ?and new filesystem gpfs_5 in Cluster-B. We want to copy all our data from gpfs_4 filesystem into gpfs_5 which has variable block size.? So, can we map NSDs of gpfs_4 to Cluster-B nodes and do a mmexportfs of gpfs_4 from Cluster-A and mmimportfs into Cluster-B so that we have both filesystems available on same node in Cluster-B for copying data across fiber channel? If mmexportfs/mmimportfs works, can we delete nodes from Cluster-A and add them to Cluster-B without upgrading RHEL or GPFS versions for now and ?plan upgrading them at a later time? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Fri May 29 08:22:52 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 29 May 2020 08:22:52 +0100 Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster In-Reply-To: <102892274.973726.1590716113411@mail.yahoo.com> References: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> <102892274.973726.1590716113411@mail.yahoo.com> Message-ID: On 29/05/2020 02:35, Jim Doherty wrote: > > What is the minimum release level of the Spectrum Scale 5.0.4 > cluster???? Is it 4.2.3.X? > According to the email Cluster-B (the one with 5.0.4) has the variable block size thing, so that is going to be a no. Besides which even if you could mmimport the file system into Cluster-B you can't "merge" the two file systems. You are going to need to use AFM, rsync or similar. Hopefull they have enough free space on gpfs_5 to copy the data from gpfs_4 :-) JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From carlz at us.ibm.com Fri May 29 13:25:46 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Fri, 29 May 2020 12:25:46 +0000 Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From prasad.surampudi at theatsgroup.com Fri May 29 16:02:12 2020 From: prasad.surampudi at theatsgroup.com (Prasad Surampudi) Date: Fri, 29 May 2020 15:02:12 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 In-Reply-To: References: Message-ID: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. ?On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** From ulmer at ulmer.org Fri May 29 20:55:01 2020 From: ulmer at ulmer.org (Stephen Ulmer) Date: Fri, 29 May 2020 15:55:01 -0400 Subject: [gpfsug-discuss] Multi-cluster question (was Re: gpfsug-discuss Digest, Vol 100, Issue 32) In-Reply-To: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: I have a question about multi-cluster, but it is related to this thread (it would be solving the same problem). Let?s say we have two clusters A and B, both clusters are normally shared-everything with no NSD servers defined. We want cluster B to be able to use a file system in cluster A. If I zone the SAN such that cluster B can see all of cluster A?s disks, can I then define a multi-cluster relationship between them and mount a file system from A on B? To state it another way, must B's I/O for the foreign file system pass though NSD servers in A, or can B?s nodes discover that they have FibreChannel paths to those disks and use them? I know that the nodes in B become sort-of ?casual" members of A, but I don?t know *how* casual the relationship is... Liberty, -- Stephen > On May 29, 2020, at 11:02 AM, Prasad Surampudi wrote: > > Jim, > The minimum release for 5.0.4 cluster is 5.0.4.3. > > We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. > > > ?On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 > cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 29 May 2020 12:25:46 +0000 > From: "Carl Zetie - carlz at us.ibm.com" > To: "gpfsug-discuss at spectrumscale.org" > > Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem > from 4.2.3 cluster to 5.0.4.3 cluster > Message-ID: > Content-Type: text/plain; charset="utf-8" > > On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. > > If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. > > Regards > > > > > Carl Zetie > Program Director > Offering Management > Spectrum Scale > ---- > (919) 473 3318 ][ Research Triangle Park > carlz at us.ibm.com > > [signature_398516855] > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: image001.png > Type: image/png > Size: 69558 bytes > Desc: image001.png > URL: > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 100, Issue 32 > *********************************************** > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Fri May 29 22:30:08 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 29 May 2020 22:30:08 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 In-Reply-To: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: <3597cc2c-47af-461f-af03-4f88069e1ca6@strath.ac.uk> On 29/05/2020 16:02, Prasad Surampudi wrote: > Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. > > We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. > Our plan is to import the gpfs_4 filesystem into the Spectrum Scale > 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters > nodes) and copy the data using rsync, across fibre channel instead > of ethernet to get higher bandwidth. > Ethernet goes *very* fast these days you know :-) In fact *much* faster than fibre channel. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From sadaniel at us.ibm.com Sat May 30 00:39:59 2020 From: sadaniel at us.ibm.com (Steven Daniels) Date: Fri, 29 May 2020 17:39:59 -0600 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 In-Reply-To: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: Prasad, One of the clients I work with have several 5.0.X ESS (owning) clusters and they share 4.2.3.9 file systems to remote client clusters. The minimum release level on the 5.0.X clusters are at the version of the cluster, i.e. 5.0.3 or 5.0.4 (we are in the process of getting all to latest). The key is that the clients are not part of this cluster as a node has to be > minReleaseLevel to be able to join. They are in their own (accessing) cluster which has a minReleaseLevel of 4.2.3-9. This is the current (until September withdrawal) minReleaseLevel that is supported in this scenario. Don't know specifically about import but it should work as long as the clients that need to mount the sub-5.0 filesystem can be in a separate (remote) cluster or the client nodes are upgraded to the minReleaseLevel of the local cluster. Thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 09:02 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Sent by: gpfsug-discuss-bounces at spectrumscale.org Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=u37vFxPtfEbuPwTW6wj7sbTFugW8l6DsRWtag_KO4Bk&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.png&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=YORJyUthJSKVhDxjwEwePTRexsjhX9F3sXPJg_Z5c5g&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: From prasad.surampudi at theatsgroup.com Sat May 30 00:55:05 2020 From: prasad.surampudi at theatsgroup.com (Prasad Surampudi) Date: Fri, 29 May 2020 23:55:05 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 34 In-Reply-To: References: Message-ID: <6DFF8E9C-EB6E-4A67-90E2-631E625F11BA@theatsgroup.com> Steve, So in essence, if the minReleaseLevel is set to 5.0.x.x for a Spectrum Scale 5.0 cluster, we can't add any new severs with Spectrum Scale 4.x.x RPMs installed without upgrading them to Spectrum Scale 5.x.x RPMs to the cluster. Is this a correct statement? Does mmiportfs also check the filesystem version which it is importing and prevent it from importing if it is created on Spectrum Scale 4.x.x? ?On 5/29/20, 7:40 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: gpfsug-discuss Digest, Vol 100, Issue 32 (Steven Daniels) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 17:39:59 -0600 From: "Steven Daniels" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Message-ID: Content-Type: text/plain; charset="iso-8859-1" Prasad, One of the clients I work with have several 5.0.X ESS (owning) clusters and they share 4.2.3.9 file systems to remote client clusters. The minimum release level on the 5.0.X clusters are at the version of the cluster, i.e. 5.0.3 or 5.0.4 (we are in the process of getting all to latest). The key is that the clients are not part of this cluster as a node has to be > minReleaseLevel to be able to join. They are in their own (accessing) cluster which has a minReleaseLevel of 4.2.3-9. This is the current (until September withdrawal) minReleaseLevel that is supported in this scenario. Don't know specifically about import but it should work as long as the clients that need to mount the sub-5.0 filesystem can be in a separate (remote) cluster or the client nodes are upgraded to the minReleaseLevel of the local cluster. Thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 09:02 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Sent by: gpfsug-discuss-bounces at spectrumscale.org Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=u37vFxPtfEbuPwTW6wj7sbTFugW8l6DsRWtag_KO4Bk&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.png&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=YORJyUthJSKVhDxjwEwePTRexsjhX9F3sXPJg_Z5c5g&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 100, Issue 34 *********************************************** From sadaniel at us.ibm.com Sat May 30 23:19:28 2020 From: sadaniel at us.ibm.com (Steven Daniels) Date: Sat, 30 May 2020 16:19:28 -0600 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 34 In-Reply-To: <6DFF8E9C-EB6E-4A67-90E2-631E625F11BA@theatsgroup.com> References: <6DFF8E9C-EB6E-4A67-90E2-631E625F11BA@theatsgroup.com> Message-ID: see below. Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 05:55 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 34 Sent by: gpfsug-discuss-bounces at spectrumscale.org Steve, So in essence, if the minReleaseLevel is set to 5.0.x.x for a Spectrum Scale 5.0 cluster, we can't add any new severs with Spectrum Scale 4.x.x RPMs installed without upgrading them to Spectrum Scale 5.x.x RPMs to the cluster. Is this a correct statement? Steve>>> This statement is true for the local cluster. However, it is pretty simple to put the 4.2.3-X client nodes(currently 4.2.3.21 is min officially recommended and only until Sep 2020) in their own access cluster. Does mmiportfs also check the filesystem version which it is importing and prevent it from importing if it is created on Spectrum Scale 4.x.x? I hope one of the FS experts can correct if wrong, but it would be in my mind a bug if mmimportfs didn't allow this. I know you can create new 4.2.3-X file systems in a cluster with 5.0.X minReleaseLevel and do that routinely. So, since the cluster supports lower version levels (actually I have one filesystem that is a 3.5.0.7 in a cluster that is 5.0.4.2 minReleaseLevel but obviously 3.5.0.7 wouldn't be anywhere near a recommended filesystm level but it does work fine. NOTE: that the remote cluster will not authenticate unless at a minReleaseLevel of 4.2.2.X (not a supported level) so even though the filesystem can be very back level. The remote accessing cluster has to be no more than one major version back or ahead. On 5/29/20, 7:40 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wzUqgJ_KdsODST1fL88GSy_nGwzeJhtx-ikgQdEt310&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: gpfsug-discuss Digest, Vol 100, Issue 32 (Steven Daniels) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 17:39:59 -0600 From: "Steven Daniels" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Message-ID: Content-Type: text/plain; charset="iso-8859-1" Prasad, One of the clients I work with have several 5.0.X ESS (owning) clusters and they share 4.2.3.9 file systems to remote client clusters. The minimum release level on the 5.0.X clusters are at the version of the cluster, i.e. 5.0.3 or 5.0.4 (we are in the process of getting all to latest). The key is that the clients are not part of this cluster as a node has to be > minReleaseLevel to be able to join. They are in their own (accessing) cluster which has a minReleaseLevel of 4.2.3-9. This is the current (until September withdrawal) minReleaseLevel that is supported in this scenario. Don't know specifically about import but it should work as long as the clients that need to mount the sub-5.0 filesystem can be in a separate (remote) cluster or the client nodes are upgraded to the minReleaseLevel of the local cluster. Thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 09:02 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Sent by: gpfsug-discuss-bounces at spectrumscale.org Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=u37vFxPtfEbuPwTW6wj7sbTFugW8l6DsRWtag_KO4Bk&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.png&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=YORJyUthJSKVhDxjwEwePTRexsjhX9F3sXPJg_Z5c5g&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_0c6df3be_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wo2skeF7H2QM7YNg9RJDB9ntSOGPeaocVSTahs6lCtc&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_0c6df3be_attachment.gif&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wvnV0A6CkiXRcWtU-nWQC_-Li0zXNfBZOW2oADBVg0o&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_0c6df3be_attachment.jpg&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=NcIKR17cT_NpUQ67OrX_fti7Tdnla2I-Q5Ptx82UP3k&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wzUqgJ_KdsODST1fL88GSy_nGwzeJhtx-ikgQdEt310&e= End of gpfsug-discuss Digest, Vol 100, Issue 34 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wzUqgJ_KdsODST1fL88GSy_nGwzeJhtx-ikgQdEt310&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: From jonathan.buzzard at strath.ac.uk Sun May 31 10:31:34 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sun, 31 May 2020 10:31:34 +0100 Subject: [gpfsug-discuss] Multi-cluster question (was Re: gpfsug-discuss Digest, Vol 100, Issue 32) In-Reply-To: References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: <53c17f6a-f3fe-9217-4000-197e1b06a105@strath.ac.uk> On 29/05/2020 20:55, Stephen Ulmer wrote: > I have a question about multi-cluster, but it is related to this thread > (it would be solving the same problem). > > Let?s say we have two clusters A and B, both clusters are normally > shared-everything with no NSD servers defined. Er, even in a shared-everything all nodes fibre channel attached you still have to define NSD servers. That is a given NSD has a server (or ideally a list of servers) that arbitrate the disk. Unless it has changed since 3.x days. Never run a 4.x or later with all the disks SAN attached on all the nodes. > We want cluster B to be > able to use a file system in cluster A. ?If I zone the SAN such that > cluster B can see all of cluster A?s disks, can I then define a > multi-cluster relationship between them and mount a file system from A on B? > > To state it another way, must B's I/O for the foreign file system pass > though NSD servers in A, or can B?s nodes discover that they have > FibreChannel paths to those disks and use them? > My understanding is that remote cluster mounts have to pass through the NSD servers. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From janfrode at tanso.net Sun May 31 17:47:40 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 31 May 2020 18:47:40 +0200 Subject: [gpfsug-discuss] Multi-cluster question (was Re: gpfsug-discuss Digest, Vol 100, Issue 32) In-Reply-To: <53c17f6a-f3fe-9217-4000-197e1b06a105@strath.ac.uk> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> <53c17f6a-f3fe-9217-4000-197e1b06a105@strath.ac.uk> Message-ID: No, this is a common misconception. You don?t need any NSD servers. NSD servers are only needed if you have nodes without direct block access. Remote cluster or not, disk access will be over local block device (without involving NSD servers in any way), or NSD server if local access isn?t available. NSD-servers are not ?arbitrators? over access to a disk, they?re just stupid proxies of IO commands. -jf s?n. 31. mai 2020 kl. 11:31 skrev Jonathan Buzzard < jonathan.buzzard at strath.ac.uk>: > On 29/05/2020 20:55, Stephen Ulmer wrote: > > I have a question about multi-cluster, but it is related to this thread > > (it would be solving the same problem). > > > > Let?s say we have two clusters A and B, both clusters are normally > > shared-everything with no NSD servers defined. > > Er, even in a shared-everything all nodes fibre channel attached you > still have to define NSD servers. That is a given NSD has a server (or > ideally a list of servers) that arbitrate the disk. Unless it has > changed since 3.x days. Never run a 4.x or later with all the disks SAN > attached on all the nodes. > > > We want cluster B to be > > able to use a file system in cluster A. If I zone the SAN such that > > cluster B can see all of cluster A?s disks, can I then define a > > multi-cluster relationship between them and mount a file system from A > on B? > > > > To state it another way, must B's I/O for the foreign file system pass > > though NSD servers in A, or can B?s nodes discover that they have > > FibreChannel paths to those disks and use them? > > > > My understanding is that remote cluster mounts have to pass through the > NSD servers. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Mon May 4 10:11:50 2020 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Mon, 4 May 2020 11:11:50 +0200 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? Message-ID: Hi all, I didn't see any updates in the faq yet about the new 5.0.4.4 release. Does this release support rhel 7.8 ? Thank you! Kenneth From jonathan.buzzard at strath.ac.uk Mon May 4 11:13:00 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 4 May 2020 11:13:00 +0100 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? In-Reply-To: References: Message-ID: <94ec87a7-64df-1623-37ef-81f2055c70db@strath.ac.uk> On 04/05/2020 10:11, Kenneth Waegeman wrote: > Hi all, > > I didn't see any updates in the faq yet about the new 5.0.4.4 release. > > Does this release support rhel 7.8 ? > Has the fix for 7.7 with a kernel >= 3.10.0-1062.18.1.el7 been released yet? If it hasn't then there is no chance of 7.8 working... JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From kenneth.waegeman at ugent.be Mon May 4 12:58:49 2020 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Mon, 4 May 2020 13:58:49 +0200 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? In-Reply-To: <94ec87a7-64df-1623-37ef-81f2055c70db@strath.ac.uk> References: <94ec87a7-64df-1623-37ef-81f2055c70db@strath.ac.uk> Message-ID: <531e4116-c196-a7b9-61bd-2a7adb98f376@ugent.be> On 04/05/2020 12:13, Jonathan Buzzard wrote: > On 04/05/2020 10:11, Kenneth Waegeman wrote: >> Hi all, >> >> I didn't see any updates in the faq yet about the new 5.0.4.4 release. >> >> Does this release support rhel 7.8 ? >> > > Has the fix for 7.7 with a kernel >= 3.10.0-1062.18.1.el7 been > released yet? If it hasn't then there is no chance of 7.8 working... Yes, this fix is in the release notes of 5.0.4.4: * Item: IJ24294 * Problem description: On RHEL 7.7 node, with supported GPFS versions 4.2.3.18 or higher and 5.0.4.0 or higher, when the kernel is upgraded to a version 3.10.0-1062.18.1 or higher, the node may encounter a kernel crash when accessing a deleted directory * Work around: None * Problem trigger: Accessing a directory which has been deleted * Symptom: Abend/Crash * Platforms affected: All RHEL 7.7 OS environments with kernel version equal or higher than 3.10.0-1062.18.1 * Functional Area affected: All * Customer Impact: High K > > JAB. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Mon May 4 13:19:59 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Mon, 4 May 2020 12:19:59 +0000 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? Message-ID: <497CD318-97FF-40D9-ACE4-D50D83D2B1CF@us.ibm.com> Right now the dev/test team is working flat-out to try to get RHEL 7.8 supported with the upcoming 5.0.5 release, expected this month. We haven't yet made a decision about 5.0.4.4. It will depend on what we learn about the changes required to support RHEL 7.8, and whether those are small enough to be worth backporting to 5.0.4.4. Bear in mind that 5.0.4.4 was the last PTF in the 5.0.4 stream, and customers who can update to 5.0.5 are advised to do so. 5.0.5 will be our first Extended Updates release, meaning you can stay there for up to two years while still receiving PTFs including security fixes. There is no additional charge for this: it's included in your existing S&S. As soon as I know more, I'll share it here. Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com From knop at us.ibm.com Mon May 4 14:16:22 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 4 May 2020 13:16:22 +0000 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? In-Reply-To: <497CD318-97FF-40D9-ACE4-D50D83D2B1CF@us.ibm.com> References: <497CD318-97FF-40D9-ACE4-D50D83D2B1CF@us.ibm.com> Message-ID: An HTML attachment was scrubbed... URL: From petr.plodik at mcomputers.cz Tue May 5 00:13:44 2020 From: petr.plodik at mcomputers.cz (=?iso-8859-1?Q?Petr_Plod=EDk?=) Date: Mon, 4 May 2020 23:13:44 +0000 Subject: [gpfsug-discuss] GPFS ECE IOPS scaling Message-ID: <8ec7eb20765b472aa5d900da9437983c@MCOMPEXCH01.mcomputers.local> Hi all, do you have any experiences / references with GPFS (ECE) scaling in terms of IOPS performance? We are looking for HPC scratch storage with 5M IOPS (4KiB block size, 80%/20% read/write) with two drives failure redundancy. Thank you! Petr Plodik From christian.vieser at 1und1.de Thu May 7 11:01:49 2020 From: christian.vieser at 1und1.de (Christian Vieser) Date: Thu, 7 May 2020 12:01:49 +0200 Subject: [gpfsug-discuss] Enabling SSL/HTTPS/ on Object S3. Message-ID: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> Hi Andi, up to now there are no instructions available on how to enable SSL on the Swift/S3 endpoints. The only thing is that you can enable SSL on the authentication path. So your connection to Swift authentication on port 35357 will be secured and the S3 authentication arriving at http port 8080 will internally take the SSL path, if configured properly. We have successfully done that in a test environment. Be sure to use the --pwd-file option with the "mmuserauth service create ..." and verify the proxy settings afterwards. It should look like this: # mmobj config list --ccrfile proxy-server.conf --section filter:s3token [filter:s3token] auth_uri = https://127.0.0.1:35357/ use = egg:swift3#s3token insecure = true You can correct wrong settings with # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property insecure --value true # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property auth_uri --value 'https://127.0.0.1:35357/' Regards, Christian > i have tried what you suggested. mmobj swift base ran fine. but after i have > deleted the userauth and try to set it up again with ks-ssl enabled it just > hangs: > > # mmuserauth service create --data-access-method object --type local > --enable-ks-ssl > > still waiting for it to finish, 15 mins now.. :) >> Basically all i need is this: >> >> https://s3.something.com:8080 https://s3.something.com:8080 which points >> to the WAN ip of the CES cluster (already configured and ready) >> >> and endpoints like this: >> >> None | keystone | identity | True | public | https://cluster_domain:5000/ >> https://cluster_domain:5000/ >> RegionOne | swift | object-store | True | public | >> https://cluster_domain:443/v1/AUTH_%(tenant_id)s >> RegionOne | swift | object-store | True | public | >> https://cluster_domain:8080/v1/AUTH_%(tenant_id)s -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3378 bytes Desc: S/MIME Cryptographic Signature URL: From andi at christiansen.xxx Thu May 7 13:59:43 2020 From: andi at christiansen.xxx (Andi Christiansen) Date: Thu, 7 May 2020 14:59:43 +0200 Subject: [gpfsug-discuss] Enabling SSL/HTTPS/ on Object S3. In-Reply-To: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> References: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> Message-ID: Hi Christian, Thanks for answering! We solved this with lab services a while back now, and ended up setting up haproxys I front of the ces nodes and then they handle the ssl encryption to the S3 API Thanks Andi Christiansen Sendt fra min iPhone > Den 7. maj 2020 kl. 12.08 skrev Christian Vieser : > > ? > Hi Andi, > up to now there are no instructions available on how to enable SSL on the Swift/S3 endpoints. > The only thing is that you can enable SSL on the authentication path. So your connection to Swift authentication on port 35357 will be secured and the S3 authentication arriving at http port 8080 will internally take the SSL path, if configured properly. We have successfully done that in a test environment. Be sure to use the --pwd-file option with the "mmuserauth service create ..." and verify the proxy settings afterwards. It should look like this: > # mmobj config list --ccrfile proxy-server.conf --section filter:s3token > > [filter:s3token] > auth_uri = https://127.0.0.1:35357/ > use = egg:swift3#s3token > insecure = true > > You can correct wrong settings with > # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property insecure --value true > # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property auth_uri --value 'https://127.0.0.1:35357/' > > Regards, > Christian > > > i have tried what you suggested. mmobj swift base ran fine. but after i have > > deleted the userauth and try to set it up again with ks-ssl enabled it just > > hangs: > > > > # mmuserauth service create --data-access-method object --type local > > --enable-ks-ssl > > > > still waiting for it to finish, 15 mins now.. :) > > > >> Basically all i need is this: > >> > >> https://s3.something.com:8080 https://s3.something.com:8080 which points > >> to the WAN ip of the CES cluster (already configured and ready) > >> > >> and endpoints like this: > >> > >> None | keystone | identity | True | public | https://cluster_domain:5000/ > >> https://cluster_domain:5000/ > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:443/v1/AUTH_%(tenant_id)s > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:8080/v1/AUTH_%(tenant_id)s > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu May 7 21:02:12 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 7 May 2020 22:02:12 +0200 Subject: [gpfsug-discuss] Enabling SSL/HTTPS/ on Object S3. In-Reply-To: References: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> Message-ID: (almost verbatim copy of my previous email ? in case anybody else needs it, or has ideas for improvements :-) The way I would do this is to install "haproxy" on all these nodes, and have haproxy terminate SSL and balance incoming requests over the 3 CES-addresses. For S3 -- we only need to provide access to the swift port at 8080. First install haproxy: # yum install haproxy Put your cert and key into /etc/haproxy/ssl.pem: # cat server.key server.crt cert-chain.crt > /etc/haproxy/ssl.pem # chmod 400 /etc/haproxy/ssl.pem Then create a /etc/haproxy/haproxy.cfg: # cat <<'EOF' > /etc/haproxy/haproxy,cfg #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon tune.ssl.default-dh-param 2048 # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 listen stats *:80 maxconn 10 timeout client 100s timeout server 100s timeout connect 100s timeout queue 100s stats enable stats hide-version stats refresh 30s stats show-node stats auth admin:password stats uri /haproxy?stats frontend s3-frontend bind *:443 ssl crt /etc/haproxy/ssl.pem default_backend s3-backend backend s3-backend balance roundrobin server ces1 10.33.23.167:8080 check server ces2 10.33.23.168:8080 check server ces3 10.33.23.169:8080 check EOF # systemctl enable haproxy # systemctl start haproxy You only need to modify the IP-addresses in the s3-backend (make sure they point at your floating CES addresses, not the static ones), and maybe make a better username/password for the stats page at " *http://hostname/haproxy?stats* ". This setup does two layers of loadbalancing. First DNS round-robin, then haproxy roundrobin -- I don't think this is a negative thing, and it does make it possible to tune the loadbalancing further with more advanced algorithms for selecting backends. F.ex. we can do weighting, if some are more powerful than others. Or "leastconn", to send new requests to backends with the least number of active connections. -jf tor. 7. mai 2020 kl. 14:59 skrev Andi Christiansen : > Hi Christian, > > Thanks for answering! > > We solved this with lab services a while back now, and ended up setting up > haproxys I front of the ces nodes and then they handle the ssl encryption > to the S3 API > > Thanks > Andi Christiansen > > Sendt fra min iPhone > > Den 7. maj 2020 kl. 12.08 skrev Christian Vieser < > christian.vieser at 1und1.de>: > > ? > > Hi Andi, > > up to now there are no instructions available on how to enable SSL on the Swift/S3 endpoints. > > The only thing is that you can enable SSL on the authentication path. So your connection to Swift authentication on port 35357 will be secured and the S3 authentication arriving at http port 8080 will internally take the SSL path, if configured properly. We have successfully done that in a test environment. Be sure to use the --pwd-file option with the "mmuserauth service create ..." and verify the proxy settings afterwards. It should look like this: > > # mmobj config list --ccrfile proxy-server.conf --section filter:s3token > > [filter:s3token] > auth_uri = https://127.0.0.1:35357/use = egg:swift3#s3token > insecure = true > > You can correct wrong settings with# mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property insecure --value true > # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property auth_uri --value 'https://127.0.0.1:35357/' > > Regards, > Christian > > > > i have tried what you suggested. mmobj swift base ran fine. but after i have > > deleted the userauth and try to set it up again with ks-ssl enabled it just > > hangs: > > > > # mmuserauth service create --data-access-method object --type local > > --enable-ks-ssl > > > > still waiting for it to finish, 15 mins now.. :) > > > >> Basically all i need is this: > >> > >> https://s3.something.com:8080 https://s3.something.com:8080 which points > >> to the WAN ip of the CES cluster (already configured and ready) > >> > >> and endpoints like this: > >> > >> None | keystone | identity | True | public | https://cluster_domain:5000/ > >> https://cluster_domain:5000/ > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:443/v1/AUTH_%(tenant_id)s > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:8080/v1/AUTH_%(tenant_id)s > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.turner at ed.ac.uk Sat May 9 11:25:28 2020 From: aaron.turner at ed.ac.uk (TURNER Aaron) Date: Sat, 9 May 2020 10:25:28 +0000 Subject: [gpfsug-discuss] Odd networking/name resolution issue Message-ID: Dear All, We are getting, on an intermittent basis with currently no obvious pattern, an issue with GPFS nodes reporting rejecting nodes of the form: nodename.domain.domain.domain.... DNS resolution using the standard command-line tools of the IP address present in the logs does not repeat the domain, and so far it seems isolated to GPFS. Ultimately the nodes are rejected as not responding on the network. Has anyone seen this sort of behaviour before? Regards Aaron Turner The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Sat May 9 12:06:44 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Sat, 9 May 2020 07:06:44 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: DNS shouldn't be relied upon on a GPFS cluster for internal communication/management or data. As a starting point, make sure the IP's and names of all managers/quorum nodes and clients have *unique* entries in the hosts files of all other nodes in the clusters, being the same as how they where joined and licensed in the first place. If you issue a 'mmlscluster' on the cluster manager for the servers and clients, those results should be used to build the common hosts file for all nodes involved. Also, all nodes should have a common ntp configuration, pointing to the same *internal* ntp server, easily accessible via name/IP also on the hosts file. And obviously, you need a stable network, eth or IB. Have a good monitoring tool in place, to rule out network as a possible culprit. In the particular case of IB, check that the fabric managers are doing their jobs properly. And keep one eye on the 'tail -f /var/mmfs/gen/mmfslog' output of the managers and the nodes being expelled for other clues. Jaime On 5/9/2020 06:25:28, TURNER Aaron wrote: > Dear All, > > We are getting, on an intermittent basis with currently no obvious pattern, an issue with GPFS nodes reporting rejecting nodes of the form: > > nodename.domain.domain.domain.... > > DNS resolution using the standard command-line tools of the IP address present in the logs does not repeat the domain, and so far it seems isolated to GPFS. > > Ultimately the nodes are rejected as not responding on the network. > > Has anyone seen this sort of behaviour before? > > Regards > > Aaron Turner > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > . . . ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From jonathan.buzzard at strath.ac.uk Sat May 9 23:22:15 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sat, 9 May 2020 23:22:15 +0100 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: On 09/05/2020 12:06, Jaime Pinto wrote: > DNS shouldn't be relied upon on a GPFS cluster for internal > communication/management or data. > The 1980's have called and want their lack of IP resolution protocols back :-) I would kindly disagree. If your DNS is not working then your cluster is fubar anyway and a zillion other things will also break very rapidly. For us at least half of the running jobs would be dead in a few minutes as failure to contact license servers would cause the software to stop. All authentication and account lookup is also going to fail as well. You could distribute a hosts file but frankly outside of a storage only cluster (as opposed to one with hundreds if not thousands of compute nodes) that is frankly madness and will inevitably come to bite you in the ass because they *will* get out of sync. The only hosts entry we have is for the Salt Stack host because it tries to do things before the DNS resolvers have been setup and consequently breaks otherwise. Which IMHO is duff on it's behalf. I would add I can't think of a time in the last 16 years where internal DNS at any University I have worked at has stopped working for even one millisecond. If DNS is that flaky at your institution then I suggest sacking the people responsible for it's maintenance as being incompetent twits. It is just such a vanishingly remote possibility that it's not worth bothering about. Frankly a aircraft falling out the sky and squishing your data centre seems more likely to me. Finally in a world of IPv6 then anything other than DNS is a utter madness IMHO. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From aaron.turner at ed.ac.uk Sun May 10 08:35:29 2020 From: aaron.turner at ed.ac.uk (TURNER Aaron) Date: Sun, 10 May 2020 07:35:29 +0000 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: , Message-ID: Following on from Jonathan Buzzards comments, I'd also like to point out that I've never known a central DNS failure in a UK HEI for as long as I can remember, and it was certainly not my intention to suggest that as I think a central DNS issue is highly unlikely. And indeed, as I originally noted, the standard command-line tools on the nodes resolve the names as expected, so whatever is going on looks like it affects GPFS only. It may even be that the repetition of the domain names in the logs is just a function of something it is doing when logging when a node is failing to connect for some other reason entirely. It's just not something I recall having seen before and wanted to see if anyone else had seen it. ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard Sent: 09 May 2020 23:22 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Odd networking/name resolution issue On 09/05/2020 12:06, Jaime Pinto wrote: > DNS shouldn't be relied upon on a GPFS cluster for internal > communication/management or data. > The 1980's have called and want their lack of IP resolution protocols back :-) I would kindly disagree. If your DNS is not working then your cluster is fubar anyway and a zillion other things will also break very rapidly. For us at least half of the running jobs would be dead in a few minutes as failure to contact license servers would cause the software to stop. All authentication and account lookup is also going to fail as well. You could distribute a hosts file but frankly outside of a storage only cluster (as opposed to one with hundreds if not thousands of compute nodes) that is frankly madness and will inevitably come to bite you in the ass because they *will* get out of sync. The only hosts entry we have is for the Salt Stack host because it tries to do things before the DNS resolvers have been setup and consequently breaks otherwise. Which IMHO is duff on it's behalf. I would add I can't think of a time in the last 16 years where internal DNS at any University I have worked at has stopped working for even one millisecond. If DNS is that flaky at your institution then I suggest sacking the people responsible for it's maintenance as being incompetent twits. It is just such a vanishingly remote possibility that it's not worth bothering about. Frankly a aircraft falling out the sky and squishing your data centre seems more likely to me. Finally in a world of IPv6 then anything other than DNS is a utter madness IMHO. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Sun May 10 14:28:15 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Sun, 10 May 2020 09:28:15 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: The rationale for my suggestion doesn't have much to do with the central DNS server, but everything to do with the DNS client side of the service. If you have a very busy cluster at times, and a number of nodes really busy with 1000+ IOPs for instance, so much that the OS on the client can't barely spare a cycle to query the DSN server on what the IP associated with the name of interface leading to the GPFS infrastructure is, or even process that response when it returns, on the same interface where it's having contentions and trying to process all the gpfs data transactions, you can have temporary catch 22 situations. This can generate a backlog of waiters, and eventual expelling of some nodes when the cluster managers don't hear from them in reasonable time. It's doesn't really matter if you have a central DNS server in steroids. Jaime On 5/10/2020 03:35:29, TURNER Aaron wrote: > Following on from Jonathan Buzzards comments, I'd also like to point out that I've never known a central DNS failure in a UK HEI for as long as I can remember, and it was certainly not my intention to suggest that as I think a central DNS issue is highly unlikely. And indeed, as I originally noted, the standard command-line tools on the nodes resolve the names as expected, so whatever is going on looks like it affects GPFS only. It may even be that the repetition of the domain names in the logs is just a function of something it is doing when logging when a node is failing to connect for some other reason entirely. It's just not something I recall having seen before and wanted to see if anyone else had seen it. > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard > *Sent:* 09 May 2020 23:22 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* Re: [gpfsug-discuss] Odd networking/name resolution issue > On 09/05/2020 12:06, Jaime Pinto wrote: >> DNS shouldn't be relied upon on a GPFS cluster for internal >> communication/management or data. >> > > The 1980's have called and want their lack of IP resolution protocols > back :-) > > I would kindly disagree. If your DNS is not working then your cluster is > fubar anyway and a zillion other things will also break very rapidly. > For us at least half of the running jobs would be dead in a few minutes > as failure to contact license servers would cause the software to stop. > All authentication and account lookup is also going to fail as well. > > You could distribute a hosts file but frankly outside of a storage only > cluster (as opposed to one with hundreds if not thousands of compute > nodes) that is frankly madness and will inevitably come to bite you in > the ass because they *will* get out of sync. The only hosts entry we > have is for the Salt Stack host because it tries to do things before the > DNS resolvers have been setup and consequently breaks otherwise. Which > IMHO is duff on it's behalf. > > I would add I can't think of a time in the last 16 years where internal > DNS at any University I have worked at has stopped working for even one > millisecond. If DNS is that flaky at your institution then I suggest > sacking the people responsible for it's maintenance as being incompetent > twits. It is just such a vanishingly remote possibility that it's not > worth bothering about. Frankly a aircraft falling out the sky and > squishing your data centre seems more likely to me. > > Finally in a world of IPv6 then anything other than DNS is a utter > madness IMHO. > > > JAB. > > -- > Jonathan A. Buzzard???????????????????????? Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From richard.rupp at us.ibm.com Sun May 10 19:31:39 2020 From: richard.rupp at us.ibm.com (RICHARD RUPP) Date: Sun, 10 May 2020 14:31:39 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: Normally the DNS server publishes a TTL and the client side caches the info until the TTL expires. Could the server side be mis-configured for a very short TTL? Regards, Richard Rupp, Sales Specialist, Phone: 1-347-510-6746 From: Jaime Pinto To: gpfsug main discussion list , TURNER Aaron Date: 05/10/2020 09:28 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] Odd networking/name resolution issue Sent by: gpfsug-discuss-bounces at spectrumscale.org The rationale for my suggestion doesn't have much to do with the central DNS server, but everything to do with the DNS client side of the service. If you have a very busy cluster at times, and a number of nodes really busy with 1000+ IOPs for instance, so much that the OS on the client can't barely spare a cycle to query the DSN server on what the IP associated with the name of interface leading to the GPFS infrastructure is, or even process that response when it returns, on the same interface where it's having contentions and trying to process all the gpfs data transactions, you can have temporary catch 22 situations. This can generate a backlog of waiters, and eventual expelling of some nodes when the cluster managers don't hear from them in reasonable time. It's doesn't really matter if you have a central DNS server in steroids. Jaime On 5/10/2020 03:35:29, TURNER Aaron wrote: > Following on from Jonathan Buzzards comments, I'd also like to point out that I've never known a central DNS failure in a UK HEI for as long as I can remember, and it was certainly not my intention to suggest that as I think a central DNS issue is highly unlikely. And indeed, as I originally noted, the standard command-line tools on the nodes resolve the names as expected, so whatever is going on looks like it affects GPFS only. It may even be that the repetition of the domain names in the logs is just a function of something it is doing when logging when a node is failing to connect for some other reason entirely. It's just not something I recall having seen before and wanted to see if anyone else had seen it. > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard > *Sent:* 09 May 2020 23:22 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* Re: [gpfsug-discuss] Odd networking/name resolution issue > On 09/05/2020 12:06, Jaime Pinto wrote: >> DNS shouldn't be relied upon on a GPFS cluster for internal >> communication/management or data. >> > > The 1980's have called and want their lack of IP resolution protocols > back :-) > > I would kindly disagree. If your DNS is not working then your cluster is > fubar anyway and a zillion other things will also break very rapidly. > For us at least half of the running jobs would be dead in a few minutes > as failure to contact license servers would cause the software to stop. > All authentication and account lookup is also going to fail as well. > > You could distribute a hosts file but frankly outside of a storage only > cluster (as opposed to one with hundreds if not thousands of compute > nodes) that is frankly madness and will inevitably come to bite you in > the ass because they *will* get out of sync. The only hosts entry we > have is for the Salt Stack host because it tries to do things before the > DNS resolvers have been setup and consequently breaks otherwise. Which > IMHO is duff on it's behalf. > > I would add I can't think of a time in the last 16 years where internal > DNS at any University I have worked at has stopped working for even one > millisecond. If DNS is that flaky at your institution then I suggest > sacking the people responsible for it's maintenance as being incompetent > twits. It is just such a vanishingly remote possibility that it's not > worth bothering about. Frankly a aircraft falling out the sky and > squishing your data centre seems more likely to me. > > Finally in a world of IPv6 then anything other than DNS is a utter > madness IMHO. > > > JAB. > > -- > Jonathan A. Buzzard???????????????????????? Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=EXL-jEd1jmdzvOIhT87C7SIqmAS9uhVQ6J3kObct4OY&m=celJq_mDruJ5qISvmV3mGvLXYvIVcOQlhK3uohc8ZWI&s=31S9Rqbtlv7T0AqAEsLWo8BMPRuHOi9z29DLUYW_PAs&e= > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=EXL-jEd1jmdzvOIhT87C7SIqmAS9uhVQ6J3kObct4OY&m=celJq_mDruJ5qISvmV3mGvLXYvIVcOQlhK3uohc8ZWI&s=31S9Rqbtlv7T0AqAEsLWo8BMPRuHOi9z29DLUYW_PAs&e= > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=EXL-jEd1jmdzvOIhT87C7SIqmAS9uhVQ6J3kObct4OY&m=celJq_mDruJ5qISvmV3mGvLXYvIVcOQlhK3uohc8ZWI&s=31S9Rqbtlv7T0AqAEsLWo8BMPRuHOi9z29DLUYW_PAs&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From jcatana at gmail.com Sun May 10 20:09:39 2020 From: jcatana at gmail.com (Josh Catana) Date: Sun, 10 May 2020 15:09:39 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: I've seen odd behavior like this before and it is to do with name resolution. It might be from your local /etc/hosts entries or potentially the names used to add the nodes to the cluster or even if you are using DNS aliases that are configured improperly. In my case someone added DNS aliases to use in our cluster with fqdn instead of shortname which caused the triple append to appear in the logs you mentioned. I don't think it hurts anything since GPFS has its own name-to-ip table, but you probably want to track it down and fix it to be safe. On Sun, May 10, 2020, 2:31 PM RICHARD RUPP wrote: > *Normally the DNS server publishes a TTL and the client side caches the > info until the TTL expires. Could the server side be mis-configured for a > very short TTL?* > > > Regards, > > *Richard Rupp*, Sales Specialist, *Phone:* *1-347-510-6746* > > > [image: Inactive hide details for Jaime Pinto ---05/10/2020 09:28:46 > AM---The rationale for my suggestion doesn't have much to do with]Jaime > Pinto ---05/10/2020 09:28:46 AM---The rationale for my suggestion doesn't > have much to do with the central DNS server, but everything > > From: Jaime Pinto > To: gpfsug main discussion list , > TURNER Aaron > Date: 05/10/2020 09:28 AM > Subject: [EXTERNAL] Re: [gpfsug-discuss] Odd networking/name resolution > issue > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > The rationale for my suggestion doesn't have much to do with the central > DNS server, but everything to do with the DNS client side of the service. > If you have a very busy cluster at times, and a number of nodes really > busy with 1000+ IOPs for instance, so much that the OS on the client can't > barely spare a cycle to query the DSN server on what the IP associated with > the name of interface leading to the GPFS infrastructure is, or even > process that response when it returns, on the same interface where it's > having contentions and trying to process all the gpfs data transactions, > you can have temporary catch 22 situations. This can generate a backlog of > waiters, and eventual expelling of some nodes when the cluster managers > don't hear from them in reasonable time. > > It's doesn't really matter if you have a central DNS server in steroids. > > Jaime > > On 5/10/2020 03:35:29, TURNER Aaron wrote: > > Following on from Jonathan Buzzards comments, I'd also like to point out > that I've never known a central DNS failure in a UK HEI for as long as I > can remember, and it was certainly not my intention to suggest that as I > think a central DNS issue is highly unlikely. And indeed, as I originally > noted, the standard command-line tools on the nodes resolve the names as > expected, so whatever is going on looks like it affects GPFS only. It may > even be that the repetition of the domain names in the logs is just a > function of something it is doing when logging when a node is failing to > connect for some other reason entirely. It's just not something I recall > having seen before and wanted to see if anyone else had seen it. > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > *From:* gpfsug-discuss-bounces at spectrumscale.org < > gpfsug-discuss-bounces at spectrumscale.org> on behalf of Jonathan Buzzard < > jonathan.buzzard at strath.ac.uk> > > *Sent:* 09 May 2020 23:22 > > *To:* gpfsug-discuss at spectrumscale.org > > > *Subject:* Re: [gpfsug-discuss] Odd networking/name resolution issue > > On 09/05/2020 12:06, Jaime Pinto wrote: > >> DNS shouldn't be relied upon on a GPFS cluster for internal > >> communication/management or data. > >> > > > > The 1980's have called and want their lack of IP resolution protocols > > back :-) > > > > I would kindly disagree. If your DNS is not working then your cluster is > > fubar anyway and a zillion other things will also break very rapidly. > > For us at least half of the running jobs would be dead in a few minutes > > as failure to contact license servers would cause the software to stop. > > All authentication and account lookup is also going to fail as well. > > > > You could distribute a hosts file but frankly outside of a storage only > > cluster (as opposed to one with hundreds if not thousands of compute > > nodes) that is frankly madness and will inevitably come to bite you in > > the ass because they *will* get out of sync. The only hosts entry we > > have is for the Salt Stack host because it tries to do things before the > > DNS resolvers have been setup and consequently breaks otherwise. Which > > IMHO is duff on it's behalf. > > > > I would add I can't think of a time in the last 16 years where internal > > DNS at any University I have worked at has stopped working for even one > > millisecond. If DNS is that flaky at your institution then I suggest > > sacking the people responsible for it's maintenance as being incompetent > > twits. It is just such a vanishingly remote possibility that it's not > > worth bothering about. Frankly a aircraft falling out the sky and > > squishing your data centre seems more likely to me. > > > > Finally in a world of IPv6 then anything other than DNS is a utter > > madness IMHO. > > > > > > JAB. > > > > -- > > Jonathan A. Buzzard Tel: +44141-5483420 > > HPC System Administrator, ARCHIE-WeSt. > > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From knop at us.ibm.com Mon May 11 03:54:13 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 11 May 2020 02:54:13 +0000 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon May 11 11:01:49 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 11 May 2020 11:01:49 +0100 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: <1bc652e7-9c81-82ed-a962-23c24a5c83b2@strath.ac.uk> On 10/05/2020 14:28, Jaime Pinto wrote: > The rationale for my suggestion doesn't have much to do with the central > DNS server, but everything to do with the DNS client side of the service. > If you have a very busy cluster at times, and a number of nodes really > busy with 1000+ IOPs for instance, so much that the OS on the client > can't barely spare a cycle to query the DSN server on what the IP > associated with the name of interface leading to the GPFS infrastructure > is, or even process that response when it returns, on the same interface > where it's having contentions and trying to process all the gpfs data > transactions, you can have temporary catch 22 situations. This can > generate a backlog of waiters, and eventual expelling of some nodes when > the cluster managers don't hear from them in reasonable time. > > It's doesn't really matter if you have a central DNS server in steroids. > If that is the scenario it will struggle to look up the DNS to IP lists in /etc/hosts. GPFS itself will struggle to get scheduled. Besides which systemd is caching it all locally anyways which will get hit *before* /etc/hosts does. In an none systemd install then most likely nscd is running which provides similar service. You don't appear to a full grasp of how it hostname to IP resolution works. It is a outdated notion to suggest using a system that was deprecated over 30 years ago that needs stomping on because it comes from IMHO a lack of seeing the whole picture. Finally as I understand it GPFS maintains it own hostname to IP lookup table which makes it a completely moot point. You can see this from the fact you cannot change the IP address of a node in a cluster. You must remove it and then rejoin with the new IP address. If it was storing client information as hostnames and using hostname to IP resolution to work out the IP addresses that would not be the case. As such your suggestion is dreaming up a solution to a none existent problem, which makes it an even worse idea IMHO. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From TROPPENS at de.ibm.com Tue May 12 13:41:00 2020 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Tue, 12 May 2020 12:41:00 +0000 Subject: [gpfsug-discuss] THINK Virtual User Group Day In-Reply-To: <5BE5B210-5FEE-45E0-AC0D-1B184B5B8E45@spectrumscale.org> References: <5BE5B210-5FEE-45E0-AC0D-1B184B5B8E45@spectrumscale.org> Message-ID: An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Thu May 14 13:20:57 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Thu, 14 May 2020 12:20:57 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Message-ID: Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_2020152979] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From dean.flanders at fmi.ch Thu May 14 13:31:06 2020 From: dean.flanders at fmi.ch (Flanders, Dean) Date: Thu, 14 May 2020 12:31:06 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: References: Message-ID: Hello, It is great, that RHEL 7.8 is supported on SS 4.2.3.22, when will RHEL 8.x be supported on GPFS SS 4.2.3.X?? Thanks, Dean From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Carl Zetie - carlz at us.ibm.com Sent: Thursday, May 14, 2020 2:21 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_2020152979] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 13822 bytes Desc: image002.png URL: From Dugan.Witherick at warwick.ac.uk Thu May 14 14:03:17 2020 From: Dugan.Witherick at warwick.ac.uk (Witherick, Dugan) Date: Thu, 14 May 2020 13:03:17 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: References: Message-ID: <69508fba4e506dfed64703801f3756cfd29e8f1c.camel@warwick.ac.uk> Thanks Carl! Best regards, Dugan On Thu, 2020-05-14 at 12:20 +0000, Carl Zetie - carlz at us.ibm.com wrote: Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_2020152979] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From jonathan.buzzard at strath.ac.uk Thu May 14 15:22:25 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 14 May 2020 15:22:25 +0100 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: References: Message-ID: <0a7278de-33c6-ad20-d818-f3bbf5d19bcb@strath.ac.uk> On 14/05/2020 13:31, Flanders, Dean wrote: > Hello, It is great, that RHEL 7.8 is supported on SS 4.2.3.22, when will > RHEL 8.x be supported on GPFS SS 4.2.3.X?? Thanks, Dean That would be never, 4.2.3 goes out of support in September. Is 5.x supported in 7.8 yet? Some urgent upgrading of systems being looked into right now and I would prefer to jump to 5.x than 4.2.3.22 and have to upgrade again. However needs must. Oh and if you haven't yet I would recommend any HPC site revoking all your users SSH keys immediately. To many users have been creating SSH keys without passphrases and jumping from system to system. Several sites in the UK are currently down and I understand it has effected Germany too :-( There are zero guesses to the origin of the hacks. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From stockf at us.ibm.com Thu May 14 15:55:04 2020 From: stockf at us.ibm.com (Frederick Stock) Date: Thu, 14 May 2020 14:55:04 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: <0a7278de-33c6-ad20-d818-f3bbf5d19bcb@strath.ac.uk> References: <0a7278de-33c6-ad20-d818-f3bbf5d19bcb@strath.ac.uk>, Message-ID: An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Fri May 15 13:07:27 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Fri, 15 May 2020 12:07:27 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Message-ID: <04A3D7D0-749D-4B5C-A6EC-117CA02B003E@us.ibm.com> JAB> Is 5.x supported in 7.8 yet? Support is planned for 5.0.5 later this month, subject to final regression testing now in progress Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_1898525102] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From carlz at us.ibm.com Mon May 18 13:06:55 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Mon, 18 May 2020 12:06:55 +0000 Subject: [gpfsug-discuss] Scale and Linux distros FAQ improvement Message-ID: Inspired by the RHEL 7.8 compatibility issues (and some earlier incidents too), we have adopted a new practice for the Scale FAQ where we will list new distro releases explicitly with a note indicating if they are not yet supported. In the past we should say things like ?7.7.xxx or later? for the supported release level, and not say anything at all about new OS releases until they were supported. That would cause confusion when 7.8 came along, since 7.8 is definitely later than 7.7.xxx... Hopefully this small change will help you with your admin. Regards, Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_565114115] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From b.cregan at imperial.ac.uk Mon May 18 13:59:09 2020 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Mon, 18 May 2020 12:59:09 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues Message-ID: Hi, At Imperial we have been experiencing an issue with SS 5.0.x and quotas. The main symptom is a slow decay in the accuracy of reported quota usage when compared to the actual usage as reported by "du". This discrepancy can be as little as a few percent and as much as many X100% . We also sometimes see bizarre effects such negative file number counts being reported. We have been working with IBM and have put in the latest 5.0.4-4 (that has a fix for mmcheckquota) that we have been pinning our hopes on, but this has not worked. Is anyone else experiencing similar issues? We need to try and get an idea if this is an issue peculiar to our set up or a more general SS problem. We are using user and group quotas in a fileset context. Thanks Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From pinto at scinet.utoronto.ca Mon May 18 14:55:47 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 18 May 2020 09:55:47 -0400 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: References: Message-ID: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> So Bob, Yes, we too have observed an uncharacteristic lag on the correction of the internal quota accounting on GPFS since we updated from version 3.3 to version 4.x some 7-8 years ago. That lag remains through version 5.0.x as well. And it persisted through several appliances (DDN, G200, GSS, ESS and now DSS-G). In our university environment there is also a lot of data churning, in particular small files. The workaround has always been to periodically run mmcheckquota on the top independent fileset to expedite that correction (I have a crontab script that measures the relative size of the in-dought columns on the mmrepquota report, size and inodes, and if either exceeds 2% for any USR/GRP/FILESET I run mmrepquota) We have opened supports calls with IBM about this issue in the past, and we never got a solution, to possibly adjust some GPFS configuration parameter, and have this correction done automatically. We gave up. And that begs the question: what do you mean by "... 5.0.4-4 ... that has a fix for mmcheckquota"? Isn't mmcheckquota zeroing the in-doubt columns when you run it? The fix should be for gpfs (something buried in the code over many versions). As far as I can tell there has never been anything wrong with mmcheckquota. Thanks Jaime On 5/18/2020 08:59:09, Cregan, Bob wrote: > Hi, > ? ? ? At Imperial we? have been experiencing an issue with SS 5.0.x and quotas. The main symptom is a slow decay in the accuracy of reported quota usage when compared to the actual usage as reported by "du". This discrepancy can be as little as a few percent and as much as many? X100% . We also sometimes see bizarre effects such negative file number counts being reported. > > We have been working with IBM? and have put in the latest 5.0.4-4 (that has a fix for mmcheckquota) that we have been pinning our hopes on, but this has not worked. > > Is anyone else experiencing similar issues? We need to try and get an idea if this is an issue peculiar to our set up or a more general SS problem. > > We are using user and group quotas in a fileset context. > > Thanks > > > *Bob Cregan* > HPC Systems Analyst > > Information & Communication Technologies > > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > > W: www.imperial.ac.uk/ict/rcs > > _1505984389175_twitter.png @imperialRCS @imperialRSE > > 1505983882959_Imperial-RCS.png > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > . . . ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From knop at us.ibm.com Mon May 18 17:58:06 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 18 May 2020 16:58:06 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> References: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca>, Message-ID: An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Mon May 18 18:34:06 2020 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Mon, 18 May 2020 17:34:06 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: References: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> Message-ID: Is there a simple way to tell if we?re currently being affected by this bug? We run 5.0.4-1 on the client side and 5.0.3-2.3 on the server side (DSS-G 2.4b I believe it is). -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' > On May 18, 2020, at 12:58 PM, Felipe Knop wrote: > > All, > > Likely not an answer to these situations, but a fix was introduced in 5.0.4.4 (APAR IJ22894) and 4.2.3.22 (APAR IJ24661) to address a (long-standing) problem where mmcheckquota would compute quotas incorrectly in file systems where metadata pool and data pool subblocksizes are different. > > Felipe > > ---- > Felipe Knop knop at us.ibm.com > GPFS Development and Security > IBM Systems > IBM Building 008 > 2455 South Rd, Poughkeepsie, NY 12601 > (845) 433-9314 T/L 293-9314 > > > > ----- Original message ----- > From: Jaime Pinto > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] SS 5.0.x and quota issues > Date: Mon, May 18, 2020 9:56 AM > > So Bob, > Yes, we too have observed an uncharacteristic lag on the correction of the internal quota accounting on GPFS since we updated from version 3.3 to version 4.x some 7-8 years ago. That lag remains through version 5.0.x as well. And it persisted through several appliances (DDN, G200, GSS, ESS and now DSS-G). In our university environment there is also a lot of data churning, in particular small files. > > The workaround has always been to periodically run mmcheckquota on the top independent fileset to expedite that correction (I have a crontab script that measures the relative size of the in-dought columns on the mmrepquota report, size and inodes, and if either exceeds 2% for any USR/GRP/FILESET I run mmrepquota) > > We have opened supports calls with IBM about this issue in the past, and we never got a solution, to possibly adjust some GPFS configuration parameter, and have this correction done automatically. We gave up. > > And that begs the question: what do you mean by "... 5.0.4-4 ... that has a fix for mmcheckquota"? Isn't mmcheckquota zeroing the in-doubt columns when you run it? > > The fix should be for gpfs (something buried in the code over many versions). As far as I can tell there has never been anything wrong with mmcheckquota. > > Thanks > Jaime > > > On 5/18/2020 08:59:09, Cregan, Bob wrote: > > Hi, > > At Imperial we have been experiencing an issue with SS 5.0.x and quotas. The main symptom is a slow decay in the accuracy of reported quota usage when compared to the actual usage as reported by "du". This discrepancy can be as little as a few percent and as much as many X100% . We also sometimes see bizarre effects such negative file number counts being reported. > > > > We have been working with IBM and have put in the latest 5.0.4-4 (that has a fix for mmcheckquota) that we have been pinning our hopes on, but this has not worked. > > > > Is anyone else experiencing similar issues? We need to try and get an idea if this is an issue peculiar to our set up or a more general SS problem. > > > > We are using user and group quotas in a fileset context. > > > > Thanks > > > > > > *Bob Cregan* > > HPC Systems Analyst > > > > Information & Communication Technologies > > > > Imperial College London, > > South Kensington Campus London, SW7 2AZ > > T: 07712388129 > > E: b.cregan at imperial.ac.uk > > > > W: www.imperial.ac.uk/ict/rcs > > > > _1505984389175_twitter.png @imperialRCS @imperialRSE > > > > 1505983882959_Imperial-RCS.png > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > . > . > . ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto - Storage Analyst > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From knop at us.ibm.com Mon May 18 21:05:07 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 18 May 2020 20:05:07 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: References: , <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> Message-ID: An HTML attachment was scrubbed... URL: From juergen.hannappel at desy.de Tue May 19 15:30:34 2020 From: juergen.hannappel at desy.de (Hannappel, Juergen) Date: Tue, 19 May 2020 16:30:34 +0200 (CEST) Subject: [gpfsug-discuss] gpfs_fcntl fails on read-only-fs Message-ID: <1008080204.14431165.1589898634575.JavaMail.zimbra@desy.de> Hi, I tried to do gpfs_fcntl with the GPFS_ACCESS_RANGE hint, with the isWrite field set to 0 and start = 0, lenght = size of the file to indicate that I want to read the entire file now, but on a readonly remote file system the gpfs_fcntl() call returns -1 and sets errno to "Read-only file system" whic is true but should not matter in my Opinion. Why is announcing a range of the file for reading not allowed on a read onoy file system? -- Dr. J?rgen Hannappel DESY/IT Tel. : +49 40 8998-4616 From carlz at us.ibm.com Wed May 20 14:05:11 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Wed, 20 May 2020 13:05:11 +0000 Subject: [gpfsug-discuss] Are you using rsync? Message-ID: Folks, I?m looking for a handful of volunteers who are using rsync with Scale and willing to jump on a 15 minute call (separate calls for each volunteer, to be clear) to discuss how you are using rsync, what features are essential to you, what its shortcomings are, and anything else you want to share. And if you?re *not* using rsync because of some specific limitation but would like to, I?m also interested in hearing from you. And finally: if you have thoughts on this topic but don?t want to get on the phone, please feel free to email me. Please respond OFF LIST to the email address below. Thanks, Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_74924333] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From Ivano.Talamo at psi.ch Fri May 22 08:47:45 2020 From: Ivano.Talamo at psi.ch (Talamo Ivano Giuseppe (PSI)) Date: Fri, 22 May 2020 07:47:45 +0000 Subject: [gpfsug-discuss] selinux context Message-ID: Hi all, I?m configuring a set of login nodes with home directories in GPFS (but not on /home), with SElinux in enforcing mode and auto creation of home directory (via PAM). I?ve been able to partially achieve my target, by basically running the two following commands: semanage fcontext -a -e /home /das/home restorecon -v /das/home After having done this on one node, the context on the directory is the expected one (system_u:object_r:home_root_t:s0). And everything works as expected (a new user logs in and his directory is created). But on all the other nodes of the cluster still the old context is shown (system_u:object_r:unlabeled_t:s0). Unless I run the restorecon on them too. Furthermore, since the filesystem is a remote-cluster mount, on all the nodes on the central (storage) cluster, the corrent (home_root_t) context is shown. I was expecting the SElinux context to be stored in the inodes, but now the situation looks mixed and I?m puzzled. In case it can help, the login nodes are RHEL 7.7 with Spectrum Scale 5.0.4. The storage is RHEL 7.6 with 5.0.3. Does someone have any experience/idea? Thanks, __________________________________________ Paul Scherrer Institut Ivano Talamo WHGA/038 Forschungsstrasse 111 5232 Villigen PSI Schweiz Telefon: +41 56 310 47 71 E-Mail: ivano.talamo at psi.ch From lists at esquad.de Fri May 22 19:22:51 2020 From: lists at esquad.de (Dieter Mosbach) Date: Fri, 22 May 2020 20:22:51 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) Message-ID: From valdis.kletnieks at vt.edu Sun May 24 09:42:13 2020 From: valdis.kletnieks at vt.edu (Valdis Kl=?utf-8?Q?=c4=93?=tnieks) Date: Sun, 24 May 2020 04:42:13 -0400 Subject: [gpfsug-discuss] selinux context In-Reply-To: References: Message-ID: <34924.1590309733@turing-police> On Fri, 22 May 2020 07:47:45 -0000, "Talamo Ivano Giuseppe (PSI)" said: > After having done this on one node, the context on the directory is the expec > expected one (system_u:object_r:home_root_t:s0). And everything works as expected (a > new user logs in and his directory is created). > But on all the other nodes of the cluster still the old context is shown > (system_u:object_r:unlabeled_t:s0). Unless I run the restorecon on them too. > Furthermore, since the filesystem is a remote-cluster mount, on all the nodes > on the central (storage) cluster, the corrent (home_root_t) context is shown. > I was expecting the SElinux context to be stored in the inodes, but now the > situation looks mixed and I???m puzzled. I suspect the issue is that the other nodes have that inode cached already, and they don't find out that that the SELinux context has been changed. I can't tell from here from whether GPFS is failing to realize that a context change means the old inode is stale just like any other inode change, or if there's something else that has gone astray. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From scottg at emailhosting.com Sun May 24 14:57:57 2020 From: scottg at emailhosting.com (scott) Date: Sun, 24 May 2020 09:57:57 -0400 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: References: Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Is there a bug-fix list? we cant seem to find it - we can just find the new/change features. On 5/22/20 2:22 PM, Dieter Mosbach wrote: > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Achim.Rehor at de.ibm.com Sun May 24 16:58:35 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Sun, 24 May 2020 17:58:35 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Message-ID: An HTML attachment was scrubbed... URL: From Achim.Rehor at de.ibm.com Mon May 25 08:48:42 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Mon, 25 May 2020 09:48:42 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Message-ID: no, as this is a .0 package .. there are no 'bug-fixes' ;) you will see this again, when PTF 1 is announced. Mit freundlichen Gr??en / Kind regards Achim Rehor gpfsug-discuss-bounces at spectrumscale.org wrote on 24/05/2020 15:57:57: > From: scott > To: gpfsug-discuss at spectrumscale.org > Date: 24/05/2020 16:04 > Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is > available on FixCentral (n/t) > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Is there a bug-fix list? we cant seem to find it - we can just find the > new/change features. > > On 5/22/20 2:22 PM, Dieter Mosbach wrote: > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=RGTETs2tk0Kz_VOpznDVDkqChhnfLapOTkxLvgmR2- > M&m=P0Br9hWVFau3jYGAI4PCmhGihPTI0UNbIc- > S68rfjy0&s=FeqsNGo3OsZ9n7DfC_Rrx5MyiXeG3hO7oR3YwhWuU8s&e= > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=RGTETs2tk0Kz_VOpznDVDkqChhnfLapOTkxLvgmR2- > M&m=P0Br9hWVFau3jYGAI4PCmhGihPTI0UNbIc- > S68rfjy0&s=FeqsNGo3OsZ9n7DfC_Rrx5MyiXeG3hO7oR3YwhWuU8s&e= > From jonathan.buzzard at strath.ac.uk Mon May 25 19:23:07 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 25 May 2020 19:23:07 +0100 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Message-ID: <06948041-50e5-4150-083d-cdcf52f27515@strath.ac.uk> On 24/05/2020 16:58, Achim Rehor wrote: > > no, as this is a .0 package .. there are no 'bug-fixes' ;) ?you will see > this again, when PTF 1 is announced. So you are saying that no bugs that existed in 5.0.4.x have been fixed in 5.0.5.0? That has the credibility of Dominic Cummings driving to Barnard Castle to "test his eyesight". JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From S.J.Thompson at bham.ac.uk Tue May 26 12:59:30 2020 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Tue, 26 May 2020 11:59:30 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: <06948041-50e5-4150-083d-cdcf52f27515@strath.ac.uk> References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> <06948041-50e5-4150-083d-cdcf52f27515@strath.ac.uk> Message-ID: <35762876-A525-4CD9-8C91-A1CE5DC05135@bham.ac.uk> Well there is an interesting question there ... Where would you start counting the "bugs" from? 5.0.4.0? or 0.0? Or 5.0.4.4? So I think there is a valid point somewhere in there __ Simon ?On 25/05/2020, 19:23, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard" wrote: On 24/05/2020 16:58, Achim Rehor wrote: > > no, as this is a .0 package .. there are no 'bug-fixes' ;) you will see > this again, when PTF 1 is announced. So you are saying that no bugs that existed in 5.0.4.x have been fixed in 5.0.5.0? That has the credibility of Dominic Cummings driving to Barnard Castle to "test his eyesight". JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From carlz at us.ibm.com Tue May 26 14:44:45 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Tue, 26 May 2020 13:44:45 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral Message-ID: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Achim, I think the request here (lost in translation?) is for a list of the bugs that 5.0.5.0 addresses. And we're currently looking to see if we can generate that list. Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com ?On 5/25/20, 7:00 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=obB2s7QQTgU9QMn1708Vpg&m=vmAWHys4rqj2tGjdDeGEv0DQVLBf8oN-bBlHJc23SQ0&s=G30K6mkoWpJkWBqltaiUM49Bn_RWnK2Ke9VTThHnUeo&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) (scott) 2. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) (Achim Rehor) 3. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) (Achim Rehor) ---------------------------------------------------------------------- Message: 1 Date: Sun, 24 May 2020 09:57:57 -0400 From: scott To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b at emailhosting.com> Content-Type: text/plain; charset=utf-8; format=flowed Is there a bug-fix list? we cant seem to find it - we can just find the new/change features. From carlz at us.ibm.com Tue May 26 14:47:05 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Tue, 26 May 2020 13:47:05 +0000 Subject: [gpfsug-discuss] Follow up on Scale and rsync Message-ID: Folks, I want to thank everybody who responded to my request about rsync usage with Scale. I got many more replies than I expected, and several of you went ahead and provided information without even waiting for a phone call. Consequently I have a lot of information to read through and digest, and will probably be rewriting my questions as a result, so it?s going to take me a little longer to get back to those of you who kindly offered your time to discuss the topic. However, the first takeaway is that clearly, a lot of you are heavily dependent on rsync. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_1890439710] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From janfrode at tanso.net Tue May 26 15:51:27 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Tue, 26 May 2020 16:51:27 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral In-Reply-To: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> References: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Message-ID: Seeing that as a %changelog in the RPMs would be fantastic.. :-) -jf tir. 26. mai 2020 kl. 15:44 skrev Carl Zetie - carlz at us.ibm.com < carlz at us.ibm.com>: > Achim, I think the request here (lost in translation?) is for a list of > the bugs that 5.0.5.0 addresses. And we're currently looking to see if we > can generate that list. > > > > Carl Zetie > Program Director > Offering Management > Spectrum Scale > ---- > (919) 473 3318 ][ Research Triangle Park > carlz at us.ibm.com > > > ?On 5/25/20, 7:00 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of gpfsug-discuss-request at spectrumscale.org" < > gpfsug-discuss-bounces at spectrumscale.org on behalf of > gpfsug-discuss-request at spectrumscale.org> wrote: > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=obB2s7QQTgU9QMn1708Vpg&m=vmAWHys4rqj2tGjdDeGEv0DQVLBf8oN-bBlHJc23SQ0&s=G30K6mkoWpJkWBqltaiUM49Bn_RWnK2Ke9VTThHnUeo&e= > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (scott) > 2. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > 3. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 24 May 2020 09:57:57 -0400 > From: scott > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on > FixCentral (n/t) > Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b at emailhosting.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > Is there a bug-fix list? we cant seem to find it - we can just find > the > new/change features. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Achim.Rehor at de.ibm.com Tue May 26 16:04:17 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Tue, 26 May 2020 17:04:17 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral In-Reply-To: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> References: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Message-ID: Yes, Carl, i guess that's what he is looking for ... but Simon has a valid point. Against what would we count bug fixes, as it is a .0 release ? ;) I would guess that's why we never did list "Problems fixed in ..." in the README of the .0 releases. Changes are documented, but these are feature changes new to 5.0.5 (as opposed to 5.0.4) Mit freundlichen Gr??en / Kind regards Achim Rehor Technical Support Specialist Spectrum Scale and ESS (SME) Advisory Product Services Professional IBM Systems Storage Support - EMEA gpfsug-discuss-bounces at spectrumscale.org wrote on 26/05/2020 15:44:45: > From: "Carl Zetie - carlz at us.ibm.com" > To: "gpfsug-discuss at spectrumscale.org" > Date: 26/05/2020 15:44 > Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is > available on FixCentral > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Achim, I think the request here (lost in translation?) is for a list > of the bugs that 5.0.5.0 addresses. And we're currently looking to > see if we can generate that list. > > > > Carl Zetie > Program Director > Offering Management > Spectrum Scale > ---- > (919) 473 3318 ][ Research Triangle Park > carlz at us.ibm.com > > > On 5/25/20, 7:00 AM, "gpfsug-discuss-bounces at spectrumscale.org on > behalf of gpfsug-discuss-request at spectrumscale.org" bounces at spectrumscale.org on behalf of gpfsug-discuss- > request at spectrumscale.org> wrote: > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=obB2s7QQTgU9QMn1708Vpg&m=vmAWHys4rqj2tGjdDeGEv0DQVLBf8oN- > bBlHJc23SQ0&s=G30K6mkoWpJkWBqltaiUM49Bn_RWnK2Ke9VTThHnUeo&e= > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (scott) > 2. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > 3. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 24 May 2020 09:57:57 -0400 > From: scott > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on > FixCentral (n/t) > Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b at emailhosting.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > Is there a bug-fix list? we cant seem to find it - we can just find the > new/change features. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=RGTETs2tk0Kz_VOpznDVDkqChhnfLapOTkxLvgmR2- > M&m=9yPfjB7JuhNahhwFwELNpS_PTi9-2RKvKXsoavY3mgs&s=- > JzPycQHt6vWBH_qWUEBBn8WISeK4Xbfxh4HRJ2zbsE&e= > From christian.vieser at 1und1.de Wed May 27 08:18:44 2020 From: christian.vieser at 1und1.de (Christian Vieser) Date: Wed, 27 May 2020 09:18:44 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral In-Reply-To: References: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Message-ID: <77257593-9ae5-8bf6-4aff-cf6a4ec3aca5@1und1.de> While these all are valid points, I had several support cases in the last months where I got feedback that the issue will be fixed in 5.0.5 or 5.1.0 . So there has to be a list of fixes even in a .0 release, that didn't made it to the last PTF of the previous release. Christian Achim Rehor wrote: > Yes, Carl, > > i guess that's what he is looking for ... but Simon has a valid point. > Against what would we count bug fixes, as it is a .0 release ? ;) > I would guess that's why we never did list "Problems fixed in ..." in the > README of the .0 releases. > Changes are documented, but these are feature changes new to 5.0.5 (as > opposed to 5.0.4) > > > Mit freundlichen Gr??en / Kind regards > > Achim Rehor > Simon Thompson wrote: > Well there is an interesting question there ... > > Where would you start counting the "bugs" from? > > 5.0.4.0? or 0.0? Or 5.0.4.4? > > So I think there is a valid point somewhere in there __ > > Simon -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3378 bytes Desc: S/MIME Cryptographic Signature URL: From heinrich.billich at id.ethz.ch Wed May 27 09:41:11 2020 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Wed, 27 May 2020 08:41:11 +0000 Subject: [gpfsug-discuss] Parse -Y command output Message-ID: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> Hello, I wonder if any python or bash functions exist to parse the output of mm-command's -Y format, i.e. colon-separated with HEADER rows. I would be nice to convert the output to a python list of named tuples or even better pandas dataframe. I would like to access the values by column name and row index. This would allow to do quick scripts or reports easily. Imagine using a jupyter notebook to dig into mmrepquota or mmlsdisk output, even doing charts, ..., easily. I know the ReST interface does exisit, which I could call to get the data in Json format, but to parse the Y-format seems much less cumbersome and allows to collect the data at one time and process it later. If you have other suggestions please let me know. What I would like to get: A list of NSDs which shows the connection to vdisk-set. Vdisk, recovery group and ESS system for each NSD. To get this I need to merge at least two tables. A list of user of fileset quota values highlighting those having non-default vaulues or close to the limit. The ability to get the data quickly in an uniform way into some tool, preferably python, to do analysis or formatting. Cheers, Heiner From juergen.hannappel at desy.de Wed May 27 10:04:00 2020 From: juergen.hannappel at desy.de (Hannappel, Juergen) Date: Wed, 27 May 2020 11:04:00 +0200 (CEST) Subject: [gpfsug-discuss] Parse -Y command output In-Reply-To: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> References: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> Message-ID: <1635082966.18343904.1590570240877.JavaMail.zimbra@desy.de> Hi, an example in python 2.7 (for Python 3 you need to add e.g. errors='ignore' to the parameters of the Popen call to get the proper kind of text stream as output. import subprocess import csv mmlsfileset = subprocess.Popen(['/usr/lpp/mmfs/bin/mmlsfileset', args.filesystem, '-Y'], stdout=subprocess.PIPE) reader = csv.DictReader(mmlsfileset.stdout, delimiter=':') filesets=[] for row in reader: if row['status'] == 'Linked': filesets.append({'name':row['filesetName'], 'linkdir':urllib.unquote(row['path'])}) mmlsfileset.wait() ----- Original Message ----- > From: "Billich Heinrich Rainer (ID SD)" > To: "gpfsug main discussion list" > Sent: Wednesday, 27 May, 2020 10:41:11 > Subject: [gpfsug-discuss] Parse -Y command output > Hello, > > I wonder if any python or bash functions exist to parse the output of > mm-command's -Y format, i.e. colon-separated with HEADER rows. I would be nice > to convert the output to a python list of named tuples or even better pandas > dataframe. I would like to access the values by column name and row index. > This would allow to do quick scripts or reports easily. Imagine using a > jupyter notebook to dig into mmrepquota or mmlsdisk output, even doing charts, > ..., easily. > > I know the ReST interface does exisit, which I could call to get the data in > Json format, but to parse the Y-format seems much less cumbersome and allows to > collect the data at one time and process it later. > > If you have other suggestions please let me know. What I would like to get: > A list of NSDs which shows the connection to vdisk-set. Vdisk, recovery group > and ESS system for each NSD. To get this I need to merge at least two tables. > A list of user of fileset quota values highlighting those having non-default > vaulues or close to the limit. > The ability to get the data quickly in an uniform way into some tool, preferably > python, to do analysis or formatting. > > Cheers, > > Heiner > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From achim.christ at de.ibm.com Wed May 27 10:08:26 2020 From: achim.christ at de.ibm.com (Achim Christ) Date: Wed, 27 May 2020 09:08:26 +0000 Subject: [gpfsug-discuss] Parse -Y command output In-Reply-To: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> References: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> Message-ID: An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Wed May 27 13:39:49 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Wed, 27 May 2020 12:39:49 +0000 Subject: [gpfsug-discuss] Scale release changelog (was: 5.0.5 available on Fix Central) Message-ID: <07847A5D-ED06-43B4-A69A-98A2B02787A3@us.ibm.com> > i guess that's what he is looking for ... but Simon has a valid point. > Against what would we count bug fixes, as it is a .0 release ? ;) The previous Mod level? But really, I'd like to know what the users think would be most useful to them. So I've created an RFE and invite people to submit their comments and votes there: https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=142683 Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com ? From prasad.surampudi at theatsgroup.com Thu May 28 23:31:04 2020 From: prasad.surampudi at theatsgroup.com (Prasad Surampudi) Date: Thu, 28 May 2020 22:31:04 +0000 Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> We have two scale clusters, cluster-A running version Scale 4.2.3 and RHEL6/7 and Cluster-B running Spectrum Scale 5.0.4 and RHEL 8.1. All the nodes in both Cluster-A and Cluster-B are direct attached and no NSD servers. We have our current filesystem gpfs_4 in Cluster-A and new filesystem gpfs_5 in Cluster-B. We want to copy all our data from gpfs_4 filesystem into gpfs_5 which has variable block size. So, can we map NSDs of gpfs_4 to Cluster-B nodes and do a mmexportfs of gpfs_4 from Cluster-A and mmimportfs into Cluster-B so that we have both filesystems available on same node in Cluster-B for copying data across fiber channel? If mmexportfs/mmimportfs works, can we delete nodes from Cluster-A and add them to Cluster-B without upgrading RHEL or GPFS versions for now and plan upgrading them at a later time? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjdoherty at yahoo.com Fri May 29 02:35:13 2020 From: jjdoherty at yahoo.com (Jim Doherty) Date: Fri, 29 May 2020 01:35:13 +0000 (UTC) Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster In-Reply-To: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> References: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> Message-ID: <102892274.973726.1590716113411@mail.yahoo.com> What is the minimum release level of the Spectrum Scale 5.0.4 cluster???? Is it 4.2.3.X?? Jim Doherty On Thursday, May 28, 2020, 6:31:21 PM EDT, Prasad Surampudi wrote: We have two scale clusters, cluster-A running version Scale 4.2.3 and RHEL6/7 and Cluster-B running Spectrum Scale ?5.0.4 and RHEL 8.1. All the nodes in both Cluster-A and Cluster-B are direct attached and no NSD servers. We have our current filesystem gpfs_4 in Cluster-A ?and new filesystem gpfs_5 in Cluster-B. We want to copy all our data from gpfs_4 filesystem into gpfs_5 which has variable block size.? So, can we map NSDs of gpfs_4 to Cluster-B nodes and do a mmexportfs of gpfs_4 from Cluster-A and mmimportfs into Cluster-B so that we have both filesystems available on same node in Cluster-B for copying data across fiber channel? If mmexportfs/mmimportfs works, can we delete nodes from Cluster-A and add them to Cluster-B without upgrading RHEL or GPFS versions for now and ?plan upgrading them at a later time? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Fri May 29 08:22:52 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 29 May 2020 08:22:52 +0100 Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster In-Reply-To: <102892274.973726.1590716113411@mail.yahoo.com> References: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> <102892274.973726.1590716113411@mail.yahoo.com> Message-ID: On 29/05/2020 02:35, Jim Doherty wrote: > > What is the minimum release level of the Spectrum Scale 5.0.4 > cluster???? Is it 4.2.3.X? > According to the email Cluster-B (the one with 5.0.4) has the variable block size thing, so that is going to be a no. Besides which even if you could mmimport the file system into Cluster-B you can't "merge" the two file systems. You are going to need to use AFM, rsync or similar. Hopefull they have enough free space on gpfs_5 to copy the data from gpfs_4 :-) JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From carlz at us.ibm.com Fri May 29 13:25:46 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Fri, 29 May 2020 12:25:46 +0000 Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From prasad.surampudi at theatsgroup.com Fri May 29 16:02:12 2020 From: prasad.surampudi at theatsgroup.com (Prasad Surampudi) Date: Fri, 29 May 2020 15:02:12 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 In-Reply-To: References: Message-ID: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. ?On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** From ulmer at ulmer.org Fri May 29 20:55:01 2020 From: ulmer at ulmer.org (Stephen Ulmer) Date: Fri, 29 May 2020 15:55:01 -0400 Subject: [gpfsug-discuss] Multi-cluster question (was Re: gpfsug-discuss Digest, Vol 100, Issue 32) In-Reply-To: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: I have a question about multi-cluster, but it is related to this thread (it would be solving the same problem). Let?s say we have two clusters A and B, both clusters are normally shared-everything with no NSD servers defined. We want cluster B to be able to use a file system in cluster A. If I zone the SAN such that cluster B can see all of cluster A?s disks, can I then define a multi-cluster relationship between them and mount a file system from A on B? To state it another way, must B's I/O for the foreign file system pass though NSD servers in A, or can B?s nodes discover that they have FibreChannel paths to those disks and use them? I know that the nodes in B become sort-of ?casual" members of A, but I don?t know *how* casual the relationship is... Liberty, -- Stephen > On May 29, 2020, at 11:02 AM, Prasad Surampudi wrote: > > Jim, > The minimum release for 5.0.4 cluster is 5.0.4.3. > > We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. > > > ?On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 > cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 29 May 2020 12:25:46 +0000 > From: "Carl Zetie - carlz at us.ibm.com" > To: "gpfsug-discuss at spectrumscale.org" > > Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem > from 4.2.3 cluster to 5.0.4.3 cluster > Message-ID: > Content-Type: text/plain; charset="utf-8" > > On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. > > If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. > > Regards > > > > > Carl Zetie > Program Director > Offering Management > Spectrum Scale > ---- > (919) 473 3318 ][ Research Triangle Park > carlz at us.ibm.com > > [signature_398516855] > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: image001.png > Type: image/png > Size: 69558 bytes > Desc: image001.png > URL: > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 100, Issue 32 > *********************************************** > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Fri May 29 22:30:08 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 29 May 2020 22:30:08 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 In-Reply-To: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: <3597cc2c-47af-461f-af03-4f88069e1ca6@strath.ac.uk> On 29/05/2020 16:02, Prasad Surampudi wrote: > Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. > > We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. > Our plan is to import the gpfs_4 filesystem into the Spectrum Scale > 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters > nodes) and copy the data using rsync, across fibre channel instead > of ethernet to get higher bandwidth. > Ethernet goes *very* fast these days you know :-) In fact *much* faster than fibre channel. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From sadaniel at us.ibm.com Sat May 30 00:39:59 2020 From: sadaniel at us.ibm.com (Steven Daniels) Date: Fri, 29 May 2020 17:39:59 -0600 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 In-Reply-To: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: Prasad, One of the clients I work with have several 5.0.X ESS (owning) clusters and they share 4.2.3.9 file systems to remote client clusters. The minimum release level on the 5.0.X clusters are at the version of the cluster, i.e. 5.0.3 or 5.0.4 (we are in the process of getting all to latest). The key is that the clients are not part of this cluster as a node has to be > minReleaseLevel to be able to join. They are in their own (accessing) cluster which has a minReleaseLevel of 4.2.3-9. This is the current (until September withdrawal) minReleaseLevel that is supported in this scenario. Don't know specifically about import but it should work as long as the clients that need to mount the sub-5.0 filesystem can be in a separate (remote) cluster or the client nodes are upgraded to the minReleaseLevel of the local cluster. Thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 09:02 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Sent by: gpfsug-discuss-bounces at spectrumscale.org Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=u37vFxPtfEbuPwTW6wj7sbTFugW8l6DsRWtag_KO4Bk&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.png&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=YORJyUthJSKVhDxjwEwePTRexsjhX9F3sXPJg_Z5c5g&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: From prasad.surampudi at theatsgroup.com Sat May 30 00:55:05 2020 From: prasad.surampudi at theatsgroup.com (Prasad Surampudi) Date: Fri, 29 May 2020 23:55:05 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 34 In-Reply-To: References: Message-ID: <6DFF8E9C-EB6E-4A67-90E2-631E625F11BA@theatsgroup.com> Steve, So in essence, if the minReleaseLevel is set to 5.0.x.x for a Spectrum Scale 5.0 cluster, we can't add any new severs with Spectrum Scale 4.x.x RPMs installed without upgrading them to Spectrum Scale 5.x.x RPMs to the cluster. Is this a correct statement? Does mmiportfs also check the filesystem version which it is importing and prevent it from importing if it is created on Spectrum Scale 4.x.x? ?On 5/29/20, 7:40 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: gpfsug-discuss Digest, Vol 100, Issue 32 (Steven Daniels) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 17:39:59 -0600 From: "Steven Daniels" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Message-ID: Content-Type: text/plain; charset="iso-8859-1" Prasad, One of the clients I work with have several 5.0.X ESS (owning) clusters and they share 4.2.3.9 file systems to remote client clusters. The minimum release level on the 5.0.X clusters are at the version of the cluster, i.e. 5.0.3 or 5.0.4 (we are in the process of getting all to latest). The key is that the clients are not part of this cluster as a node has to be > minReleaseLevel to be able to join. They are in their own (accessing) cluster which has a minReleaseLevel of 4.2.3-9. This is the current (until September withdrawal) minReleaseLevel that is supported in this scenario. Don't know specifically about import but it should work as long as the clients that need to mount the sub-5.0 filesystem can be in a separate (remote) cluster or the client nodes are upgraded to the minReleaseLevel of the local cluster. Thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 09:02 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Sent by: gpfsug-discuss-bounces at spectrumscale.org Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=u37vFxPtfEbuPwTW6wj7sbTFugW8l6DsRWtag_KO4Bk&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.png&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=YORJyUthJSKVhDxjwEwePTRexsjhX9F3sXPJg_Z5c5g&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 100, Issue 34 *********************************************** From sadaniel at us.ibm.com Sat May 30 23:19:28 2020 From: sadaniel at us.ibm.com (Steven Daniels) Date: Sat, 30 May 2020 16:19:28 -0600 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 34 In-Reply-To: <6DFF8E9C-EB6E-4A67-90E2-631E625F11BA@theatsgroup.com> References: <6DFF8E9C-EB6E-4A67-90E2-631E625F11BA@theatsgroup.com> Message-ID: see below. Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 05:55 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 34 Sent by: gpfsug-discuss-bounces at spectrumscale.org Steve, So in essence, if the minReleaseLevel is set to 5.0.x.x for a Spectrum Scale 5.0 cluster, we can't add any new severs with Spectrum Scale 4.x.x RPMs installed without upgrading them to Spectrum Scale 5.x.x RPMs to the cluster. Is this a correct statement? Steve>>> This statement is true for the local cluster. However, it is pretty simple to put the 4.2.3-X client nodes(currently 4.2.3.21 is min officially recommended and only until Sep 2020) in their own access cluster. Does mmiportfs also check the filesystem version which it is importing and prevent it from importing if it is created on Spectrum Scale 4.x.x? I hope one of the FS experts can correct if wrong, but it would be in my mind a bug if mmimportfs didn't allow this. I know you can create new 4.2.3-X file systems in a cluster with 5.0.X minReleaseLevel and do that routinely. So, since the cluster supports lower version levels (actually I have one filesystem that is a 3.5.0.7 in a cluster that is 5.0.4.2 minReleaseLevel but obviously 3.5.0.7 wouldn't be anywhere near a recommended filesystm level but it does work fine. NOTE: that the remote cluster will not authenticate unless at a minReleaseLevel of 4.2.2.X (not a supported level) so even though the filesystem can be very back level. The remote accessing cluster has to be no more than one major version back or ahead. On 5/29/20, 7:40 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wzUqgJ_KdsODST1fL88GSy_nGwzeJhtx-ikgQdEt310&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: gpfsug-discuss Digest, Vol 100, Issue 32 (Steven Daniels) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 17:39:59 -0600 From: "Steven Daniels" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Message-ID: Content-Type: text/plain; charset="iso-8859-1" Prasad, One of the clients I work with have several 5.0.X ESS (owning) clusters and they share 4.2.3.9 file systems to remote client clusters. The minimum release level on the 5.0.X clusters are at the version of the cluster, i.e. 5.0.3 or 5.0.4 (we are in the process of getting all to latest). The key is that the clients are not part of this cluster as a node has to be > minReleaseLevel to be able to join. They are in their own (accessing) cluster which has a minReleaseLevel of 4.2.3-9. This is the current (until September withdrawal) minReleaseLevel that is supported in this scenario. Don't know specifically about import but it should work as long as the clients that need to mount the sub-5.0 filesystem can be in a separate (remote) cluster or the client nodes are upgraded to the minReleaseLevel of the local cluster. Thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 09:02 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Sent by: gpfsug-discuss-bounces at spectrumscale.org Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=u37vFxPtfEbuPwTW6wj7sbTFugW8l6DsRWtag_KO4Bk&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.png&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=YORJyUthJSKVhDxjwEwePTRexsjhX9F3sXPJg_Z5c5g&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_0c6df3be_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wo2skeF7H2QM7YNg9RJDB9ntSOGPeaocVSTahs6lCtc&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_0c6df3be_attachment.gif&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wvnV0A6CkiXRcWtU-nWQC_-Li0zXNfBZOW2oADBVg0o&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_0c6df3be_attachment.jpg&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=NcIKR17cT_NpUQ67OrX_fti7Tdnla2I-Q5Ptx82UP3k&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wzUqgJ_KdsODST1fL88GSy_nGwzeJhtx-ikgQdEt310&e= End of gpfsug-discuss Digest, Vol 100, Issue 34 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wzUqgJ_KdsODST1fL88GSy_nGwzeJhtx-ikgQdEt310&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: From jonathan.buzzard at strath.ac.uk Sun May 31 10:31:34 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sun, 31 May 2020 10:31:34 +0100 Subject: [gpfsug-discuss] Multi-cluster question (was Re: gpfsug-discuss Digest, Vol 100, Issue 32) In-Reply-To: References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: <53c17f6a-f3fe-9217-4000-197e1b06a105@strath.ac.uk> On 29/05/2020 20:55, Stephen Ulmer wrote: > I have a question about multi-cluster, but it is related to this thread > (it would be solving the same problem). > > Let?s say we have two clusters A and B, both clusters are normally > shared-everything with no NSD servers defined. Er, even in a shared-everything all nodes fibre channel attached you still have to define NSD servers. That is a given NSD has a server (or ideally a list of servers) that arbitrate the disk. Unless it has changed since 3.x days. Never run a 4.x or later with all the disks SAN attached on all the nodes. > We want cluster B to be > able to use a file system in cluster A. ?If I zone the SAN such that > cluster B can see all of cluster A?s disks, can I then define a > multi-cluster relationship between them and mount a file system from A on B? > > To state it another way, must B's I/O for the foreign file system pass > though NSD servers in A, or can B?s nodes discover that they have > FibreChannel paths to those disks and use them? > My understanding is that remote cluster mounts have to pass through the NSD servers. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From janfrode at tanso.net Sun May 31 17:47:40 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 31 May 2020 18:47:40 +0200 Subject: [gpfsug-discuss] Multi-cluster question (was Re: gpfsug-discuss Digest, Vol 100, Issue 32) In-Reply-To: <53c17f6a-f3fe-9217-4000-197e1b06a105@strath.ac.uk> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> <53c17f6a-f3fe-9217-4000-197e1b06a105@strath.ac.uk> Message-ID: No, this is a common misconception. You don?t need any NSD servers. NSD servers are only needed if you have nodes without direct block access. Remote cluster or not, disk access will be over local block device (without involving NSD servers in any way), or NSD server if local access isn?t available. NSD-servers are not ?arbitrators? over access to a disk, they?re just stupid proxies of IO commands. -jf s?n. 31. mai 2020 kl. 11:31 skrev Jonathan Buzzard < jonathan.buzzard at strath.ac.uk>: > On 29/05/2020 20:55, Stephen Ulmer wrote: > > I have a question about multi-cluster, but it is related to this thread > > (it would be solving the same problem). > > > > Let?s say we have two clusters A and B, both clusters are normally > > shared-everything with no NSD servers defined. > > Er, even in a shared-everything all nodes fibre channel attached you > still have to define NSD servers. That is a given NSD has a server (or > ideally a list of servers) that arbitrate the disk. Unless it has > changed since 3.x days. Never run a 4.x or later with all the disks SAN > attached on all the nodes. > > > We want cluster B to be > > able to use a file system in cluster A. If I zone the SAN such that > > cluster B can see all of cluster A?s disks, can I then define a > > multi-cluster relationship between them and mount a file system from A > on B? > > > > To state it another way, must B's I/O for the foreign file system pass > > though NSD servers in A, or can B?s nodes discover that they have > > FibreChannel paths to those disks and use them? > > > > My understanding is that remote cluster mounts have to pass through the > NSD servers. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Mon May 4 10:11:50 2020 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Mon, 4 May 2020 11:11:50 +0200 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? Message-ID: Hi all, I didn't see any updates in the faq yet about the new 5.0.4.4 release. Does this release support rhel 7.8 ? Thank you! Kenneth From jonathan.buzzard at strath.ac.uk Mon May 4 11:13:00 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 4 May 2020 11:13:00 +0100 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? In-Reply-To: References: Message-ID: <94ec87a7-64df-1623-37ef-81f2055c70db@strath.ac.uk> On 04/05/2020 10:11, Kenneth Waegeman wrote: > Hi all, > > I didn't see any updates in the faq yet about the new 5.0.4.4 release. > > Does this release support rhel 7.8 ? > Has the fix for 7.7 with a kernel >= 3.10.0-1062.18.1.el7 been released yet? If it hasn't then there is no chance of 7.8 working... JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From kenneth.waegeman at ugent.be Mon May 4 12:58:49 2020 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Mon, 4 May 2020 13:58:49 +0200 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? In-Reply-To: <94ec87a7-64df-1623-37ef-81f2055c70db@strath.ac.uk> References: <94ec87a7-64df-1623-37ef-81f2055c70db@strath.ac.uk> Message-ID: <531e4116-c196-a7b9-61bd-2a7adb98f376@ugent.be> On 04/05/2020 12:13, Jonathan Buzzard wrote: > On 04/05/2020 10:11, Kenneth Waegeman wrote: >> Hi all, >> >> I didn't see any updates in the faq yet about the new 5.0.4.4 release. >> >> Does this release support rhel 7.8 ? >> > > Has the fix for 7.7 with a kernel >= 3.10.0-1062.18.1.el7 been > released yet? If it hasn't then there is no chance of 7.8 working... Yes, this fix is in the release notes of 5.0.4.4: * Item: IJ24294 * Problem description: On RHEL 7.7 node, with supported GPFS versions 4.2.3.18 or higher and 5.0.4.0 or higher, when the kernel is upgraded to a version 3.10.0-1062.18.1 or higher, the node may encounter a kernel crash when accessing a deleted directory * Work around: None * Problem trigger: Accessing a directory which has been deleted * Symptom: Abend/Crash * Platforms affected: All RHEL 7.7 OS environments with kernel version equal or higher than 3.10.0-1062.18.1 * Functional Area affected: All * Customer Impact: High K > > JAB. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Mon May 4 13:19:59 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Mon, 4 May 2020 12:19:59 +0000 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? Message-ID: <497CD318-97FF-40D9-ACE4-D50D83D2B1CF@us.ibm.com> Right now the dev/test team is working flat-out to try to get RHEL 7.8 supported with the upcoming 5.0.5 release, expected this month. We haven't yet made a decision about 5.0.4.4. It will depend on what we learn about the changes required to support RHEL 7.8, and whether those are small enough to be worth backporting to 5.0.4.4. Bear in mind that 5.0.4.4 was the last PTF in the 5.0.4 stream, and customers who can update to 5.0.5 are advised to do so. 5.0.5 will be our first Extended Updates release, meaning you can stay there for up to two years while still receiving PTFs including security fixes. There is no additional charge for this: it's included in your existing S&S. As soon as I know more, I'll share it here. Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com From knop at us.ibm.com Mon May 4 14:16:22 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 4 May 2020 13:16:22 +0000 Subject: [gpfsug-discuss] support for 7.8 in 5.0.4.4? In-Reply-To: <497CD318-97FF-40D9-ACE4-D50D83D2B1CF@us.ibm.com> References: <497CD318-97FF-40D9-ACE4-D50D83D2B1CF@us.ibm.com> Message-ID: An HTML attachment was scrubbed... URL: From petr.plodik at mcomputers.cz Tue May 5 00:13:44 2020 From: petr.plodik at mcomputers.cz (=?iso-8859-1?Q?Petr_Plod=EDk?=) Date: Mon, 4 May 2020 23:13:44 +0000 Subject: [gpfsug-discuss] GPFS ECE IOPS scaling Message-ID: <8ec7eb20765b472aa5d900da9437983c@MCOMPEXCH01.mcomputers.local> Hi all, do you have any experiences / references with GPFS (ECE) scaling in terms of IOPS performance? We are looking for HPC scratch storage with 5M IOPS (4KiB block size, 80%/20% read/write) with two drives failure redundancy. Thank you! Petr Plodik From christian.vieser at 1und1.de Thu May 7 11:01:49 2020 From: christian.vieser at 1und1.de (Christian Vieser) Date: Thu, 7 May 2020 12:01:49 +0200 Subject: [gpfsug-discuss] Enabling SSL/HTTPS/ on Object S3. Message-ID: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> Hi Andi, up to now there are no instructions available on how to enable SSL on the Swift/S3 endpoints. The only thing is that you can enable SSL on the authentication path. So your connection to Swift authentication on port 35357 will be secured and the S3 authentication arriving at http port 8080 will internally take the SSL path, if configured properly. We have successfully done that in a test environment. Be sure to use the --pwd-file option with the "mmuserauth service create ..." and verify the proxy settings afterwards. It should look like this: # mmobj config list --ccrfile proxy-server.conf --section filter:s3token [filter:s3token] auth_uri = https://127.0.0.1:35357/ use = egg:swift3#s3token insecure = true You can correct wrong settings with # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property insecure --value true # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property auth_uri --value 'https://127.0.0.1:35357/' Regards, Christian > i have tried what you suggested. mmobj swift base ran fine. but after i have > deleted the userauth and try to set it up again with ks-ssl enabled it just > hangs: > > # mmuserauth service create --data-access-method object --type local > --enable-ks-ssl > > still waiting for it to finish, 15 mins now.. :) >> Basically all i need is this: >> >> https://s3.something.com:8080 https://s3.something.com:8080 which points >> to the WAN ip of the CES cluster (already configured and ready) >> >> and endpoints like this: >> >> None | keystone | identity | True | public | https://cluster_domain:5000/ >> https://cluster_domain:5000/ >> RegionOne | swift | object-store | True | public | >> https://cluster_domain:443/v1/AUTH_%(tenant_id)s >> RegionOne | swift | object-store | True | public | >> https://cluster_domain:8080/v1/AUTH_%(tenant_id)s -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3378 bytes Desc: S/MIME Cryptographic Signature URL: From andi at christiansen.xxx Thu May 7 13:59:43 2020 From: andi at christiansen.xxx (Andi Christiansen) Date: Thu, 7 May 2020 14:59:43 +0200 Subject: [gpfsug-discuss] Enabling SSL/HTTPS/ on Object S3. In-Reply-To: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> References: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> Message-ID: Hi Christian, Thanks for answering! We solved this with lab services a while back now, and ended up setting up haproxys I front of the ces nodes and then they handle the ssl encryption to the S3 API Thanks Andi Christiansen Sendt fra min iPhone > Den 7. maj 2020 kl. 12.08 skrev Christian Vieser : > > ? > Hi Andi, > up to now there are no instructions available on how to enable SSL on the Swift/S3 endpoints. > The only thing is that you can enable SSL on the authentication path. So your connection to Swift authentication on port 35357 will be secured and the S3 authentication arriving at http port 8080 will internally take the SSL path, if configured properly. We have successfully done that in a test environment. Be sure to use the --pwd-file option with the "mmuserauth service create ..." and verify the proxy settings afterwards. It should look like this: > # mmobj config list --ccrfile proxy-server.conf --section filter:s3token > > [filter:s3token] > auth_uri = https://127.0.0.1:35357/ > use = egg:swift3#s3token > insecure = true > > You can correct wrong settings with > # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property insecure --value true > # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property auth_uri --value 'https://127.0.0.1:35357/' > > Regards, > Christian > > > i have tried what you suggested. mmobj swift base ran fine. but after i have > > deleted the userauth and try to set it up again with ks-ssl enabled it just > > hangs: > > > > # mmuserauth service create --data-access-method object --type local > > --enable-ks-ssl > > > > still waiting for it to finish, 15 mins now.. :) > > > >> Basically all i need is this: > >> > >> https://s3.something.com:8080 https://s3.something.com:8080 which points > >> to the WAN ip of the CES cluster (already configured and ready) > >> > >> and endpoints like this: > >> > >> None | keystone | identity | True | public | https://cluster_domain:5000/ > >> https://cluster_domain:5000/ > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:443/v1/AUTH_%(tenant_id)s > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:8080/v1/AUTH_%(tenant_id)s > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu May 7 21:02:12 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 7 May 2020 22:02:12 +0200 Subject: [gpfsug-discuss] Enabling SSL/HTTPS/ on Object S3. In-Reply-To: References: <203777d1-1be0-898c-2845-b408597470f7@1und1.de> Message-ID: (almost verbatim copy of my previous email ? in case anybody else needs it, or has ideas for improvements :-) The way I would do this is to install "haproxy" on all these nodes, and have haproxy terminate SSL and balance incoming requests over the 3 CES-addresses. For S3 -- we only need to provide access to the swift port at 8080. First install haproxy: # yum install haproxy Put your cert and key into /etc/haproxy/ssl.pem: # cat server.key server.crt cert-chain.crt > /etc/haproxy/ssl.pem # chmod 400 /etc/haproxy/ssl.pem Then create a /etc/haproxy/haproxy.cfg: # cat <<'EOF' > /etc/haproxy/haproxy,cfg #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon tune.ssl.default-dh-param 2048 # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 listen stats *:80 maxconn 10 timeout client 100s timeout server 100s timeout connect 100s timeout queue 100s stats enable stats hide-version stats refresh 30s stats show-node stats auth admin:password stats uri /haproxy?stats frontend s3-frontend bind *:443 ssl crt /etc/haproxy/ssl.pem default_backend s3-backend backend s3-backend balance roundrobin server ces1 10.33.23.167:8080 check server ces2 10.33.23.168:8080 check server ces3 10.33.23.169:8080 check EOF # systemctl enable haproxy # systemctl start haproxy You only need to modify the IP-addresses in the s3-backend (make sure they point at your floating CES addresses, not the static ones), and maybe make a better username/password for the stats page at " *http://hostname/haproxy?stats* ". This setup does two layers of loadbalancing. First DNS round-robin, then haproxy roundrobin -- I don't think this is a negative thing, and it does make it possible to tune the loadbalancing further with more advanced algorithms for selecting backends. F.ex. we can do weighting, if some are more powerful than others. Or "leastconn", to send new requests to backends with the least number of active connections. -jf tor. 7. mai 2020 kl. 14:59 skrev Andi Christiansen : > Hi Christian, > > Thanks for answering! > > We solved this with lab services a while back now, and ended up setting up > haproxys I front of the ces nodes and then they handle the ssl encryption > to the S3 API > > Thanks > Andi Christiansen > > Sendt fra min iPhone > > Den 7. maj 2020 kl. 12.08 skrev Christian Vieser < > christian.vieser at 1und1.de>: > > ? > > Hi Andi, > > up to now there are no instructions available on how to enable SSL on the Swift/S3 endpoints. > > The only thing is that you can enable SSL on the authentication path. So your connection to Swift authentication on port 35357 will be secured and the S3 authentication arriving at http port 8080 will internally take the SSL path, if configured properly. We have successfully done that in a test environment. Be sure to use the --pwd-file option with the "mmuserauth service create ..." and verify the proxy settings afterwards. It should look like this: > > # mmobj config list --ccrfile proxy-server.conf --section filter:s3token > > [filter:s3token] > auth_uri = https://127.0.0.1:35357/use = egg:swift3#s3token > insecure = true > > You can correct wrong settings with# mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property insecure --value true > # mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property auth_uri --value 'https://127.0.0.1:35357/' > > Regards, > Christian > > > > i have tried what you suggested. mmobj swift base ran fine. but after i have > > deleted the userauth and try to set it up again with ks-ssl enabled it just > > hangs: > > > > # mmuserauth service create --data-access-method object --type local > > --enable-ks-ssl > > > > still waiting for it to finish, 15 mins now.. :) > > > >> Basically all i need is this: > >> > >> https://s3.something.com:8080 https://s3.something.com:8080 which points > >> to the WAN ip of the CES cluster (already configured and ready) > >> > >> and endpoints like this: > >> > >> None | keystone | identity | True | public | https://cluster_domain:5000/ > >> https://cluster_domain:5000/ > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:443/v1/AUTH_%(tenant_id)s > >> RegionOne | swift | object-store | True | public | > >> https://cluster_domain:8080/v1/AUTH_%(tenant_id)s > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.turner at ed.ac.uk Sat May 9 11:25:28 2020 From: aaron.turner at ed.ac.uk (TURNER Aaron) Date: Sat, 9 May 2020 10:25:28 +0000 Subject: [gpfsug-discuss] Odd networking/name resolution issue Message-ID: Dear All, We are getting, on an intermittent basis with currently no obvious pattern, an issue with GPFS nodes reporting rejecting nodes of the form: nodename.domain.domain.domain.... DNS resolution using the standard command-line tools of the IP address present in the logs does not repeat the domain, and so far it seems isolated to GPFS. Ultimately the nodes are rejected as not responding on the network. Has anyone seen this sort of behaviour before? Regards Aaron Turner The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Sat May 9 12:06:44 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Sat, 9 May 2020 07:06:44 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: DNS shouldn't be relied upon on a GPFS cluster for internal communication/management or data. As a starting point, make sure the IP's and names of all managers/quorum nodes and clients have *unique* entries in the hosts files of all other nodes in the clusters, being the same as how they where joined and licensed in the first place. If you issue a 'mmlscluster' on the cluster manager for the servers and clients, those results should be used to build the common hosts file for all nodes involved. Also, all nodes should have a common ntp configuration, pointing to the same *internal* ntp server, easily accessible via name/IP also on the hosts file. And obviously, you need a stable network, eth or IB. Have a good monitoring tool in place, to rule out network as a possible culprit. In the particular case of IB, check that the fabric managers are doing their jobs properly. And keep one eye on the 'tail -f /var/mmfs/gen/mmfslog' output of the managers and the nodes being expelled for other clues. Jaime On 5/9/2020 06:25:28, TURNER Aaron wrote: > Dear All, > > We are getting, on an intermittent basis with currently no obvious pattern, an issue with GPFS nodes reporting rejecting nodes of the form: > > nodename.domain.domain.domain.... > > DNS resolution using the standard command-line tools of the IP address present in the logs does not repeat the domain, and so far it seems isolated to GPFS. > > Ultimately the nodes are rejected as not responding on the network. > > Has anyone seen this sort of behaviour before? > > Regards > > Aaron Turner > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > . . . ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From jonathan.buzzard at strath.ac.uk Sat May 9 23:22:15 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sat, 9 May 2020 23:22:15 +0100 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: On 09/05/2020 12:06, Jaime Pinto wrote: > DNS shouldn't be relied upon on a GPFS cluster for internal > communication/management or data. > The 1980's have called and want their lack of IP resolution protocols back :-) I would kindly disagree. If your DNS is not working then your cluster is fubar anyway and a zillion other things will also break very rapidly. For us at least half of the running jobs would be dead in a few minutes as failure to contact license servers would cause the software to stop. All authentication and account lookup is also going to fail as well. You could distribute a hosts file but frankly outside of a storage only cluster (as opposed to one with hundreds if not thousands of compute nodes) that is frankly madness and will inevitably come to bite you in the ass because they *will* get out of sync. The only hosts entry we have is for the Salt Stack host because it tries to do things before the DNS resolvers have been setup and consequently breaks otherwise. Which IMHO is duff on it's behalf. I would add I can't think of a time in the last 16 years where internal DNS at any University I have worked at has stopped working for even one millisecond. If DNS is that flaky at your institution then I suggest sacking the people responsible for it's maintenance as being incompetent twits. It is just such a vanishingly remote possibility that it's not worth bothering about. Frankly a aircraft falling out the sky and squishing your data centre seems more likely to me. Finally in a world of IPv6 then anything other than DNS is a utter madness IMHO. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From aaron.turner at ed.ac.uk Sun May 10 08:35:29 2020 From: aaron.turner at ed.ac.uk (TURNER Aaron) Date: Sun, 10 May 2020 07:35:29 +0000 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: , Message-ID: Following on from Jonathan Buzzards comments, I'd also like to point out that I've never known a central DNS failure in a UK HEI for as long as I can remember, and it was certainly not my intention to suggest that as I think a central DNS issue is highly unlikely. And indeed, as I originally noted, the standard command-line tools on the nodes resolve the names as expected, so whatever is going on looks like it affects GPFS only. It may even be that the repetition of the domain names in the logs is just a function of something it is doing when logging when a node is failing to connect for some other reason entirely. It's just not something I recall having seen before and wanted to see if anyone else had seen it. ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard Sent: 09 May 2020 23:22 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Odd networking/name resolution issue On 09/05/2020 12:06, Jaime Pinto wrote: > DNS shouldn't be relied upon on a GPFS cluster for internal > communication/management or data. > The 1980's have called and want their lack of IP resolution protocols back :-) I would kindly disagree. If your DNS is not working then your cluster is fubar anyway and a zillion other things will also break very rapidly. For us at least half of the running jobs would be dead in a few minutes as failure to contact license servers would cause the software to stop. All authentication and account lookup is also going to fail as well. You could distribute a hosts file but frankly outside of a storage only cluster (as opposed to one with hundreds if not thousands of compute nodes) that is frankly madness and will inevitably come to bite you in the ass because they *will* get out of sync. The only hosts entry we have is for the Salt Stack host because it tries to do things before the DNS resolvers have been setup and consequently breaks otherwise. Which IMHO is duff on it's behalf. I would add I can't think of a time in the last 16 years where internal DNS at any University I have worked at has stopped working for even one millisecond. If DNS is that flaky at your institution then I suggest sacking the people responsible for it's maintenance as being incompetent twits. It is just such a vanishingly remote possibility that it's not worth bothering about. Frankly a aircraft falling out the sky and squishing your data centre seems more likely to me. Finally in a world of IPv6 then anything other than DNS is a utter madness IMHO. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Sun May 10 14:28:15 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Sun, 10 May 2020 09:28:15 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: The rationale for my suggestion doesn't have much to do with the central DNS server, but everything to do with the DNS client side of the service. If you have a very busy cluster at times, and a number of nodes really busy with 1000+ IOPs for instance, so much that the OS on the client can't barely spare a cycle to query the DSN server on what the IP associated with the name of interface leading to the GPFS infrastructure is, or even process that response when it returns, on the same interface where it's having contentions and trying to process all the gpfs data transactions, you can have temporary catch 22 situations. This can generate a backlog of waiters, and eventual expelling of some nodes when the cluster managers don't hear from them in reasonable time. It's doesn't really matter if you have a central DNS server in steroids. Jaime On 5/10/2020 03:35:29, TURNER Aaron wrote: > Following on from Jonathan Buzzards comments, I'd also like to point out that I've never known a central DNS failure in a UK HEI for as long as I can remember, and it was certainly not my intention to suggest that as I think a central DNS issue is highly unlikely. And indeed, as I originally noted, the standard command-line tools on the nodes resolve the names as expected, so whatever is going on looks like it affects GPFS only. It may even be that the repetition of the domain names in the logs is just a function of something it is doing when logging when a node is failing to connect for some other reason entirely. It's just not something I recall having seen before and wanted to see if anyone else had seen it. > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard > *Sent:* 09 May 2020 23:22 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* Re: [gpfsug-discuss] Odd networking/name resolution issue > On 09/05/2020 12:06, Jaime Pinto wrote: >> DNS shouldn't be relied upon on a GPFS cluster for internal >> communication/management or data. >> > > The 1980's have called and want their lack of IP resolution protocols > back :-) > > I would kindly disagree. If your DNS is not working then your cluster is > fubar anyway and a zillion other things will also break very rapidly. > For us at least half of the running jobs would be dead in a few minutes > as failure to contact license servers would cause the software to stop. > All authentication and account lookup is also going to fail as well. > > You could distribute a hosts file but frankly outside of a storage only > cluster (as opposed to one with hundreds if not thousands of compute > nodes) that is frankly madness and will inevitably come to bite you in > the ass because they *will* get out of sync. The only hosts entry we > have is for the Salt Stack host because it tries to do things before the > DNS resolvers have been setup and consequently breaks otherwise. Which > IMHO is duff on it's behalf. > > I would add I can't think of a time in the last 16 years where internal > DNS at any University I have worked at has stopped working for even one > millisecond. If DNS is that flaky at your institution then I suggest > sacking the people responsible for it's maintenance as being incompetent > twits. It is just such a vanishingly remote possibility that it's not > worth bothering about. Frankly a aircraft falling out the sky and > squishing your data centre seems more likely to me. > > Finally in a world of IPv6 then anything other than DNS is a utter > madness IMHO. > > > JAB. > > -- > Jonathan A. Buzzard???????????????????????? Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From richard.rupp at us.ibm.com Sun May 10 19:31:39 2020 From: richard.rupp at us.ibm.com (RICHARD RUPP) Date: Sun, 10 May 2020 14:31:39 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: Normally the DNS server publishes a TTL and the client side caches the info until the TTL expires. Could the server side be mis-configured for a very short TTL? Regards, Richard Rupp, Sales Specialist, Phone: 1-347-510-6746 From: Jaime Pinto To: gpfsug main discussion list , TURNER Aaron Date: 05/10/2020 09:28 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] Odd networking/name resolution issue Sent by: gpfsug-discuss-bounces at spectrumscale.org The rationale for my suggestion doesn't have much to do with the central DNS server, but everything to do with the DNS client side of the service. If you have a very busy cluster at times, and a number of nodes really busy with 1000+ IOPs for instance, so much that the OS on the client can't barely spare a cycle to query the DSN server on what the IP associated with the name of interface leading to the GPFS infrastructure is, or even process that response when it returns, on the same interface where it's having contentions and trying to process all the gpfs data transactions, you can have temporary catch 22 situations. This can generate a backlog of waiters, and eventual expelling of some nodes when the cluster managers don't hear from them in reasonable time. It's doesn't really matter if you have a central DNS server in steroids. Jaime On 5/10/2020 03:35:29, TURNER Aaron wrote: > Following on from Jonathan Buzzards comments, I'd also like to point out that I've never known a central DNS failure in a UK HEI for as long as I can remember, and it was certainly not my intention to suggest that as I think a central DNS issue is highly unlikely. And indeed, as I originally noted, the standard command-line tools on the nodes resolve the names as expected, so whatever is going on looks like it affects GPFS only. It may even be that the repetition of the domain names in the logs is just a function of something it is doing when logging when a node is failing to connect for some other reason entirely. It's just not something I recall having seen before and wanted to see if anyone else had seen it. > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > *From:* gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard > *Sent:* 09 May 2020 23:22 > *To:* gpfsug-discuss at spectrumscale.org > *Subject:* Re: [gpfsug-discuss] Odd networking/name resolution issue > On 09/05/2020 12:06, Jaime Pinto wrote: >> DNS shouldn't be relied upon on a GPFS cluster for internal >> communication/management or data. >> > > The 1980's have called and want their lack of IP resolution protocols > back :-) > > I would kindly disagree. If your DNS is not working then your cluster is > fubar anyway and a zillion other things will also break very rapidly. > For us at least half of the running jobs would be dead in a few minutes > as failure to contact license servers would cause the software to stop. > All authentication and account lookup is also going to fail as well. > > You could distribute a hosts file but frankly outside of a storage only > cluster (as opposed to one with hundreds if not thousands of compute > nodes) that is frankly madness and will inevitably come to bite you in > the ass because they *will* get out of sync. The only hosts entry we > have is for the Salt Stack host because it tries to do things before the > DNS resolvers have been setup and consequently breaks otherwise. Which > IMHO is duff on it's behalf. > > I would add I can't think of a time in the last 16 years where internal > DNS at any University I have worked at has stopped working for even one > millisecond. If DNS is that flaky at your institution then I suggest > sacking the people responsible for it's maintenance as being incompetent > twits. It is just such a vanishingly remote possibility that it's not > worth bothering about. Frankly a aircraft falling out the sky and > squishing your data centre seems more likely to me. > > Finally in a world of IPv6 then anything other than DNS is a utter > madness IMHO. > > > JAB. > > -- > Jonathan A. Buzzard???????????????????????? Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=EXL-jEd1jmdzvOIhT87C7SIqmAS9uhVQ6J3kObct4OY&m=celJq_mDruJ5qISvmV3mGvLXYvIVcOQlhK3uohc8ZWI&s=31S9Rqbtlv7T0AqAEsLWo8BMPRuHOi9z29DLUYW_PAs&e= > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=EXL-jEd1jmdzvOIhT87C7SIqmAS9uhVQ6J3kObct4OY&m=celJq_mDruJ5qISvmV3mGvLXYvIVcOQlhK3uohc8ZWI&s=31S9Rqbtlv7T0AqAEsLWo8BMPRuHOi9z29DLUYW_PAs&e= > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=EXL-jEd1jmdzvOIhT87C7SIqmAS9uhVQ6J3kObct4OY&m=celJq_mDruJ5qISvmV3mGvLXYvIVcOQlhK3uohc8ZWI&s=31S9Rqbtlv7T0AqAEsLWo8BMPRuHOi9z29DLUYW_PAs&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From jcatana at gmail.com Sun May 10 20:09:39 2020 From: jcatana at gmail.com (Josh Catana) Date: Sun, 10 May 2020 15:09:39 -0400 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: I've seen odd behavior like this before and it is to do with name resolution. It might be from your local /etc/hosts entries or potentially the names used to add the nodes to the cluster or even if you are using DNS aliases that are configured improperly. In my case someone added DNS aliases to use in our cluster with fqdn instead of shortname which caused the triple append to appear in the logs you mentioned. I don't think it hurts anything since GPFS has its own name-to-ip table, but you probably want to track it down and fix it to be safe. On Sun, May 10, 2020, 2:31 PM RICHARD RUPP wrote: > *Normally the DNS server publishes a TTL and the client side caches the > info until the TTL expires. Could the server side be mis-configured for a > very short TTL?* > > > Regards, > > *Richard Rupp*, Sales Specialist, *Phone:* *1-347-510-6746* > > > [image: Inactive hide details for Jaime Pinto ---05/10/2020 09:28:46 > AM---The rationale for my suggestion doesn't have much to do with]Jaime > Pinto ---05/10/2020 09:28:46 AM---The rationale for my suggestion doesn't > have much to do with the central DNS server, but everything > > From: Jaime Pinto > To: gpfsug main discussion list , > TURNER Aaron > Date: 05/10/2020 09:28 AM > Subject: [EXTERNAL] Re: [gpfsug-discuss] Odd networking/name resolution > issue > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > The rationale for my suggestion doesn't have much to do with the central > DNS server, but everything to do with the DNS client side of the service. > If you have a very busy cluster at times, and a number of nodes really > busy with 1000+ IOPs for instance, so much that the OS on the client can't > barely spare a cycle to query the DSN server on what the IP associated with > the name of interface leading to the GPFS infrastructure is, or even > process that response when it returns, on the same interface where it's > having contentions and trying to process all the gpfs data transactions, > you can have temporary catch 22 situations. This can generate a backlog of > waiters, and eventual expelling of some nodes when the cluster managers > don't hear from them in reasonable time. > > It's doesn't really matter if you have a central DNS server in steroids. > > Jaime > > On 5/10/2020 03:35:29, TURNER Aaron wrote: > > Following on from Jonathan Buzzards comments, I'd also like to point out > that I've never known a central DNS failure in a UK HEI for as long as I > can remember, and it was certainly not my intention to suggest that as I > think a central DNS issue is highly unlikely. And indeed, as I originally > noted, the standard command-line tools on the nodes resolve the names as > expected, so whatever is going on looks like it affects GPFS only. It may > even be that the repetition of the domain names in the logs is just a > function of something it is doing when logging when a node is failing to > connect for some other reason entirely. It's just not something I recall > having seen before and wanted to see if anyone else had seen it. > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > *From:* gpfsug-discuss-bounces at spectrumscale.org < > gpfsug-discuss-bounces at spectrumscale.org> on behalf of Jonathan Buzzard < > jonathan.buzzard at strath.ac.uk> > > *Sent:* 09 May 2020 23:22 > > *To:* gpfsug-discuss at spectrumscale.org > > > *Subject:* Re: [gpfsug-discuss] Odd networking/name resolution issue > > On 09/05/2020 12:06, Jaime Pinto wrote: > >> DNS shouldn't be relied upon on a GPFS cluster for internal > >> communication/management or data. > >> > > > > The 1980's have called and want their lack of IP resolution protocols > > back :-) > > > > I would kindly disagree. If your DNS is not working then your cluster is > > fubar anyway and a zillion other things will also break very rapidly. > > For us at least half of the running jobs would be dead in a few minutes > > as failure to contact license servers would cause the software to stop. > > All authentication and account lookup is also going to fail as well. > > > > You could distribute a hosts file but frankly outside of a storage only > > cluster (as opposed to one with hundreds if not thousands of compute > > nodes) that is frankly madness and will inevitably come to bite you in > > the ass because they *will* get out of sync. The only hosts entry we > > have is for the Salt Stack host because it tries to do things before the > > DNS resolvers have been setup and consequently breaks otherwise. Which > > IMHO is duff on it's behalf. > > > > I would add I can't think of a time in the last 16 years where internal > > DNS at any University I have worked at has stopped working for even one > > millisecond. If DNS is that flaky at your institution then I suggest > > sacking the people responsible for it's maintenance as being incompetent > > twits. It is just such a vanishingly remote possibility that it's not > > worth bothering about. Frankly a aircraft falling out the sky and > > squishing your data centre seems more likely to me. > > > > Finally in a world of IPv6 then anything other than DNS is a utter > > madness IMHO. > > > > > > JAB. > > > > -- > > Jonathan A. Buzzard Tel: +44141-5483420 > > HPC System Administrator, ARCHIE-WeSt. > > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From knop at us.ibm.com Mon May 11 03:54:13 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 11 May 2020 02:54:13 +0000 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon May 11 11:01:49 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 11 May 2020 11:01:49 +0100 Subject: [gpfsug-discuss] Odd networking/name resolution issue In-Reply-To: References: Message-ID: <1bc652e7-9c81-82ed-a962-23c24a5c83b2@strath.ac.uk> On 10/05/2020 14:28, Jaime Pinto wrote: > The rationale for my suggestion doesn't have much to do with the central > DNS server, but everything to do with the DNS client side of the service. > If you have a very busy cluster at times, and a number of nodes really > busy with 1000+ IOPs for instance, so much that the OS on the client > can't barely spare a cycle to query the DSN server on what the IP > associated with the name of interface leading to the GPFS infrastructure > is, or even process that response when it returns, on the same interface > where it's having contentions and trying to process all the gpfs data > transactions, you can have temporary catch 22 situations. This can > generate a backlog of waiters, and eventual expelling of some nodes when > the cluster managers don't hear from them in reasonable time. > > It's doesn't really matter if you have a central DNS server in steroids. > If that is the scenario it will struggle to look up the DNS to IP lists in /etc/hosts. GPFS itself will struggle to get scheduled. Besides which systemd is caching it all locally anyways which will get hit *before* /etc/hosts does. In an none systemd install then most likely nscd is running which provides similar service. You don't appear to a full grasp of how it hostname to IP resolution works. It is a outdated notion to suggest using a system that was deprecated over 30 years ago that needs stomping on because it comes from IMHO a lack of seeing the whole picture. Finally as I understand it GPFS maintains it own hostname to IP lookup table which makes it a completely moot point. You can see this from the fact you cannot change the IP address of a node in a cluster. You must remove it and then rejoin with the new IP address. If it was storing client information as hostnames and using hostname to IP resolution to work out the IP addresses that would not be the case. As such your suggestion is dreaming up a solution to a none existent problem, which makes it an even worse idea IMHO. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From TROPPENS at de.ibm.com Tue May 12 13:41:00 2020 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Tue, 12 May 2020 12:41:00 +0000 Subject: [gpfsug-discuss] THINK Virtual User Group Day In-Reply-To: <5BE5B210-5FEE-45E0-AC0D-1B184B5B8E45@spectrumscale.org> References: <5BE5B210-5FEE-45E0-AC0D-1B184B5B8E45@spectrumscale.org> Message-ID: An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Thu May 14 13:20:57 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Thu, 14 May 2020 12:20:57 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Message-ID: Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_2020152979] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From dean.flanders at fmi.ch Thu May 14 13:31:06 2020 From: dean.flanders at fmi.ch (Flanders, Dean) Date: Thu, 14 May 2020 12:31:06 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: References: Message-ID: Hello, It is great, that RHEL 7.8 is supported on SS 4.2.3.22, when will RHEL 8.x be supported on GPFS SS 4.2.3.X?? Thanks, Dean From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Carl Zetie - carlz at us.ibm.com Sent: Thursday, May 14, 2020 2:21 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_2020152979] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 13822 bytes Desc: image002.png URL: From Dugan.Witherick at warwick.ac.uk Thu May 14 14:03:17 2020 From: Dugan.Witherick at warwick.ac.uk (Witherick, Dugan) Date: Thu, 14 May 2020 13:03:17 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: References: Message-ID: <69508fba4e506dfed64703801f3756cfd29e8f1c.camel@warwick.ac.uk> Thanks Carl! Best regards, Dugan On Thu, 2020-05-14 at 12:20 +0000, Carl Zetie - carlz at us.ibm.com wrote: Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_2020152979] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From jonathan.buzzard at strath.ac.uk Thu May 14 15:22:25 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Thu, 14 May 2020 15:22:25 +0100 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: References: Message-ID: <0a7278de-33c6-ad20-d818-f3bbf5d19bcb@strath.ac.uk> On 14/05/2020 13:31, Flanders, Dean wrote: > Hello, It is great, that RHEL 7.8 is supported on SS 4.2.3.22, when will > RHEL 8.x be supported on GPFS SS 4.2.3.X?? Thanks, Dean That would be never, 4.2.3 goes out of support in September. Is 5.x supported in 7.8 yet? Some urgent upgrading of systems being looked into right now and I would prefer to jump to 5.x than 4.2.3.22 and have to upgrade again. However needs must. Oh and if you haven't yet I would recommend any HPC site revoking all your users SSH keys immediately. To many users have been creating SSH keys without passphrases and jumping from system to system. Several sites in the UK are currently down and I understand it has effected Germany too :-( There are zero guesses to the origin of the hacks. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From stockf at us.ibm.com Thu May 14 15:55:04 2020 From: stockf at us.ibm.com (Frederick Stock) Date: Thu, 14 May 2020 14:55:04 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Central In-Reply-To: <0a7278de-33c6-ad20-d818-f3bbf5d19bcb@strath.ac.uk> References: <0a7278de-33c6-ad20-d818-f3bbf5d19bcb@strath.ac.uk>, Message-ID: An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Fri May 15 13:07:27 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Fri, 15 May 2020 12:07:27 +0000 Subject: [gpfsug-discuss] Scale 4.2.3.22 with support for RHEL 7.8 is now on Fix Message-ID: <04A3D7D0-749D-4B5C-A6EC-117CA02B003E@us.ibm.com> JAB> Is 5.x supported in 7.8 yet? Support is planned for 5.0.5 later this month, subject to final regression testing now in progress Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_1898525102] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From carlz at us.ibm.com Mon May 18 13:06:55 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Mon, 18 May 2020 12:06:55 +0000 Subject: [gpfsug-discuss] Scale and Linux distros FAQ improvement Message-ID: Inspired by the RHEL 7.8 compatibility issues (and some earlier incidents too), we have adopted a new practice for the Scale FAQ where we will list new distro releases explicitly with a note indicating if they are not yet supported. In the past we should say things like ?7.7.xxx or later? for the supported release level, and not say anything at all about new OS releases until they were supported. That would cause confusion when 7.8 came along, since 7.8 is definitely later than 7.7.xxx... Hopefully this small change will help you with your admin. Regards, Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_565114115] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From b.cregan at imperial.ac.uk Mon May 18 13:59:09 2020 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Mon, 18 May 2020 12:59:09 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues Message-ID: Hi, At Imperial we have been experiencing an issue with SS 5.0.x and quotas. The main symptom is a slow decay in the accuracy of reported quota usage when compared to the actual usage as reported by "du". This discrepancy can be as little as a few percent and as much as many X100% . We also sometimes see bizarre effects such negative file number counts being reported. We have been working with IBM and have put in the latest 5.0.4-4 (that has a fix for mmcheckquota) that we have been pinning our hopes on, but this has not worked. Is anyone else experiencing similar issues? We need to try and get an idea if this is an issue peculiar to our set up or a more general SS problem. We are using user and group quotas in a fileset context. Thanks Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From pinto at scinet.utoronto.ca Mon May 18 14:55:47 2020 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 18 May 2020 09:55:47 -0400 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: References: Message-ID: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> So Bob, Yes, we too have observed an uncharacteristic lag on the correction of the internal quota accounting on GPFS since we updated from version 3.3 to version 4.x some 7-8 years ago. That lag remains through version 5.0.x as well. And it persisted through several appliances (DDN, G200, GSS, ESS and now DSS-G). In our university environment there is also a lot of data churning, in particular small files. The workaround has always been to periodically run mmcheckquota on the top independent fileset to expedite that correction (I have a crontab script that measures the relative size of the in-dought columns on the mmrepquota report, size and inodes, and if either exceeds 2% for any USR/GRP/FILESET I run mmrepquota) We have opened supports calls with IBM about this issue in the past, and we never got a solution, to possibly adjust some GPFS configuration parameter, and have this correction done automatically. We gave up. And that begs the question: what do you mean by "... 5.0.4-4 ... that has a fix for mmcheckquota"? Isn't mmcheckquota zeroing the in-doubt columns when you run it? The fix should be for gpfs (something buried in the code over many versions). As far as I can tell there has never been anything wrong with mmcheckquota. Thanks Jaime On 5/18/2020 08:59:09, Cregan, Bob wrote: > Hi, > ? ? ? At Imperial we? have been experiencing an issue with SS 5.0.x and quotas. The main symptom is a slow decay in the accuracy of reported quota usage when compared to the actual usage as reported by "du". This discrepancy can be as little as a few percent and as much as many? X100% . We also sometimes see bizarre effects such negative file number counts being reported. > > We have been working with IBM? and have put in the latest 5.0.4-4 (that has a fix for mmcheckquota) that we have been pinning our hopes on, but this has not worked. > > Is anyone else experiencing similar issues? We need to try and get an idea if this is an issue peculiar to our set up or a more general SS problem. > > We are using user and group quotas in a fileset context. > > Thanks > > > *Bob Cregan* > HPC Systems Analyst > > Information & Communication Technologies > > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > > W: www.imperial.ac.uk/ict/rcs > > _1505984389175_twitter.png @imperialRCS @imperialRSE > > 1505983882959_Imperial-RCS.png > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > . . . ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From knop at us.ibm.com Mon May 18 17:58:06 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 18 May 2020 16:58:06 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> References: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca>, Message-ID: An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Mon May 18 18:34:06 2020 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Mon, 18 May 2020 17:34:06 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: References: <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> Message-ID: Is there a simple way to tell if we?re currently being affected by this bug? We run 5.0.4-1 on the client side and 5.0.3-2.3 on the server side (DSS-G 2.4b I believe it is). -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' > On May 18, 2020, at 12:58 PM, Felipe Knop wrote: > > All, > > Likely not an answer to these situations, but a fix was introduced in 5.0.4.4 (APAR IJ22894) and 4.2.3.22 (APAR IJ24661) to address a (long-standing) problem where mmcheckquota would compute quotas incorrectly in file systems where metadata pool and data pool subblocksizes are different. > > Felipe > > ---- > Felipe Knop knop at us.ibm.com > GPFS Development and Security > IBM Systems > IBM Building 008 > 2455 South Rd, Poughkeepsie, NY 12601 > (845) 433-9314 T/L 293-9314 > > > > ----- Original message ----- > From: Jaime Pinto > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] SS 5.0.x and quota issues > Date: Mon, May 18, 2020 9:56 AM > > So Bob, > Yes, we too have observed an uncharacteristic lag on the correction of the internal quota accounting on GPFS since we updated from version 3.3 to version 4.x some 7-8 years ago. That lag remains through version 5.0.x as well. And it persisted through several appliances (DDN, G200, GSS, ESS and now DSS-G). In our university environment there is also a lot of data churning, in particular small files. > > The workaround has always been to periodically run mmcheckquota on the top independent fileset to expedite that correction (I have a crontab script that measures the relative size of the in-dought columns on the mmrepquota report, size and inodes, and if either exceeds 2% for any USR/GRP/FILESET I run mmrepquota) > > We have opened supports calls with IBM about this issue in the past, and we never got a solution, to possibly adjust some GPFS configuration parameter, and have this correction done automatically. We gave up. > > And that begs the question: what do you mean by "... 5.0.4-4 ... that has a fix for mmcheckquota"? Isn't mmcheckquota zeroing the in-doubt columns when you run it? > > The fix should be for gpfs (something buried in the code over many versions). As far as I can tell there has never been anything wrong with mmcheckquota. > > Thanks > Jaime > > > On 5/18/2020 08:59:09, Cregan, Bob wrote: > > Hi, > > At Imperial we have been experiencing an issue with SS 5.0.x and quotas. The main symptom is a slow decay in the accuracy of reported quota usage when compared to the actual usage as reported by "du". This discrepancy can be as little as a few percent and as much as many X100% . We also sometimes see bizarre effects such negative file number counts being reported. > > > > We have been working with IBM and have put in the latest 5.0.4-4 (that has a fix for mmcheckquota) that we have been pinning our hopes on, but this has not worked. > > > > Is anyone else experiencing similar issues? We need to try and get an idea if this is an issue peculiar to our set up or a more general SS problem. > > > > We are using user and group quotas in a fileset context. > > > > Thanks > > > > > > *Bob Cregan* > > HPC Systems Analyst > > > > Information & Communication Technologies > > > > Imperial College London, > > South Kensington Campus London, SW7 2AZ > > T: 07712388129 > > E: b.cregan at imperial.ac.uk > > > > W: www.imperial.ac.uk/ict/rcs > > > > _1505984389175_twitter.png @imperialRCS @imperialRSE > > > > 1505983882959_Imperial-RCS.png > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > . > . > . ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto - Storage Analyst > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From knop at us.ibm.com Mon May 18 21:05:07 2020 From: knop at us.ibm.com (Felipe Knop) Date: Mon, 18 May 2020 20:05:07 +0000 Subject: [gpfsug-discuss] SS 5.0.x and quota issues In-Reply-To: References: , <9094c563-d2d5-2c32-28f9-bde1c1682b33@scinet.utoronto.ca> Message-ID: An HTML attachment was scrubbed... URL: From juergen.hannappel at desy.de Tue May 19 15:30:34 2020 From: juergen.hannappel at desy.de (Hannappel, Juergen) Date: Tue, 19 May 2020 16:30:34 +0200 (CEST) Subject: [gpfsug-discuss] gpfs_fcntl fails on read-only-fs Message-ID: <1008080204.14431165.1589898634575.JavaMail.zimbra@desy.de> Hi, I tried to do gpfs_fcntl with the GPFS_ACCESS_RANGE hint, with the isWrite field set to 0 and start = 0, lenght = size of the file to indicate that I want to read the entire file now, but on a readonly remote file system the gpfs_fcntl() call returns -1 and sets errno to "Read-only file system" whic is true but should not matter in my Opinion. Why is announcing a range of the file for reading not allowed on a read onoy file system? -- Dr. J?rgen Hannappel DESY/IT Tel. : +49 40 8998-4616 From carlz at us.ibm.com Wed May 20 14:05:11 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Wed, 20 May 2020 13:05:11 +0000 Subject: [gpfsug-discuss] Are you using rsync? Message-ID: Folks, I?m looking for a handful of volunteers who are using rsync with Scale and willing to jump on a 15 minute call (separate calls for each volunteer, to be clear) to discuss how you are using rsync, what features are essential to you, what its shortcomings are, and anything else you want to share. And if you?re *not* using rsync because of some specific limitation but would like to, I?m also interested in hearing from you. And finally: if you have thoughts on this topic but don?t want to get on the phone, please feel free to email me. Please respond OFF LIST to the email address below. Thanks, Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_74924333] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From Ivano.Talamo at psi.ch Fri May 22 08:47:45 2020 From: Ivano.Talamo at psi.ch (Talamo Ivano Giuseppe (PSI)) Date: Fri, 22 May 2020 07:47:45 +0000 Subject: [gpfsug-discuss] selinux context Message-ID: Hi all, I?m configuring a set of login nodes with home directories in GPFS (but not on /home), with SElinux in enforcing mode and auto creation of home directory (via PAM). I?ve been able to partially achieve my target, by basically running the two following commands: semanage fcontext -a -e /home /das/home restorecon -v /das/home After having done this on one node, the context on the directory is the expected one (system_u:object_r:home_root_t:s0). And everything works as expected (a new user logs in and his directory is created). But on all the other nodes of the cluster still the old context is shown (system_u:object_r:unlabeled_t:s0). Unless I run the restorecon on them too. Furthermore, since the filesystem is a remote-cluster mount, on all the nodes on the central (storage) cluster, the corrent (home_root_t) context is shown. I was expecting the SElinux context to be stored in the inodes, but now the situation looks mixed and I?m puzzled. In case it can help, the login nodes are RHEL 7.7 with Spectrum Scale 5.0.4. The storage is RHEL 7.6 with 5.0.3. Does someone have any experience/idea? Thanks, __________________________________________ Paul Scherrer Institut Ivano Talamo WHGA/038 Forschungsstrasse 111 5232 Villigen PSI Schweiz Telefon: +41 56 310 47 71 E-Mail: ivano.talamo at psi.ch From lists at esquad.de Fri May 22 19:22:51 2020 From: lists at esquad.de (Dieter Mosbach) Date: Fri, 22 May 2020 20:22:51 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) Message-ID: From valdis.kletnieks at vt.edu Sun May 24 09:42:13 2020 From: valdis.kletnieks at vt.edu (Valdis Kl=?utf-8?Q?=c4=93?=tnieks) Date: Sun, 24 May 2020 04:42:13 -0400 Subject: [gpfsug-discuss] selinux context In-Reply-To: References: Message-ID: <34924.1590309733@turing-police> On Fri, 22 May 2020 07:47:45 -0000, "Talamo Ivano Giuseppe (PSI)" said: > After having done this on one node, the context on the directory is the expec > expected one (system_u:object_r:home_root_t:s0). And everything works as expected (a > new user logs in and his directory is created). > But on all the other nodes of the cluster still the old context is shown > (system_u:object_r:unlabeled_t:s0). Unless I run the restorecon on them too. > Furthermore, since the filesystem is a remote-cluster mount, on all the nodes > on the central (storage) cluster, the corrent (home_root_t) context is shown. > I was expecting the SElinux context to be stored in the inodes, but now the > situation looks mixed and I???m puzzled. I suspect the issue is that the other nodes have that inode cached already, and they don't find out that that the SELinux context has been changed. I can't tell from here from whether GPFS is failing to realize that a context change means the old inode is stale just like any other inode change, or if there's something else that has gone astray. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From scottg at emailhosting.com Sun May 24 14:57:57 2020 From: scottg at emailhosting.com (scott) Date: Sun, 24 May 2020 09:57:57 -0400 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: References: Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Is there a bug-fix list? we cant seem to find it - we can just find the new/change features. On 5/22/20 2:22 PM, Dieter Mosbach wrote: > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Achim.Rehor at de.ibm.com Sun May 24 16:58:35 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Sun, 24 May 2020 17:58:35 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Message-ID: An HTML attachment was scrubbed... URL: From Achim.Rehor at de.ibm.com Mon May 25 08:48:42 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Mon, 25 May 2020 09:48:42 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Message-ID: no, as this is a .0 package .. there are no 'bug-fixes' ;) you will see this again, when PTF 1 is announced. Mit freundlichen Gr??en / Kind regards Achim Rehor gpfsug-discuss-bounces at spectrumscale.org wrote on 24/05/2020 15:57:57: > From: scott > To: gpfsug-discuss at spectrumscale.org > Date: 24/05/2020 16:04 > Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is > available on FixCentral (n/t) > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Is there a bug-fix list? we cant seem to find it - we can just find the > new/change features. > > On 5/22/20 2:22 PM, Dieter Mosbach wrote: > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=RGTETs2tk0Kz_VOpznDVDkqChhnfLapOTkxLvgmR2- > M&m=P0Br9hWVFau3jYGAI4PCmhGihPTI0UNbIc- > S68rfjy0&s=FeqsNGo3OsZ9n7DfC_Rrx5MyiXeG3hO7oR3YwhWuU8s&e= > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=RGTETs2tk0Kz_VOpznDVDkqChhnfLapOTkxLvgmR2- > M&m=P0Br9hWVFau3jYGAI4PCmhGihPTI0UNbIc- > S68rfjy0&s=FeqsNGo3OsZ9n7DfC_Rrx5MyiXeG3hO7oR3YwhWuU8s&e= > From jonathan.buzzard at strath.ac.uk Mon May 25 19:23:07 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 25 May 2020 19:23:07 +0100 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> Message-ID: <06948041-50e5-4150-083d-cdcf52f27515@strath.ac.uk> On 24/05/2020 16:58, Achim Rehor wrote: > > no, as this is a .0 package .. there are no 'bug-fixes' ;) ?you will see > this again, when PTF 1 is announced. So you are saying that no bugs that existed in 5.0.4.x have been fixed in 5.0.5.0? That has the credibility of Dominic Cummings driving to Barnard Castle to "test his eyesight". JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From S.J.Thompson at bham.ac.uk Tue May 26 12:59:30 2020 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Tue, 26 May 2020 11:59:30 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) In-Reply-To: <06948041-50e5-4150-083d-cdcf52f27515@strath.ac.uk> References: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b@emailhosting.com> <06948041-50e5-4150-083d-cdcf52f27515@strath.ac.uk> Message-ID: <35762876-A525-4CD9-8C91-A1CE5DC05135@bham.ac.uk> Well there is an interesting question there ... Where would you start counting the "bugs" from? 5.0.4.0? or 0.0? Or 5.0.4.4? So I think there is a valid point somewhere in there __ Simon ?On 25/05/2020, 19:23, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jonathan Buzzard" wrote: On 24/05/2020 16:58, Achim Rehor wrote: > > no, as this is a .0 package .. there are no 'bug-fixes' ;) you will see > this again, when PTF 1 is announced. So you are saying that no bugs that existed in 5.0.4.x have been fixed in 5.0.5.0? That has the credibility of Dominic Cummings driving to Barnard Castle to "test his eyesight". JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From carlz at us.ibm.com Tue May 26 14:44:45 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Tue, 26 May 2020 13:44:45 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral Message-ID: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Achim, I think the request here (lost in translation?) is for a list of the bugs that 5.0.5.0 addresses. And we're currently looking to see if we can generate that list. Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com ?On 5/25/20, 7:00 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=obB2s7QQTgU9QMn1708Vpg&m=vmAWHys4rqj2tGjdDeGEv0DQVLBf8oN-bBlHJc23SQ0&s=G30K6mkoWpJkWBqltaiUM49Bn_RWnK2Ke9VTThHnUeo&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) (scott) 2. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) (Achim Rehor) 3. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) (Achim Rehor) ---------------------------------------------------------------------- Message: 1 Date: Sun, 24 May 2020 09:57:57 -0400 From: scott To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b at emailhosting.com> Content-Type: text/plain; charset=utf-8; format=flowed Is there a bug-fix list? we cant seem to find it - we can just find the new/change features. From carlz at us.ibm.com Tue May 26 14:47:05 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Tue, 26 May 2020 13:47:05 +0000 Subject: [gpfsug-discuss] Follow up on Scale and rsync Message-ID: Folks, I want to thank everybody who responded to my request about rsync usage with Scale. I got many more replies than I expected, and several of you went ahead and provided information without even waiting for a phone call. Consequently I have a lot of information to read through and digest, and will probably be rewriting my questions as a result, so it?s going to take me a little longer to get back to those of you who kindly offered your time to discuss the topic. However, the first takeaway is that clearly, a lot of you are heavily dependent on rsync. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_1890439710] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From janfrode at tanso.net Tue May 26 15:51:27 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Tue, 26 May 2020 16:51:27 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral In-Reply-To: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> References: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Message-ID: Seeing that as a %changelog in the RPMs would be fantastic.. :-) -jf tir. 26. mai 2020 kl. 15:44 skrev Carl Zetie - carlz at us.ibm.com < carlz at us.ibm.com>: > Achim, I think the request here (lost in translation?) is for a list of > the bugs that 5.0.5.0 addresses. And we're currently looking to see if we > can generate that list. > > > > Carl Zetie > Program Director > Offering Management > Spectrum Scale > ---- > (919) 473 3318 ][ Research Triangle Park > carlz at us.ibm.com > > > ?On 5/25/20, 7:00 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf > of gpfsug-discuss-request at spectrumscale.org" < > gpfsug-discuss-bounces at spectrumscale.org on behalf of > gpfsug-discuss-request at spectrumscale.org> wrote: > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=obB2s7QQTgU9QMn1708Vpg&m=vmAWHys4rqj2tGjdDeGEv0DQVLBf8oN-bBlHJc23SQ0&s=G30K6mkoWpJkWBqltaiUM49Bn_RWnK2Ke9VTThHnUeo&e= > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (scott) > 2. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > 3. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 24 May 2020 09:57:57 -0400 > From: scott > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on > FixCentral (n/t) > Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b at emailhosting.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > Is there a bug-fix list? we cant seem to find it - we can just find > the > new/change features. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Achim.Rehor at de.ibm.com Tue May 26 16:04:17 2020 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Tue, 26 May 2020 17:04:17 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral In-Reply-To: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> References: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Message-ID: Yes, Carl, i guess that's what he is looking for ... but Simon has a valid point. Against what would we count bug fixes, as it is a .0 release ? ;) I would guess that's why we never did list "Problems fixed in ..." in the README of the .0 releases. Changes are documented, but these are feature changes new to 5.0.5 (as opposed to 5.0.4) Mit freundlichen Gr??en / Kind regards Achim Rehor Technical Support Specialist Spectrum Scale and ESS (SME) Advisory Product Services Professional IBM Systems Storage Support - EMEA gpfsug-discuss-bounces at spectrumscale.org wrote on 26/05/2020 15:44:45: > From: "Carl Zetie - carlz at us.ibm.com" > To: "gpfsug-discuss at spectrumscale.org" > Date: 26/05/2020 15:44 > Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is > available on FixCentral > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Achim, I think the request here (lost in translation?) is for a list > of the bugs that 5.0.5.0 addresses. And we're currently looking to > see if we can generate that list. > > > > Carl Zetie > Program Director > Offering Management > Spectrum Scale > ---- > (919) 473 3318 ][ Research Triangle Park > carlz at us.ibm.com > > > On 5/25/20, 7:00 AM, "gpfsug-discuss-bounces at spectrumscale.org on > behalf of gpfsug-discuss-request at spectrumscale.org" bounces at spectrumscale.org on behalf of gpfsug-discuss- > request at spectrumscale.org> wrote: > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=obB2s7QQTgU9QMn1708Vpg&m=vmAWHys4rqj2tGjdDeGEv0DQVLBf8oN- > bBlHJc23SQ0&s=G30K6mkoWpJkWBqltaiUM49Bn_RWnK2Ke9VTThHnUeo&e= > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (scott) > 2. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > 3. Re: Spectrum Scale 5.0.5.0 is available on FixCentral (n/t) > (Achim Rehor) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 24 May 2020 09:57:57 -0400 > From: scott > To: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on > FixCentral (n/t) > Message-ID: <2fde2b2c-3b45-1f63-9653-cf727cecaf1b at emailhosting.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > Is there a bug-fix list? we cant seem to find it - we can just find the > new/change features. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=RGTETs2tk0Kz_VOpznDVDkqChhnfLapOTkxLvgmR2- > M&m=9yPfjB7JuhNahhwFwELNpS_PTi9-2RKvKXsoavY3mgs&s=- > JzPycQHt6vWBH_qWUEBBn8WISeK4Xbfxh4HRJ2zbsE&e= > From christian.vieser at 1und1.de Wed May 27 08:18:44 2020 From: christian.vieser at 1und1.de (Christian Vieser) Date: Wed, 27 May 2020 09:18:44 +0200 Subject: [gpfsug-discuss] Spectrum Scale 5.0.5.0 is available on FixCentral In-Reply-To: References: <0C87190D-035D-4405-AF5F-A459AEDEBE44@us.ibm.com> Message-ID: <77257593-9ae5-8bf6-4aff-cf6a4ec3aca5@1und1.de> While these all are valid points, I had several support cases in the last months where I got feedback that the issue will be fixed in 5.0.5 or 5.1.0 . So there has to be a list of fixes even in a .0 release, that didn't made it to the last PTF of the previous release. Christian Achim Rehor wrote: > Yes, Carl, > > i guess that's what he is looking for ... but Simon has a valid point. > Against what would we count bug fixes, as it is a .0 release ? ;) > I would guess that's why we never did list "Problems fixed in ..." in the > README of the .0 releases. > Changes are documented, but these are feature changes new to 5.0.5 (as > opposed to 5.0.4) > > > Mit freundlichen Gr??en / Kind regards > > Achim Rehor > Simon Thompson wrote: > Well there is an interesting question there ... > > Where would you start counting the "bugs" from? > > 5.0.4.0? or 0.0? Or 5.0.4.4? > > So I think there is a valid point somewhere in there __ > > Simon -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3378 bytes Desc: S/MIME Cryptographic Signature URL: From heinrich.billich at id.ethz.ch Wed May 27 09:41:11 2020 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Wed, 27 May 2020 08:41:11 +0000 Subject: [gpfsug-discuss] Parse -Y command output Message-ID: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> Hello, I wonder if any python or bash functions exist to parse the output of mm-command's -Y format, i.e. colon-separated with HEADER rows. I would be nice to convert the output to a python list of named tuples or even better pandas dataframe. I would like to access the values by column name and row index. This would allow to do quick scripts or reports easily. Imagine using a jupyter notebook to dig into mmrepquota or mmlsdisk output, even doing charts, ..., easily. I know the ReST interface does exisit, which I could call to get the data in Json format, but to parse the Y-format seems much less cumbersome and allows to collect the data at one time and process it later. If you have other suggestions please let me know. What I would like to get: A list of NSDs which shows the connection to vdisk-set. Vdisk, recovery group and ESS system for each NSD. To get this I need to merge at least two tables. A list of user of fileset quota values highlighting those having non-default vaulues or close to the limit. The ability to get the data quickly in an uniform way into some tool, preferably python, to do analysis or formatting. Cheers, Heiner From juergen.hannappel at desy.de Wed May 27 10:04:00 2020 From: juergen.hannappel at desy.de (Hannappel, Juergen) Date: Wed, 27 May 2020 11:04:00 +0200 (CEST) Subject: [gpfsug-discuss] Parse -Y command output In-Reply-To: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> References: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> Message-ID: <1635082966.18343904.1590570240877.JavaMail.zimbra@desy.de> Hi, an example in python 2.7 (for Python 3 you need to add e.g. errors='ignore' to the parameters of the Popen call to get the proper kind of text stream as output. import subprocess import csv mmlsfileset = subprocess.Popen(['/usr/lpp/mmfs/bin/mmlsfileset', args.filesystem, '-Y'], stdout=subprocess.PIPE) reader = csv.DictReader(mmlsfileset.stdout, delimiter=':') filesets=[] for row in reader: if row['status'] == 'Linked': filesets.append({'name':row['filesetName'], 'linkdir':urllib.unquote(row['path'])}) mmlsfileset.wait() ----- Original Message ----- > From: "Billich Heinrich Rainer (ID SD)" > To: "gpfsug main discussion list" > Sent: Wednesday, 27 May, 2020 10:41:11 > Subject: [gpfsug-discuss] Parse -Y command output > Hello, > > I wonder if any python or bash functions exist to parse the output of > mm-command's -Y format, i.e. colon-separated with HEADER rows. I would be nice > to convert the output to a python list of named tuples or even better pandas > dataframe. I would like to access the values by column name and row index. > This would allow to do quick scripts or reports easily. Imagine using a > jupyter notebook to dig into mmrepquota or mmlsdisk output, even doing charts, > ..., easily. > > I know the ReST interface does exisit, which I could call to get the data in > Json format, but to parse the Y-format seems much less cumbersome and allows to > collect the data at one time and process it later. > > If you have other suggestions please let me know. What I would like to get: > A list of NSDs which shows the connection to vdisk-set. Vdisk, recovery group > and ESS system for each NSD. To get this I need to merge at least two tables. > A list of user of fileset quota values highlighting those having non-default > vaulues or close to the limit. > The ability to get the data quickly in an uniform way into some tool, preferably > python, to do analysis or formatting. > > Cheers, > > Heiner > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From achim.christ at de.ibm.com Wed May 27 10:08:26 2020 From: achim.christ at de.ibm.com (Achim Christ) Date: Wed, 27 May 2020 09:08:26 +0000 Subject: [gpfsug-discuss] Parse -Y command output In-Reply-To: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> References: <545325FA-C5F6-4DFD-A355-DB5489CCE39F@id.ethz.ch> Message-ID: An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Wed May 27 13:39:49 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Wed, 27 May 2020 12:39:49 +0000 Subject: [gpfsug-discuss] Scale release changelog (was: 5.0.5 available on Fix Central) Message-ID: <07847A5D-ED06-43B4-A69A-98A2B02787A3@us.ibm.com> > i guess that's what he is looking for ... but Simon has a valid point. > Against what would we count bug fixes, as it is a .0 release ? ;) The previous Mod level? But really, I'd like to know what the users think would be most useful to them. So I've created an RFE and invite people to submit their comments and votes there: https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=142683 Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com ? From prasad.surampudi at theatsgroup.com Thu May 28 23:31:04 2020 From: prasad.surampudi at theatsgroup.com (Prasad Surampudi) Date: Thu, 28 May 2020 22:31:04 +0000 Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> We have two scale clusters, cluster-A running version Scale 4.2.3 and RHEL6/7 and Cluster-B running Spectrum Scale 5.0.4 and RHEL 8.1. All the nodes in both Cluster-A and Cluster-B are direct attached and no NSD servers. We have our current filesystem gpfs_4 in Cluster-A and new filesystem gpfs_5 in Cluster-B. We want to copy all our data from gpfs_4 filesystem into gpfs_5 which has variable block size. So, can we map NSDs of gpfs_4 to Cluster-B nodes and do a mmexportfs of gpfs_4 from Cluster-A and mmimportfs into Cluster-B so that we have both filesystems available on same node in Cluster-B for copying data across fiber channel? If mmexportfs/mmimportfs works, can we delete nodes from Cluster-A and add them to Cluster-B without upgrading RHEL or GPFS versions for now and plan upgrading them at a later time? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjdoherty at yahoo.com Fri May 29 02:35:13 2020 From: jjdoherty at yahoo.com (Jim Doherty) Date: Fri, 29 May 2020 01:35:13 +0000 (UTC) Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster In-Reply-To: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> References: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> Message-ID: <102892274.973726.1590716113411@mail.yahoo.com> What is the minimum release level of the Spectrum Scale 5.0.4 cluster???? Is it 4.2.3.X?? Jim Doherty On Thursday, May 28, 2020, 6:31:21 PM EDT, Prasad Surampudi wrote: We have two scale clusters, cluster-A running version Scale 4.2.3 and RHEL6/7 and Cluster-B running Spectrum Scale ?5.0.4 and RHEL 8.1. All the nodes in both Cluster-A and Cluster-B are direct attached and no NSD servers. We have our current filesystem gpfs_4 in Cluster-A ?and new filesystem gpfs_5 in Cluster-B. We want to copy all our data from gpfs_4 filesystem into gpfs_5 which has variable block size.? So, can we map NSDs of gpfs_4 to Cluster-B nodes and do a mmexportfs of gpfs_4 from Cluster-A and mmimportfs into Cluster-B so that we have both filesystems available on same node in Cluster-B for copying data across fiber channel? If mmexportfs/mmimportfs works, can we delete nodes from Cluster-A and add them to Cluster-B without upgrading RHEL or GPFS versions for now and ?plan upgrading them at a later time? _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Fri May 29 08:22:52 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 29 May 2020 08:22:52 +0100 Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster In-Reply-To: <102892274.973726.1590716113411@mail.yahoo.com> References: <8A9F7C61-E669-41F7-B74D-70B9BC4B3DB1@theatsgroup.com> <102892274.973726.1590716113411@mail.yahoo.com> Message-ID: On 29/05/2020 02:35, Jim Doherty wrote: > > What is the minimum release level of the Spectrum Scale 5.0.4 > cluster???? Is it 4.2.3.X? > According to the email Cluster-B (the one with 5.0.4) has the variable block size thing, so that is going to be a no. Besides which even if you could mmimport the file system into Cluster-B you can't "merge" the two file systems. You are going to need to use AFM, rsync or similar. Hopefull they have enough free space on gpfs_5 to copy the data from gpfs_4 :-) JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From carlz at us.ibm.com Fri May 29 13:25:46 2020 From: carlz at us.ibm.com (Carl Zetie - carlz@us.ibm.com) Date: Fri, 29 May 2020 12:25:46 +0000 Subject: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: From prasad.surampudi at theatsgroup.com Fri May 29 16:02:12 2020 From: prasad.surampudi at theatsgroup.com (Prasad Surampudi) Date: Fri, 29 May 2020 15:02:12 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 In-Reply-To: References: Message-ID: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. ?On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** From ulmer at ulmer.org Fri May 29 20:55:01 2020 From: ulmer at ulmer.org (Stephen Ulmer) Date: Fri, 29 May 2020 15:55:01 -0400 Subject: [gpfsug-discuss] Multi-cluster question (was Re: gpfsug-discuss Digest, Vol 100, Issue 32) In-Reply-To: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: I have a question about multi-cluster, but it is related to this thread (it would be solving the same problem). Let?s say we have two clusters A and B, both clusters are normally shared-everything with no NSD servers defined. We want cluster B to be able to use a file system in cluster A. If I zone the SAN such that cluster B can see all of cluster A?s disks, can I then define a multi-cluster relationship between them and mount a file system from A on B? To state it another way, must B's I/O for the foreign file system pass though NSD servers in A, or can B?s nodes discover that they have FibreChannel paths to those disks and use them? I know that the nodes in B become sort-of ?casual" members of A, but I don?t know *how* casual the relationship is... Liberty, -- Stephen > On May 29, 2020, at 11:02 AM, Prasad Surampudi wrote: > > Jim, > The minimum release for 5.0.4 cluster is 5.0.4.3. > > We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. > > > ?On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 > cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 29 May 2020 12:25:46 +0000 > From: "Carl Zetie - carlz at us.ibm.com" > To: "gpfsug-discuss at spectrumscale.org" > > Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem > from 4.2.3 cluster to 5.0.4.3 cluster > Message-ID: > Content-Type: text/plain; charset="utf-8" > > On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. > > If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. > > Regards > > > > > Carl Zetie > Program Director > Offering Management > Spectrum Scale > ---- > (919) 473 3318 ][ Research Triangle Park > carlz at us.ibm.com > > [signature_398516855] > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: image001.png > Type: image/png > Size: 69558 bytes > Desc: image001.png > URL: > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 100, Issue 32 > *********************************************** > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Fri May 29 22:30:08 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Fri, 29 May 2020 22:30:08 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 In-Reply-To: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: <3597cc2c-47af-461f-af03-4f88069e1ca6@strath.ac.uk> On 29/05/2020 16:02, Prasad Surampudi wrote: > Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. > > We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. > Our plan is to import the gpfs_4 filesystem into the Spectrum Scale > 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters > nodes) and copy the data using rsync, across fibre channel instead > of ethernet to get higher bandwidth. > Ethernet goes *very* fast these days you know :-) In fact *much* faster than fibre channel. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From sadaniel at us.ibm.com Sat May 30 00:39:59 2020 From: sadaniel at us.ibm.com (Steven Daniels) Date: Fri, 29 May 2020 17:39:59 -0600 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 In-Reply-To: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: Prasad, One of the clients I work with have several 5.0.X ESS (owning) clusters and they share 4.2.3.9 file systems to remote client clusters. The minimum release level on the 5.0.X clusters are at the version of the cluster, i.e. 5.0.3 or 5.0.4 (we are in the process of getting all to latest). The key is that the clients are not part of this cluster as a node has to be > minReleaseLevel to be able to join. They are in their own (accessing) cluster which has a minReleaseLevel of 4.2.3-9. This is the current (until September withdrawal) minReleaseLevel that is supported in this scenario. Don't know specifically about import but it should work as long as the clients that need to mount the sub-5.0 filesystem can be in a separate (remote) cluster or the client nodes are upgraded to the minReleaseLevel of the local cluster. Thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 09:02 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Sent by: gpfsug-discuss-bounces at spectrumscale.org Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=u37vFxPtfEbuPwTW6wj7sbTFugW8l6DsRWtag_KO4Bk&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.png&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=YORJyUthJSKVhDxjwEwePTRexsjhX9F3sXPJg_Z5c5g&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: From prasad.surampudi at theatsgroup.com Sat May 30 00:55:05 2020 From: prasad.surampudi at theatsgroup.com (Prasad Surampudi) Date: Fri, 29 May 2020 23:55:05 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 34 In-Reply-To: References: Message-ID: <6DFF8E9C-EB6E-4A67-90E2-631E625F11BA@theatsgroup.com> Steve, So in essence, if the minReleaseLevel is set to 5.0.x.x for a Spectrum Scale 5.0 cluster, we can't add any new severs with Spectrum Scale 4.x.x RPMs installed without upgrading them to Spectrum Scale 5.x.x RPMs to the cluster. Is this a correct statement? Does mmiportfs also check the filesystem version which it is importing and prevent it from importing if it is created on Spectrum Scale 4.x.x? ?On 5/29/20, 7:40 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: gpfsug-discuss Digest, Vol 100, Issue 32 (Steven Daniels) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 17:39:59 -0600 From: "Steven Daniels" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Message-ID: Content-Type: text/plain; charset="iso-8859-1" Prasad, One of the clients I work with have several 5.0.X ESS (owning) clusters and they share 4.2.3.9 file systems to remote client clusters. The minimum release level on the 5.0.X clusters are at the version of the cluster, i.e. 5.0.3 or 5.0.4 (we are in the process of getting all to latest). The key is that the clients are not part of this cluster as a node has to be > minReleaseLevel to be able to join. They are in their own (accessing) cluster which has a minReleaseLevel of 4.2.3-9. This is the current (until September withdrawal) minReleaseLevel that is supported in this scenario. Don't know specifically about import but it should work as long as the clients that need to mount the sub-5.0 filesystem can be in a separate (remote) cluster or the client nodes are upgraded to the minReleaseLevel of the local cluster. Thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 09:02 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Sent by: gpfsug-discuss-bounces at spectrumscale.org Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=u37vFxPtfEbuPwTW6wj7sbTFugW8l6DsRWtag_KO4Bk&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.png&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=YORJyUthJSKVhDxjwEwePTRexsjhX9F3sXPJg_Z5c5g&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 100, Issue 34 *********************************************** From sadaniel at us.ibm.com Sat May 30 23:19:28 2020 From: sadaniel at us.ibm.com (Steven Daniels) Date: Sat, 30 May 2020 16:19:28 -0600 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 34 In-Reply-To: <6DFF8E9C-EB6E-4A67-90E2-631E625F11BA@theatsgroup.com> References: <6DFF8E9C-EB6E-4A67-90E2-631E625F11BA@theatsgroup.com> Message-ID: see below. Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 05:55 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 34 Sent by: gpfsug-discuss-bounces at spectrumscale.org Steve, So in essence, if the minReleaseLevel is set to 5.0.x.x for a Spectrum Scale 5.0 cluster, we can't add any new severs with Spectrum Scale 4.x.x RPMs installed without upgrading them to Spectrum Scale 5.x.x RPMs to the cluster. Is this a correct statement? Steve>>> This statement is true for the local cluster. However, it is pretty simple to put the 4.2.3-X client nodes(currently 4.2.3.21 is min officially recommended and only until Sep 2020) in their own access cluster. Does mmiportfs also check the filesystem version which it is importing and prevent it from importing if it is created on Spectrum Scale 4.x.x? I hope one of the FS experts can correct if wrong, but it would be in my mind a bug if mmimportfs didn't allow this. I know you can create new 4.2.3-X file systems in a cluster with 5.0.X minReleaseLevel and do that routinely. So, since the cluster supports lower version levels (actually I have one filesystem that is a 3.5.0.7 in a cluster that is 5.0.4.2 minReleaseLevel but obviously 3.5.0.7 wouldn't be anywhere near a recommended filesystm level but it does work fine. NOTE: that the remote cluster will not authenticate unless at a minReleaseLevel of 4.2.2.X (not a supported level) so even though the filesystem can be very back level. The remote accessing cluster has to be no more than one major version back or ahead. On 5/29/20, 7:40 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wzUqgJ_KdsODST1fL88GSy_nGwzeJhtx-ikgQdEt310&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: gpfsug-discuss Digest, Vol 100, Issue 32 (Steven Daniels) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 17:39:59 -0600 From: "Steven Daniels" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Message-ID: Content-Type: text/plain; charset="iso-8859-1" Prasad, One of the clients I work with have several 5.0.X ESS (owning) clusters and they share 4.2.3.9 file systems to remote client clusters. The minimum release level on the 5.0.X clusters are at the version of the cluster, i.e. 5.0.3 or 5.0.4 (we are in the process of getting all to latest). The key is that the clients are not part of this cluster as a node has to be > minReleaseLevel to be able to join. They are in their own (accessing) cluster which has a minReleaseLevel of 4.2.3-9. This is the current (until September withdrawal) minReleaseLevel that is supported in this scenario. Don't know specifically about import but it should work as long as the clients that need to mount the sub-5.0 filesystem can be in a separate (remote) cluster or the client nodes are upgraded to the minReleaseLevel of the local cluster. Thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com (See attached file: opencits-d.jpg) From: Prasad Surampudi To: "gpfsug-discuss at spectrumscale.org" Date: 05/29/2020 09:02 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 100, Issue 32 Sent by: gpfsug-discuss-bounces at spectrumscale.org Jim, The minimum release for 5.0.4 cluster is 5.0.4.3. We are aware that we can't merge both gpfs_4 and gpfs_5 filesystems. Our plan is to import the gpfs_4 filesystem into the Spectrum Scale 5.0 cluster (with its NSDs mapped to Spectrum Scale 5.0 clusters nodes) and copy the data using rsync, across fibre channel instead of ethernet to get higher bandwidth. On 5/29/20, 8:27 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org" wrote: Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster (Carl Zetie - carlz at us.ibm.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 29 May 2020 12:25:46 +0000 From: "Carl Zetie - carlz at us.ibm.com" To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Importing a Spectrum Scale a filesystem from 4.2.3 cluster to 5.0.4.3 cluster Message-ID: Content-Type: text/plain; charset="utf-8" On a non-technical note, I wanted to remind everybody that if you?re doing a migration and you need to temporarily run both systems, IBM licensing allows you to do that without ceremony or formality, for up to 90 days under a thing called the ?Temporary Additional Use Policy?. You don?t need to register, ask beforehand, etc. Just go ahead. If you need the additional usage for more than 90 days, contact your seller and we?ll figure something out. Regards Carl Zetie Program Director Offering Management Spectrum Scale ---- (919) 473 3318 ][ Research Triangle Park carlz at us.ibm.com [signature_398516855] -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=u37vFxPtfEbuPwTW6wj7sbTFugW8l6DsRWtag_KO4Bk&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 69558 bytes Desc: image001.png URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_010f20c4_attachment.png&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=YORJyUthJSKVhDxjwEwePTRexsjhX9F3sXPJg_Z5c5g&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= End of gpfsug-discuss Digest, Vol 100, Issue 32 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=WcmE664kj3NZPPiz1qPlCIU0Bf6L44LDM3IvSECU4Ns&s=qqvxYxweD4X5sCAocNwqQpO0wC1NrQV_ekx7sCZao7M&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_0c6df3be_attachment.html&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wo2skeF7H2QM7YNg9RJDB9ntSOGPeaocVSTahs6lCtc&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_0c6df3be_attachment.gif&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wvnV0A6CkiXRcWtU-nWQC_-Li0zXNfBZOW2oADBVg0o&e= > -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20200529_0c6df3be_attachment.jpg&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=NcIKR17cT_NpUQ67OrX_fti7Tdnla2I-Q5Ptx82UP3k&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wzUqgJ_KdsODST1fL88GSy_nGwzeJhtx-ikgQdEt310&e= End of gpfsug-discuss Digest, Vol 100, Issue 34 *********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=XRbZDBaBrsz8ZyuPauqiiziGWdkEuzpnYPDCasYZ6gE&s=wzUqgJ_KdsODST1fL88GSy_nGwzeJhtx-ikgQdEt310&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: opencits-d.jpg Type: image/jpeg Size: 182862 bytes Desc: not available URL: From jonathan.buzzard at strath.ac.uk Sun May 31 10:31:34 2020 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sun, 31 May 2020 10:31:34 +0100 Subject: [gpfsug-discuss] Multi-cluster question (was Re: gpfsug-discuss Digest, Vol 100, Issue 32) In-Reply-To: References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> Message-ID: <53c17f6a-f3fe-9217-4000-197e1b06a105@strath.ac.uk> On 29/05/2020 20:55, Stephen Ulmer wrote: > I have a question about multi-cluster, but it is related to this thread > (it would be solving the same problem). > > Let?s say we have two clusters A and B, both clusters are normally > shared-everything with no NSD servers defined. Er, even in a shared-everything all nodes fibre channel attached you still have to define NSD servers. That is a given NSD has a server (or ideally a list of servers) that arbitrate the disk. Unless it has changed since 3.x days. Never run a 4.x or later with all the disks SAN attached on all the nodes. > We want cluster B to be > able to use a file system in cluster A. ?If I zone the SAN such that > cluster B can see all of cluster A?s disks, can I then define a > multi-cluster relationship between them and mount a file system from A on B? > > To state it another way, must B's I/O for the foreign file system pass > though NSD servers in A, or can B?s nodes discover that they have > FibreChannel paths to those disks and use them? > My understanding is that remote cluster mounts have to pass through the NSD servers. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From janfrode at tanso.net Sun May 31 17:47:40 2020 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 31 May 2020 18:47:40 +0200 Subject: [gpfsug-discuss] Multi-cluster question (was Re: gpfsug-discuss Digest, Vol 100, Issue 32) In-Reply-To: <53c17f6a-f3fe-9217-4000-197e1b06a105@strath.ac.uk> References: <6E77A1C3-3D92-4FFC-B732-9FA56FE6C7ED@theatsgroup.com> <53c17f6a-f3fe-9217-4000-197e1b06a105@strath.ac.uk> Message-ID: No, this is a common misconception. You don?t need any NSD servers. NSD servers are only needed if you have nodes without direct block access. Remote cluster or not, disk access will be over local block device (without involving NSD servers in any way), or NSD server if local access isn?t available. NSD-servers are not ?arbitrators? over access to a disk, they?re just stupid proxies of IO commands. -jf s?n. 31. mai 2020 kl. 11:31 skrev Jonathan Buzzard < jonathan.buzzard at strath.ac.uk>: > On 29/05/2020 20:55, Stephen Ulmer wrote: > > I have a question about multi-cluster, but it is related to this thread > > (it would be solving the same problem). > > > > Let?s say we have two clusters A and B, both clusters are normally > > shared-everything with no NSD servers defined. > > Er, even in a shared-everything all nodes fibre channel attached you > still have to define NSD servers. That is a given NSD has a server (or > ideally a list of servers) that arbitrate the disk. Unless it has > changed since 3.x days. Never run a 4.x or later with all the disks SAN > attached on all the nodes. > > > We want cluster B to be > > able to use a file system in cluster A. If I zone the SAN such that > > cluster B can see all of cluster A?s disks, can I then define a > > multi-cluster relationship between them and mount a file system from A > on B? > > > > To state it another way, must B's I/O for the foreign file system pass > > though NSD servers in A, or can B?s nodes discover that they have > > FibreChannel paths to those disks and use them? > > > > My understanding is that remote cluster mounts have to pass through the > NSD servers. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: