From p.childs at qmul.ac.uk Mon Feb 1 15:08:19 2021 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 1 Feb 2021 15:08:19 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu>, <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> Message-ID: We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn?t really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From daniel.kidger at uk.ibm.com Mon Feb 1 16:25:11 2021 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Mon, 1 Feb 2021 16:25:11 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From Adam.Willis at ibm.com Mon Feb 1 17:15:18 2021 From: Adam.Willis at ibm.com (Adam Willis) Date: Mon, 1 Feb 2021 17:15:18 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From janfrode at tanso.net Mon Feb 1 18:11:26 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 1 Feb 2021 19:11:26 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: Agree.. Write a policy that takes a "mmapplypolicy -M var=val" argument, and figure out the workdays outside of the policy. Something like: # cat test.poilcy define( access_age, (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) /* list migrated files */ RULE EXTERNAL LIST 'oldFiles' EXEC '' RULE 'oldFiles' LIST 'oldFiles' WHERE (access_age > MINAGE) # mmapplypolicy gpfs01 -P test.policy -I defer -f ./filelist -M MINAGE=5 On Mon, Feb 1, 2021 at 5:29 PM Daniel Kidger wrote: > To me, it feels like you need to do the search the other way around. > > Firstly, from the target of say "4 weekdays ago", work out how many real > days ago that was. > Then use that as a criteria in mmfind or mmapplypolicy. > > Daniel > > _________________________________________________________ > *Daniel Kidger Ph.D.* > IBM Technical Sales Specialist > Spectrum Scale, Spectrum Discover and IBM Cloud Object Storage > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > > > > > > > > > ----- Original message ----- > From: Jonathan Buzzard > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, > Issue 18 > Date: Sat, Jan 30, 2021 10:29 > > On 30/01/2021 00:31, Owen Morgan wrote: > > [SNIP] > > > > > I would prefer to stay in the bounds of the SQL policy rule setup as > > that is the framework I have created and started to implement.. > > > In general SQL is Turing complete. Though I have not checked in detail I > believe the SQL of the policy engine is too. > > I would also note that SQL has a whole bunch of time/date functions. > > So something like > > define(offset, 4) > define(day, DAYOFWEEK(CURRENT_TIMESTAMP)) > define(age,(DAYS(CURRENT_TIMESTAMP)-DAYS(ACCESS_TIME))) > define(workingdays, > CASE > WHEN day=1 > THEN offest+1 > WHEN day=6 > THEN offset > WHEN day=7 > THEN offset+1 > ELSE > offset+2 > ) > > /* delete all files from files older than 4 working days */ > RULE purge4 DELETE > WHERE (age>workingdays) FOR FILESET dummies > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From giovanni.bracco at enea.it Mon Feb 1 19:41:43 2021 From: giovanni.bracco at enea.it (Giovanni Bracco) Date: Mon, 1 Feb 2021 20:41:43 +0100 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> Message-ID: <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> On 30/01/21 21:01, Walter Sklenka wrote: > Hi Giovanni! > Thats great! Many thanks for your fast and detailed answer!!!! > So this is the way we will go too! > > Have a nice weekend and keep healthy! > Best regards > Walter > I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works! If yu have any issues I am ready to discuss! Regards Giovanni > -----Original Message----- > From: Giovanni Bracco > Sent: Samstag, 30. J?nner 2021 18:08 > To: gpfsug main discussion list ; Walter Sklenka > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > See > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a multifabric GPFS Spectrum Scale layout," 2019 International Conference on High Performance Computing & Simulation (HPCS), Dublin, Ireland, 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > When setting up the system the main trick has been: > just use CentOS drivers and do not install OFED We do not use IPoIB. > > Giovanni > > On 30/01/21 06:45, Walter Sklenka wrote: >> Hi! >> >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? >> >> In the faq >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. >> html#rdma >> >> >> They talk about RDMA : >> >> "RDMA is NOT ?supported on a node when both Mellanox HCAs and Intel >> Omni-Path HFIs are ENABLED for RDMA." >> >> So do I understand right: When we do NOT enable ?the opa interface we >> can still enable IB ? >> >> The reason I ask ?is, that we have a gpfs cluster of 6 NSD Servers >> (wih access to storage) ?with opa interfaces which provide access to >> remote cluster ?also via OPA. >> >> A new cluster with HDR interfaces will be implemented soon >> >> They shell have access to the same filesystems >> >> When we add HDR interfaces to? NSD servers? and enable rdma on this >> network ?while disabling rdma on opa we would accept the worse >> performance via opa . We hope that this provides ?still better perf >> and less technical overhead ?than using routers >> >> Or am I totally wrong? >> >> Thank you very much and keep healthy! >> >> Best regards >> >> Walter >> >> Mit freundlichen Gr??en >> */Walter Sklenka/* >> */Technical Consultant/* >> >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 >> Wien >> Tel: +43 1 29 22 165-31 >> Fax: +43 1 29 22 165-90 >> E-Mail: sklenka at edv-design.at >> Internet: www.edv-design.at >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Giovanni Bracco > phone +39 351 8804788 > E-mail giovanni.bracco at enea.it > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco From jonathan.buzzard at strath.ac.uk Mon Feb 1 20:17:34 2021 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 1 Feb 2021 20:17:34 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: On 01/02/2021 18:11, Jan-Frode Myklebust wrote: > CAUTION: This email originated outside the University. Check before > clicking links or attachments. > Agree.. Write a policy that takes a "mmapplypolicy -M var=val" argument, > and figure out the workdays outside of the policy. Something like: > > # cat test.poilcy > define( access_age, ? ? (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) > /* list migrated files */ > RULE EXTERNAL LIST 'oldFiles' EXEC '' > RULE 'oldFiles' LIST 'oldFiles' > ? ? WHERE (access_age > MINAGE) > > #?mmapplypolicy gpfs01 ?-P test.policy -I defer -f ./filelist -M MINAGE=5 > Why bother when you can do it all in the policy? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From owen.morgan at motionpicturesolutions.com Mon Feb 1 21:09:16 2021 From: owen.morgan at motionpicturesolutions.com (Owen Morgan) Date: Mon, 1 Feb 2021 21:09:16 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 Message-ID: Jonathan, If I have a single policy file with all the related department rules and each time they want to add additional rules with different working day thresholds maybe using this -M method is easier. Its clear that the 'maths' and date/timestamp manipulation is easier in shell (my preferred is bash) than in the SQL of the policy (your example is succinct but needs to be repeated everytime a new rule is added with a different working day threshold, which is what I'm trying (if possiblr) to avoid. It seems to me the IBM SQL engine is perhaps missing more 'SQL' in built date/time functions like DateAdd and DateDiff etc.. as this would be a moot point. Its a shame I can't make one function that given a working day input as an argument spits out how many 'real' days exist between them for the file age comparison all in the SQL. It can be done for 1 specific input argument, but needs the whole function repeated manually for a different input argument, and further repeated for a different argument etc.. Maybe I'm also compounding the issue by trying to make the policy file as concise as possible (for sake of clarity as to what the rules are trying to achieve, and easy expandability), and demanding too much of the SQL-like syntax that IBM have created. I have options for mmfind or even (as suggested) -M inoput to mmapplypolicy where I us bash to create a small function that does what I need, spits out 'real days' given a working day input, and using arrays and for-loop create a dynamic calling of the mmapplypolicy command (which I'm kinda half doing anyways for other reasons in my launcher script. As always, I'm seriously amazed at people with soo much experience and knowledge taking time out to help, guide, and offer input like everyone has been doing!! I'm relatively early in my career, so being able to interact and learn from experienced persons is giving me such a wider insight! Thanks! Owen. [Sent from Front] On 1 February 2021, 20:17 GMT jonathan.buzzard at strath.ac.uk wrote: On 01/02/2021 18:11, Jan-Frode Myklebust wrote: > CAUTION: This email originated outside the University. Check before > clicking links or attachments. > Agree.. Write a policy that takes a "mmapplypolicy -M var=val" argument, > and figure out the workdays outside of the policy. Something like: > > # cat test.poilcy > define( access_age, (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) > /* list migrated files */ > RULE EXTERNAL LIST 'oldFiles' EXEC '' > RULE 'oldFiles' LIST 'oldFiles' > WHERE (access_age > MINAGE) > > # mmapplypolicy gpfs01 -P test.policy -I defer -f ./filelist -M MINAGE=5 > Why bother when you can do it all in the policy? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG Owen Morgan Data Wrangler Motion Picture Solutions Ltd T: E: owen.morgan at motionpicturesolutions.com | W: motionpicturesolutions.com A: Mission Hall, 9-11 North End Road, London, W14 8ST Motion Picture Solutions Ltd is a company registered in England and Wales under number 5388229, VAT number 201330482 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Mon Feb 1 21:43:43 2021 From: jtucker at pixitmedia.com (Jez Tucker) Date: Mon, 1 Feb 2021 21:43:43 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: <02351047-3b02-352f-48bc-b4b3e0e03918@pixitmedia.com> Hi Owen, ? This is a great thread and raises as is usual with GPFS that there is more often than not many tools in the bag and more than one way to achieve a requirement. If you stick with policy (indeed worthwhile aspect of GPFS to learn well) you can also build up libraries of centralised macros and reference them into any policy file alike so at the top of the file: include(/mmfs1/policies/macros/general_macros.m4) include(/mmfs1/policies/macros/pixit_excludes.m4) Check your PixStor system under /mmfs1/policies/.. The .m4 extension is not required, but a nod to https://www.gnu.org/savannah-checkouts/gnu/m4/manual/m4-1.4.18/m4.html which the policy engine also encapsulates.? Marc Kaplan once told me "You can do just about anything with m4 and cleverness and patience...", though he failed to mention the required amounts of coffee and pizza ;-)? Search the list digests for his valuable insights over the years. Alternatively you can achieve all this and more using the Python API https://www.arcapix.com/pixstorapi/index.html (especially your date processing). If you'd like direct support with any this, ping us over an email via support@ - alternatively there are indeed many great minds a wealth of experience and views on this list (and it's nice to meet the community too). SSUG also has a Slack channel too.. ssug-poweraiug.slack.com Most of all - have fun learning. Kind regards, Jez p.s. yes the threading has all gone wonky. ah well :-) On 01/02/2021 21:09, Owen Morgan wrote: > Jonathan, > > If I have a single policy file with all the related department rules > and each time they want to add additional rules with different working > day thresholds maybe using this -M method is easier. Its clear that > the 'maths' and date/timestamp manipulation is easier in shell (my > preferred is bash) than in the SQL of the policy (your example is > succinct but needs to be repeated everytime a new rule is added with a > different working day threshold, which is what I'm trying (if > possiblr) to avoid. > > It seems to me the IBM SQL engine is perhaps missing more 'SQL' in > built date/time functions like DateAdd and DateDiff etc..? as this > would be a moot point. Its a shame I can't make one function that > given a working day input as an argument spits out how many 'real' > days exist between them for the file age comparison all in the SQL. It > can be done for 1 specific input argument, but needs the whole > function repeated manually for a different input argument, and further > repeated for a different argument etc.. > > Maybe I'm also compounding the issue by trying to make the policy file > as concise as possible (for sake of clarity as to what the rules are > trying to achieve, and easy expandability), and demanding too much of > the SQL-like syntax that IBM have created. > > I have options for mmfind or even (as suggested) -M inoput to > mmapplypolicy where I us bash to create a small function that does > what I need, spits out 'real days' given a working day input, and > using arrays and for-loop create a dynamic calling of the > mmapplypolicy command (which I'm kinda half doing anyways for other > reasons in my launcher script. > > > As always, I'm seriously amazed at people with soo much experience and > knowledge taking time out to help, guide, and offer input like > everyone has been doing!! I'm relatively early in my career, so being > able to interact and learn from experienced persons is giving me such > a wider insight! > > Thanks! > > Owen. > Sent from Front > ? ? > > Owen?Morgan? > Data?Wrangler > Motion?Picture?Solutions?Ltd > T:?** > > E:?*owen.morgan at motionpicturesolutions.com* > ?|? > W:?*motionpicturesolutions.com* > > A:? Mission?Hall,?9?11?North?End?Road ,? London ,? W14?8ST > > Motion Picture Solutions Ltd is a company registered in England and > Wales under number 5388229, VAT number 201330482 > >> On 1 February 2021, 20:17 GMT jonathan.buzzard at strath.ac.uk >> wrote: >> >> On 01/02/2021 18:11, Jan-Frode Myklebust wrote: >>> >>> > CAUTION: This email originated outside the University. Check before >>> > clicking links or attachments. >>> > Agree.. Write a policy that takes a "mmapplypolicy -M var=val" >>> argument, >>> > and figure out the workdays outside of the policy. Something like: >>> > >>> > # cat test.poilcy >>> > define( access_age, ? ? (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) >>> > /* list migrated files */ >>> > RULE EXTERNAL LIST 'oldFiles' EXEC '' >>> > RULE 'oldFiles' LIST 'oldFiles' >>> > ? ? WHERE (access_age > MINAGE) >>> > >>> > #?mmapplypolicy gpfs01 ?-P test.policy -I defer -f ./filelist -M >>> MINAGE=5 >>> > >>> >>> Why bother when you can do it all in the policy? >>> >>> JAB. >>> >>> -- >>> Jonathan A. Buzzard Tel: +44141-5483420 >>> HPC System Administrator, ARCHIE-WeSt. >>> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG >>> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- *Jez Tucker* VP Research and Development | Pixit Media m: +44 (0) 776 419 3820 e: jtucker at pixitmedia.com Visit www.pixitmedia.com -- This email is confidential in that it is?intended for the exclusive attention of?the addressee(s) indicated. If you are?not the intended recipient, this email?should not be read or disclosed to?any other person. Please notify the?sender immediately and delete this?email from your computer system.?Any opinions expressed are not?necessarily those of the company?from which this email was sent and,?whilst to the best of our knowledge no?viruses or defects exist, no?responsibility can be accepted for any?loss or damage arising from its?receipt or subsequent use of this? email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon Feb 1 21:58:34 2021 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 1 Feb 2021 21:58:34 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: <3e2d8556-72a3-930f-dc41-c15ffac1e495@strath.ac.uk> On 01/02/2021 21:09, Owen Morgan wrote: > CAUTION: This email originated outside the University. Check before > clicking links or attachments. > Jonathan, > > If I have a single policy file with all the related department rules and > each time they want to add additional rules with different working day > thresholds maybe using this -M method is easier. Its clear that the > 'maths' and date/timestamp manipulation is easier in shell (my preferred > is bash) than in the SQL of the policy (your example is succinct but > needs to be repeated everytime a new rule is added with a different > working day threshold, which is what I'm trying (if possiblr) to avoid. > I actually think there is a much better way to achieve it than the case statement, but that would have required me to do some more thinking and testing to make sure it worked ;-) Anyway how many different "working" days old do you need? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From anacreo at gmail.com Mon Feb 1 22:14:39 2021 From: anacreo at gmail.com (Alec) Date: Mon, 1 Feb 2021 14:14:39 -0800 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: <3e2d8556-72a3-930f-dc41-c15ffac1e495@strath.ac.uk> References: <3e2d8556-72a3-930f-dc41-c15ffac1e495@strath.ac.uk> Message-ID: In our environment managing the GPFS policies (or anything GPFS) are a bit of an Art form and really only a few people out of a sea of engineers can/will do it. So I like to keep the GPFS policies simple and durable. If you really wanted to you could generate a list of every US (or whatever country) holiday you wanted and use Python, Awk, whatever to produce an exact number of business days versus not; You can also use directory structure, GPFS attributes in parent directory, or a config file to maintain the requirements, which may be more manageable by other team members. Anyhow my point being don't just look at the technical of how to implement, but what is the most maintainable, .documentable, and recoverable solution. Some things I would consider in my design? * Supportability - What is the skill level of the engineer required to maintain the system. * Serviceability - What component would change most frequently, what's the complexity and impact of a change. * Fail Safe - If a component fails, what will the negative impact be. * Trackability - Are all the activities logged. * Sanity Checking - Can you define safety mechanisms to prevent a failure. For instance if your purge script detects that there are 10k+ files to purge, can you interrupt the process? For GPFS we try to keep the service components simple, and the logic in shell scripting where there is a plethora of knowledgeable engineers. So my attack would be like this: * Policy runs, purges any file with purgeafter beyond the current date (maybe validate that the purge.date and modify/access date agree still). <-- is simple * I would have it run the logic twice, and bomb if a safety threshhold is exceeded. Should log the results. * Config in /gpfs/configs/purge_configuration /some/path 10bd /some/scratchspace 30d /some/reallybigfiles 1d * mmfind $(awk '{print $1}' < gpfs/configs/purge_configuration) -type f --exec /gpfs/scripts/setpurgedates * Could also be a regular mmapplypolicy command as well... * /gpfs/scripts/setpurgedates can rescan /gpfs/configs/purge_configuration and read in the date, check against the path passed, and then perform the necessary date logic, and exec the mmchattr, preserve the file's modify/access time to a purge.date time.. * Could log its decisions to a log file. Happy Hunting, I've already learned quite a few things from monitoring this list in a very short period of time. * If a candidate ever tells me they know everything about Unix... I pass. We're all very experienced noobies in this ever progressing field/platform. * Alec On Mon, Feb 1, 2021 at 1:58 PM Jonathan Buzzard < jonathan.buzzard at strath.ac.uk> wrote: > On 01/02/2021 21:09, Owen Morgan wrote: > > CAUTION: This email originated outside the University. Check before > > clicking links or attachments. > > Jonathan, > > > > If I have a single policy file with all the related department rules and > > each time they want to add additional rules with different working day > > thresholds maybe using this -M method is easier. Its clear that the > > 'maths' and date/timestamp manipulation is easier in shell (my preferred > > is bash) than in the SQL of the policy (your example is succinct but > > needs to be repeated everytime a new rule is added with a different > > working day threshold, which is what I'm trying (if possiblr) to avoid. > > > > I actually think there is a much better way to achieve it than the case > statement, but that would have required me to do some more thinking and > testing to make sure it worked ;-) > > Anyway how many different "working" days old do you need? > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Walter.Sklenka at EDV-Design.at Tue Feb 2 13:10:05 2021 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Tue, 2 Feb 2021 13:10:05 +0000 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> Message-ID: <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> Hi Giovanni! Thank you for your offer! ? it is planned to be implemented in June or so We will use RHEL 8.x and newest gpfs version available Only one question for this moment if I am allowed: Did you ever ran into any problems with IBM support? I mean they say in the FAQ shortly "not supported" , but do they in your environment or do you accept that rdma problems would be needed to be fixed without IBM Thank you very much and have great days! And keep healthy! Best regards walter -----Original Message----- From: Giovanni Bracco Sent: Montag, 1. Februar 2021 20:42 To: Walter Sklenka Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled On 30/01/21 21:01, Walter Sklenka wrote: > Hi Giovanni! > Thats great! Many thanks for your fast and detailed answer!!!! > So this is the way we will go too! > > Have a nice weekend and keep healthy! > Best regards > Walter > I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works! If yu have any issues I am ready to discuss! Regards Giovanni > -----Original Message----- > From: Giovanni Bracco > > Sent: Samstag, 30. J?nner 2021 18:08 > To: gpfsug main discussion list >; > Walter Sklenka > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > See > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a > multifabric GPFS Spectrum Scale layout," 2019 International Conference > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > When setting up the system the main trick has been: > just use CentOS drivers and do not install OFED We do not use IPoIB. > > Giovanni > > On 30/01/21 06:45, Walter Sklenka wrote: >> Hi! >> >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? >> >> In the faq >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. >> html#rdma >> >> >> They talk about RDMA : >> >> "RDMA is NOT supported on a node when both Mellanox HCAs and Intel >> Omni-Path HFIs are ENABLED for RDMA." >> >> So do I understand right: When we do NOT enable the opa interface we >> can still enable IB ? >> >> The reason I ask is, that we have a gpfs cluster of 6 NSD Servers >> (wih access to storage) with opa interfaces which provide access to >> remote cluster also via OPA. >> >> A new cluster with HDR interfaces will be implemented soon >> >> They shell have access to the same filesystems >> >> When we add HDR interfaces to NSD servers and enable rdma on this >> network while disabling rdma on opa we would accept the worse >> performance via opa . We hope that this provides still better perf >> and less technical overhead than using routers >> >> Or am I totally wrong? >> >> Thank you very much and keep healthy! >> >> Best regards >> >> Walter >> >> Mit freundlichen Gr??en >> */Walter Sklenka/* >> */Technical Consultant/* >> >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 >> Wien >> Tel: +43 1 29 22 165-31 >> Fax: +43 1 29 22 165-90 >> E-Mail: sklenka at edv-design.at >> Internet: www.edv-design.at >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Giovanni Bracco > phone +39 351 8804788 > E-mail giovanni.bracco at enea.it > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco -------------- next part -------------- An HTML attachment was scrubbed... URL: From Walter.Sklenka at EDV-Design.at Tue Feb 2 13:19:37 2021 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Tue, 2 Feb 2021 13:19:37 +0000 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> Message-ID: Hi Giovanni! Thank you very much for your offer , we really would be very grateful to be allowed to come if we run into troubles! Well, the implementation will not happen before June or later, but may I ask only one question meanwhile? Did you ever run into problems with IBM support or did you get a special ?OK? from them? Or do you accept to sove any rdma specific problems without support ? (it?s only because of the FAQ ?not supported? ) Have a great day and keep healthy! Best regards walter -----Original Message----- From: Giovanni Bracco Sent: Montag, 1. Februar 2021 20:42 To: Walter Sklenka Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled On 30/01/21 21:01, Walter Sklenka wrote: > Hi Giovanni! > Thats great! Many thanks for your fast and detailed answer!!!! > So this is the way we will go too! > > Have a nice weekend and keep healthy! > Best regards > Walter > I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works! If yu have any issues I am ready to discuss! Regards Giovanni > -----Original Message----- > From: Giovanni Bracco > > Sent: Samstag, 30. J?nner 2021 18:08 > To: gpfsug main discussion list >; > Walter Sklenka > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > See > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a > multifabric GPFS Spectrum Scale layout," 2019 International Conference > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > When setting up the system the main trick has been: > just use CentOS drivers and do not install OFED We do not use IPoIB. > > Giovanni > > On 30/01/21 06:45, Walter Sklenka wrote: >> Hi! >> >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? >> >> In the faq >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. >> html#rdma >> >> >> They talk about RDMA : >> >> "RDMA is NOT supported on a node when both Mellanox HCAs and Intel >> Omni-Path HFIs are ENABLED for RDMA." >> >> So do I understand right: When we do NOT enable the opa interface we >> can still enable IB ? >> >> The reason I ask is, that we have a gpfs cluster of 6 NSD Servers >> (wih access to storage) with opa interfaces which provide access to >> remote cluster also via OPA. >> >> A new cluster with HDR interfaces will be implemented soon >> >> They shell have access to the same filesystems >> >> When we add HDR interfaces to NSD servers and enable rdma on this >> network while disabling rdma on opa we would accept the worse >> performance via opa . We hope that this provides still better perf >> and less technical overhead than using routers >> >> Or am I totally wrong? >> >> Thank you very much and keep healthy! >> >> Best regards >> >> Walter >> >> Mit freundlichen Gr??en >> */Walter Sklenka/* >> */Technical Consultant/* >> >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 >> Wien >> Tel: +43 1 29 22 165-31 >> Fax: +43 1 29 22 165-90 >> E-Mail: sklenka at edv-design.at >> Internet: www.edv-design.at >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Giovanni Bracco > phone +39 351 8804788 > E-mail giovanni.bracco at enea.it > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Tue Feb 2 16:08:47 2021 From: ewahl at osc.edu (Wahl, Edward) Date: Tue, 2 Feb 2021 16:08:47 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180202152336.03e8bab7@osc.edu> References: <20180119163803.79fddbeb@osc.edu> , <20180202152336.03e8bab7@osc.edu> Message-ID: Replying to a 3 year old message I sent, hoping that in the last couple of years that Scale has added some ILM extensions into the policy engine that I have missed, or somehow didn't notice? Just ran into a file with an 'unbalanced' flag and I REALLY don't want to have to mmlsattr everything. AGAIN. /facepalm IBM? Bueller? Bueller? When everyone answers: "No", I'm guessing this needs to be a request for improvement/enhancement? Ed Wahl Ohio Supercomputer Center ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Edward Wahl Sent: Friday, February 2, 2018 3:23 PM To: John Hearns Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] policy ilm features? Thanks John, this was the path I was HOPING to go down as I do similar things already, but there appears to be no extended attribute in ILM for what I want. Data block replication flag exists in the ILM, but not MetaData, or balance. Yet these states ARE reported by mmlsattr, so there must be a flag somewhere. bad MD replication & balance example: mmlsattr -L /fs/scratch/sysp/ed/180days.pol file name: /fs/scratch/sysp/ed/180days.pol metadata replication: 1 max 2 data replication: 1 max 2 flags: illreplicated,unbalanced Encrypted: yes File next to it for comparison. note proper MD replication and balance. mmlsattr -L /fs/scratch/sysp/ed/120days.pol file name: /fs/scratch/sysp/ed/120days.pol metadata replication: 2 max 2 data replication: 1 max 2 flags: Encrypted: yes misc_attributes flags from a policy run showing no difference in status: FJAEu -- /fs/scratch/sysp/ed/180days.pol FJAEu -- /fs/scratch/sysp/ed/120days.pol File system has MD replication enabled, but not Data, so ALL files show "J" ilm flag mmlsfs scratch -m flag value description ------------------- ------------------------ ----------------------------------- -m 2 Default number of metadata replicas mmlsfs scratch -r flag value description ------------------- ------------------------ ----------------------------------- -r 1 Default number of data replicas I poked around a little trying to find out if perhaps using GetXattr would work and show me what I wanted, it does not. All I sem to be able to get is the File Encryption Key. I was hoping perhaps someone had found a cheaper way for this to work rather than hundreds of millions of 'mmlsattr' execs. :-( On the plus side, I've only run across a few of these and all appear to be from before we did the MD replication and re-striping. On the minus, I have NO idea where they are, and they appears to be on both of our filesystems. So several hundred million files to check. Ed On Mon, 22 Jan 2018 08:29:42 +0000 John Hearns wrote: > Ed, > This is not a perfect answer. You need to look at policies for this. I have > been doing something similar recently. > > Something like: > > RULE 'list_file' EXTERNAL LIST 'all-files' EXEC > '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files' > SHOW( varchar(kb_allocated) || ' ' || varchar(file_size) || ' ' || > varchar(misc_attributes) || ' ' || name || ' ' || fileset_name ) WHERE > REGEX(misc_attributes,'[J]') > > > So this policy shows the kbytes allocates, file size, the miscellaneous > attributes, name and fileset name For all files with miscellaneous > attributes of 'J' which means 'Some data blocks might be ill replicated' > > > > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Edward Wahl > Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] policy ilm features? > > > This one has been on my list a long time so I figured I'd ask here first > before I open an apar or request an enhancement (most likely). > > Is there a way using the policy engine to determine the following? > > -metadata replication total/current > -unbalanced file > > Looking to catch things like this that stand out on my filesystem without > having to run several hundred million 'mmlsattr's. > > metadata replication: 1 max 2 > flags: unbalanced > > Ed > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D&reserved=0 > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is provided > on an AS-IS basis without any express or implied warranties or liabilities. > To the extent you are relying on this information, you are doing so at your > own risk. If you are not the intended recipient, please notify the sender > immediately by replying to this message and destroy all copies of this > message and any attachments. Neither the sender nor the company/group of > companies he or she represents shall be liable for the proper and complete > transmission of the information contained in this communication, or for any > delay in its receipt. _______________________________________________ > gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Ed Wahl Ohio Supercomputer Center 614-292-9302 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Tue Feb 2 18:09:28 2021 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 2 Feb 2021 18:09:28 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: References: , <20180119163803.79fddbeb@osc.edu>, <20180202152336.03e8bab7@osc.edu> Message-ID: An HTML attachment was scrubbed... URL: From ewahl at osc.edu Tue Feb 2 21:26:42 2021 From: ewahl at osc.edu (Wahl, Edward) Date: Tue, 2 Feb 2021 21:26:42 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: References: , <20180119163803.79fddbeb@osc.edu>, <20180202152336.03e8bab7@osc.edu>, Message-ID: My issues were never specifically the Unbalanced flag, originally I ran into ilm/policy issues with our metadata and illreplicated files. We were working to expand our Metadata storage at the time uner 4.2.3.xx and had added a number of SSDs to the array. But AFTER a restripe files were still illreplicated. I then discovered that the policy engine had no way to tell me what files were and were not replicated. Just hoping that more ILM info had been added to the policy engine since I ran into this, and seeing the unbalanced files jogged my memory to look at the docs again, where I didn't see anything in 5.x and ask the list as well. This isn't a big deal, more curiousity on my part. I see you have some of the original thread attached, so perhaps take a glance and see if it makes sense? I Now if you REALLY want to step into a mine field, go find my thread on SKLM usage and GPFS/SS. Every single respondent to my question about filling up their SKLM logs with errors was a positive. And SKLM L2 and L3 support swears GPFS/SS is using SKLM wrong... Ed Wahl OSC ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederick Stock Sent: Tuesday, February 2, 2021 1:09 PM To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] policy ilm features? Hello Ed. Jordan contacted me about the question you are posing so I am responding to your message. Could you please provide clarification as to why the existence of the unbalanced flag is of a concern, or why you would want to know all the files that have this flag set? The flag would be cleared once the file was rebalanced either through normal access or through the execution of the mmrestripefs/mmrestripefile commands. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com ----- Original message ----- From: "Wahl, Edward" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] policy ilm features? Date: Tue, Feb 2, 2021 11:52 AM Replying to a 3 year old message I sent, hoping that in the last couple of years that Scale has added some ILM extensions into the policy engine that I have missed, or somehow didn't notice? Just ran into a file with an 'unbalanced' flag and I REALLY don't want to have to mmlsattr everything. AGAIN. /facepalm IBM? Bueller? Bueller? When everyone answers: "No", I'm guessing this needs to be a request for improvement/enhancement? Ed Wahl Ohio Supercomputer Center ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Edward Wahl Sent: Friday, February 2, 2018 3:23 PM To: John Hearns Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] policy ilm features? Thanks John, this was the path I was HOPING to go down as I do similar things already, but there appears to be no extended attribute in ILM for what I want. Data block replication flag exists in the ILM, but not MetaData, or balance. Yet these states ARE reported by mmlsattr, so there must be a flag somewhere. bad MD replication & balance example: mmlsattr -L /fs/scratch/sysp/ed/180days.pol file name: /fs/scratch/sysp/ed/180days.pol metadata replication: 1 max 2 data replication: 1 max 2 flags: illreplicated,unbalanced Encrypted: yes File next to it for comparison. note proper MD replication and balance. mmlsattr -L /fs/scratch/sysp/ed/120days.pol file name: /fs/scratch/sysp/ed/120days.pol metadata replication: 2 max 2 data replication: 1 max 2 flags: Encrypted: yes misc_attributes flags from a policy run showing no difference in status: FJAEu -- /fs/scratch/sysp/ed/180days.pol FJAEu -- /fs/scratch/sysp/ed/120days.pol File system has MD replication enabled, but not Data, so ALL files show "J" ilm flag mmlsfs scratch -m flag value description ------------------- ------------------------ ----------------------------------- -m 2 Default number of metadata replicas mmlsfs scratch -r flag value description ------------------- ------------------------ ----------------------------------- -r 1 Default number of data replicas I poked around a little trying to find out if perhaps using GetXattr would work and show me what I wanted, it does not. All I sem to be able to get is the File Encryption Key. I was hoping perhaps someone had found a cheaper way for this to work rather than hundreds of millions of 'mmlsattr' execs. :-( On the plus side, I've only run across a few of these and all appear to be from before we did the MD replication and re-striping. On the minus, I have NO idea where they are, and they appears to be on both of our filesystems. So several hundred million files to check. Ed On Mon, 22 Jan 2018 08:29:42 +0000 John Hearns wrote: > Ed, > This is not a perfect answer. You need to look at policies for this. I have > been doing something similar recently. > > Something like: > > RULE 'list_file' EXTERNAL LIST 'all-files' EXEC > '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files' > SHOW( varchar(kb_allocated) || ' ' || varchar(file_size) || ' ' || > varchar(misc_attributes) || ' ' || name || ' ' || fileset_name ) WHERE > REGEX(misc_attributes,'[J]') > > > So this policy shows the kbytes allocates, file size, the miscellaneous > attributes, name and fileset name For all files with miscellaneous > attributes of 'J' which means 'Some data blocks might be ill replicated' > > > > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Edward Wahl > Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] policy ilm features? > > > This one has been on my list a long time so I figured I'd ask here first > before I open an apar or request an enhancement (most likely). > > Is there a way using the policy engine to determine the following? > > -metadata replication total/current > -unbalanced file > > Looking to catch things like this that stand out on my filesystem without > having to run several hundred million 'mmlsattr's. > > metadata replication: 1 max 2 > flags: unbalanced > > Ed > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D&reserved=0 > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is provided > on an AS-IS basis without any express or implied warranties or liabilities. > To the extent you are relying on this information, you are doing so at your > own risk. If you are not the intended recipient, please notify the sender > immediately by replying to this message and destroy all copies of this > message and any attachments. Neither the sender nor the company/group of > companies he or she represents shall be liable for the proper and complete > transmission of the information contained in this communication, or for any > delay in its receipt. _______________________________________________ > gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Ed Wahl Ohio Supercomputer Center 614-292-9302 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From giovanni.bracco at enea.it Wed Feb 3 08:58:37 2021 From: giovanni.bracco at enea.it (Giovanni Bracco) Date: Wed, 3 Feb 2021 09:58:37 +0100 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> Message-ID: <0ecee41c-9a9b-83d4-59ae-2b0fbda526fb@enea.it> We did not explore the issue of the IBM support and for budget limitation and for the mandatory integration of the data space between the two clusters, we decided to try the setup of the multi-fabric infrastructure and up to now it has been working without problems. Giovanni On 02/02/21 14:10, Walter Sklenka wrote: > Hi Giovanni! > > Thank you for your offer! ? > > it is planned to be implemented in June or so > > We will use RHEL 8.x and newest gpfs version available > > Only one question for this moment if I am allowed: > > Did you ever ran into any problems with IBM support? I mean they say in > the FAQ shortly "not supported" , but do they in your environment or do > you accept that rdma problems would be needed to be fixed without IBM > > Thank you very much and have great days! And keep healthy! > > Best regards walter > > -----Original Message----- > From: Giovanni Bracco > Sent: Montag, 1. Februar 2021 20:42 > To: Walter Sklenka > Cc: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > On 30/01/21 21:01, Walter Sklenka wrote: > > > Hi Giovanni! > > > Thats great! Many thanks for your fast and detailed answer!!!! > > > So this is the way we will go too! > > > > > > Have a nice weekend and keep healthy! > > > Best regards > > > Walter > > > > > I suppose you will implement the solution with more recent versions of > the software components, so please let me know if everything works! > > If yu have any issues I am ready to discuss! > > Regards > > Giovanni > > > -----Original Message----- > > > From: Giovanni Bracco > > > > Sent: Samstag, 30. J?nner 2021 18:08 > > > To: gpfsug main discussion list >; > > > Walter Sklenka > > > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > > > Server with only ib rdma enabled > > > > > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, > each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes > SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA > Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN > storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > > > See > > > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a > > > multifabric GPFS Spectrum Scale layout," 2019 International Conference > > > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > > > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > > > > > When setting up the system the main trick has been: > > > just use CentOS drivers and do not install OFED We do not use IPoIB. > > > > > > Giovanni > > > > > > On 30/01/21 06:45, Walter Sklenka wrote: > > >> Hi! > > >> > > >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? > > >> > > >> In the faq > > >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. > > >> html#rdma > > >> > > >> > > >> They talk about RDMA : > > >> > > >> "RDMA is NOT ?supported on a node when both Mellanox HCAs and Intel > > >> Omni-Path HFIs are ENABLED for RDMA." > > >> > > >> So do I understand right: When we do NOT enable ?the opa interface we > > >> can still enable IB ? > > >> > > >> The reason I ask ?is, that we have a gpfs cluster of 6 NSD Servers > > >> (wih access to storage) ?with opa interfaces which provide access to > > >> remote cluster ?also via OPA. > > >> > > >> A new cluster with HDR interfaces will be implemented soon > > >> > > >> They shell have access to the same filesystems > > >> > > >> When we add HDR interfaces to? NSD servers? and enable rdma on this > > >> network ?while disabling rdma on opa we would accept the worse > > >> performance via opa . We hope that this provides ?still better perf > > >> and less technical overhead ?than using routers > > >> > > >> Or am I totally wrong? > > >> > > >> Thank you very much and keep healthy! > > >> > > >> Best regards > > >> > > >> Walter > > >> > > >> Mit freundlichen Gr??en > > >> */Walter Sklenka/* > > >> */Technical Consultant/* > > >> > > >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 > > >> Wien > > >> Tel: +43 1 29 22 165-31 > > >> Fax: +43 1 29 22 165-90 > > >> E-Mail: sklenka at edv-design.at > > > >> Internet: www.edv-design.at > > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > -- > > > Giovanni Bracco > > > phone? +39 351 8804788 > > > E-mail giovanni.bracco at enea.it > > > WWW http://www.afs.enea.it/bracco > > > > > -- > > Giovanni Bracco > > phone? +39 351 8804788 > > E-mail giovanni.bracco at enea.it > > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco From Walter.Sklenka at EDV-Design.at Wed Feb 3 15:21:18 2021 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Wed, 3 Feb 2021 15:21:18 +0000 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <0ecee41c-9a9b-83d4-59ae-2b0fbda526fb@enea.it> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> <0ecee41c-9a9b-83d4-59ae-2b0fbda526fb@enea.it> Message-ID: Hi Givanni ! I understand and am convinced that the is an excellent solution !! Thank you very much! -----Original Message----- From: Giovanni Bracco Sent: Mittwoch, 3. Februar 2021 09:59 To: Walter Sklenka Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled We did not explore the issue of the IBM support and for budget limitation and for the mandatory integration of the data space between the two clusters, we decided to try the setup of the multi-fabric infrastructure and up to now it has been working without problems. Giovanni On 02/02/21 14:10, Walter Sklenka wrote: > Hi Giovanni! > > Thank you for your offer! ? > > it is planned to be implemented in June or so > > We will use RHEL 8.x and newest gpfs version available > > Only one question for this moment if I am allowed: > > Did you ever ran into any problems with IBM support? I mean they say > in the FAQ shortly "not supported" , but do they in your environment > or do you accept that rdma problems would be needed to be fixed > without IBM > > Thank you very much and have great days! And keep healthy! > > Best regards walter > > -----Original Message----- > From: Giovanni Bracco > Sent: Montag, 1. Februar 2021 20:42 > To: Walter Sklenka > Cc: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > On 30/01/21 21:01, Walter Sklenka wrote: > > > Hi Giovanni! > > > Thats great! Many thanks for your fast and detailed answer!!!! > > > So this is the way we will go too! > > > > > > Have a nice weekend and keep healthy! > > > Best regards > > > Walter > > > > > I suppose you will implement the solution with more recent versions of > the software components, so please let me know if everything works! > > If yu have any issues I am ready to discuss! > > Regards > > Giovanni > > > -----Original Message----- > > > From: Giovanni Bracco > > > > Sent: Samstag, 30. J?nner 2021 18:08 > > > To: gpfsug main discussion list >; > > > Walter Sklenka > > > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > > > Server with only ib rdma enabled > > > > > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, > each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes > SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main > OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to > DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > > > See > > > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of > a > > > multifabric GPFS Spectrum Scale layout," 2019 International > Conference > > > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > > > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > > > > > When setting up the system the main trick has been: > > > just use CentOS drivers and do not install OFED We do not use IPoIB. > > > > > > Giovanni > > > > > > On 30/01/21 06:45, Walter Sklenka wrote: > > >> Hi! > > >> > > >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? > > >> > > >> In the faq > > >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. > > >> html#rdma > > >> > > >> > > >> They talk about RDMA : > > >> > > >> "RDMA is NOT ?supported on a node when both Mellanox HCAs and > Intel > > >> Omni-Path HFIs are ENABLED for RDMA." > > >> > > >> So do I understand right: When we do NOT enable ?the opa interface > we > > >> can still enable IB ? > > >> > > >> The reason I ask ?is, that we have a gpfs cluster of 6 NSD Servers > > >> (wih access to storage) ?with opa interfaces which provide access > to > > >> remote cluster ?also via OPA. > > >> > > >> A new cluster with HDR interfaces will be implemented soon > > >> > > >> They shell have access to the same filesystems > > >> > > >> When we add HDR interfaces to? NSD servers? and enable rdma on > this > > >> network ?while disabling rdma on opa we would accept the worse > > >> performance via opa . We hope that this provides ?still better > perf > > >> and less technical overhead ?than using routers > > >> > > >> Or am I totally wrong? > > >> > > >> Thank you very much and keep healthy! > > >> > > >> Best regards > > >> > > >> Walter > > >> > > >> Mit freundlichen Gr??en > > >> */Walter Sklenka/* > > >> */Technical Consultant/* > > >> > > >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 > > >> Wien > > >> Tel: +43 1 29 22 165-31 > > >> Fax: +43 1 29 22 165-90 > > >> E-Mail: sklenka at edv-design.at > > > >> Internet: www.edv-design.at > > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > -- > > > Giovanni Bracco > > > phone? +39 351 8804788 > > > E-mail giovanni.bracco at enea.it > > > WWW http://www.afs.enea.it/bracco > > > > > -- > > Giovanni Bracco > > phone? +39 351 8804788 > > E-mail giovanni.bracco at enea.it > > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco From henrik at morsing.cc Wed Feb 3 16:18:44 2021 From: henrik at morsing.cc (Henrik Morsing) Date: Wed, 3 Feb 2021 16:18:44 +0000 Subject: [gpfsug-discuss] Intro Message-ID: <20210203161844.GO14457@morsing.cc> Hi all, I live in Buckinghamshire in the UK, but work in portsmouth for the energy company SSE. Some years ago, we bought in two 840TB Spectrum Scale systems to replace our tape libraries. I was put on the project to get it up and running and the tape library phased out, but the project has stalled multiple times. I am back on the project now, every time it feels like I am starting over, and on top I need to patch it again which was a massive undertaking last time, so I am not looking forward to that. The clusters are connected via a six link 10Gb/s etherchannel to our Spectrum Protect backup servers. Two of these have been setup, including cross-site replication job, so we do have some idea of it working, but performance monitoring is another aspect I need to look at. I am planning to install njmon+Grafana to see how that looks. Hope to get lots of tips from this forum! Regards, Henrik Morsing From jonathan.b.mills at nasa.gov Mon Feb 8 16:00:20 2021 From: jonathan.b.mills at nasa.gov (Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC]) Date: Mon, 8 Feb 2021 16:00:20 +0000 Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Message-ID: Dear Community, We have multiple GPFS clusters within our internal networks. We want to expose more than one GPFS GUI interface, from behind a shared web proxy / firewall setup, which has SAML2 integration with our organization's IDP. (The SAML2 is not part of the question, it's just a rationale for doing it this way.) Essentially, we're trying to do something like https:///gpfs/gui1 and https:///gpfs/gui2 The websphere config in the GPFS GUI packages basically assume a document root of "/". This poses additional problems when it forms relative URLs...for example, immediately after login, it tries to redirect from "/" to "/gui". Through our web proxy, this ends up being https:///gui which is a 404. While problems might be addressable with some complex mod_rewrite in our web proxy, I've been looking for a more straightforward solution. In /opt/ibm/wlp/usr/servers/gpfsgui/server.xml, it's possible to change the document root in line 34: Changing the document root would allow me to mod_proxy from https:///gpfs/gui1 to https:///gpfs/gui1, and in theory that makes it easy to support many GPFS GUIs, and should fix the relative URL / redirect issue. However, while it seems like this ought to simplify matters, in fact it just breaks other parts of the GPFS GUI, because it calls URLs that aren't part of "ROOT.war" and those also assume a document root of "/". So, my basic question for the community is this: Has anyone out there successfully mucked with the GPFS GUI document root? Or am I just barking up the wrong tree? Thanks in advance, Jonathan -- Jonathan Mills / jonathan.mills at nasa.gov NASA GSFC / NCCS HPC (606.2) Bldg 28, Rm. S230 / c. 252-412-5710 From heinrich.billich at id.ethz.ch Mon Feb 8 21:27:27 2021 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Mon, 8 Feb 2021 21:27:27 +0000 Subject: [gpfsug-discuss] Mmapplypolicy with -I defer doesn't sort resulting list? Is this by intention? Message-ID: <9B403CA7-7785-480E-ACC1-38561F0797CD@id.ethz.ch> Hello, I want to migrate data with mmapplypolicy to a different pool: 1. create a file list with "-I defer -f /some/path" 2. execute with "-I yes -r /some/path" I noted that the file list created in 1. Is not sorted. I asked to sort by kb_allocated, the ideas is to migrate the largest files first and to stop once we freed enough space. Is this intended, or should the file list created in 1. be sorted? I want to be able to review the list before I start the migration. The list holds size/kb_allocated in hex numbers, I didn't manage to sort it with 'sort'. Hence I would prefer to get a sorted list from mmapplypolicy instead of writing some code to do this. The list holds files with sizes between 32K and 250G and 4.5M lines, hence it makes sense to sort to move the largest files on the top. Thank you and best regards, Heiner The policy is RULE 'migrate' MIGRATE FROM POOL 'data' THRESHOLD (50,50) WEIGHT (kb_allocated) TO POOL 'capacity' LIMIT (98) WHERE kb_allocated > 0 AND (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) > 90 And the command line mmapplypolicy $1 -A 600 \ -a 10 \ -P $base/policy.txt \ -I prepare \ -f $base/candidates \ -L 1 \ -N some_nodes \ --choice-algorithm fast \ --max-sort-bytes 1G \ --split-margin 2.0 The filesystem holds about 1.2G inodes. I tried to speed-up sorting with the last three arguments. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5254 bytes Desc: not available URL: From jon at well.ox.ac.uk Tue Feb 9 09:13:55 2021 From: jon at well.ox.ac.uk (Jon Diprose) Date: Tue, 9 Feb 2021 09:13:55 +0000 Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml In-Reply-To: References: Message-ID: Might it be easier to add gpfs1. and gpfs2. as CNAMEs to your web proxy / firewall , and then configure the web proxy to select the backend on the basis of the requested hostname? https://gpfs1./ redirects to https://gpfs1./gui/ and there's no mucking about with gpfs stuff required at all. Jon -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC] Sent: 08 February 2021 16:00 To: gpfsug main discussion list Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Dear Community, We have multiple GPFS clusters within our internal networks. We want to expose more than one GPFS GUI interface, from behind a shared web proxy / firewall setup, which has SAML2 integration with our organization's IDP. (The SAML2 is not part of the question, it's just a rationale for doing it this way.) Essentially, we're trying to do something like https:///gpfs/gui1 and https:///gpfs/gui2 The websphere config in the GPFS GUI packages basically assume a document root of "/". This poses additional problems when it forms relative URLs...for example, immediately after login, it tries to redirect from "/" to "/gui". Through our web proxy, this ends up being https:///gui which is a 404. While problems might be addressable with some complex mod_rewrite in our web proxy, I've been looking for a more straightforward solution. In /opt/ibm/wlp/usr/servers/gpfsgui/server.xml, it's possible to change the document root in line 34: Changing the document root would allow me to mod_proxy from https:///gpfs/gui1 to https:///gpfs/gui1, and in theory that makes it easy to support many GPFS GUIs, and should fix the relative URL / redirect issue. However, while it seems like this ought to simplify matters, in fact it just breaks other parts of the GPFS GUI, because it calls URLs that aren't part of "ROOT.war" and those also assume a document root of "/". So, my basic question for the community is this: Has anyone out there successfully mucked with the GPFS GUI document root? Or am I just barking up the wrong tree? Thanks in advance, Jonathan -- Jonathan Mills / jonathan.mills at nasa.gov NASA GSFC / NCCS HPC (606.2) Bldg 28, Rm. S230 / c. 252-412-5710 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From scale at us.ibm.com Tue Feb 9 10:55:45 2021 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 9 Feb 2021 16:25:45 +0530 Subject: [gpfsug-discuss] Mmapplypolicy with -I defer doesn't sort resulting list? Is this by intention? In-Reply-To: <9B403CA7-7785-480E-ACC1-38561F0797CD@id.ethz.ch> References: <9B403CA7-7785-480E-ACC1-38561F0797CD@id.ethz.ch> Message-ID: Hi Billich, I think the problem is that you are specifying --choice-algorithm fast and as per documentation "The fast choice method does not completely sort the candidates by weight." To sort the list you can try specifying --choice-algorithm exact which is also the default. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 09-02-2021 03.13 AM Subject: [EXTERNAL] [gpfsug-discuss] Mmapplypolicy with -I defer doesn't sort resulting list? Is this by intention? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I want to migrate data with mmapplypolicy to a different pool: 1. create a file list with "-I defer -f /some/path" 2. execute with "-I yes -r /some/path" I noted that the file list created in 1. Is not sorted. I asked to sort by kb_allocated, the ideas is to migrate the largest files first and to stop once we freed enough space. Is this intended, or should the file list created in 1. be sorted? I want to be able to review the list before I start the migration. The list holds size/kb_allocated in hex numbers, I didn't manage to sort it with 'sort'. Hence I would prefer to get a sorted list from mmapplypolicy instead of writing some code to do this. The list holds files with sizes between 32K and 250G and 4.5M lines, hence it makes sense to sort to move the largest files on the top. Thank you and best regards, Heiner The policy is RULE 'migrate' MIGRATE FROM POOL 'data' THRESHOLD (50,50) WEIGHT (kb_allocated) TO POOL 'capacity' LIMIT (98) WHERE kb_allocated > 0 AND (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) > 90 And the command line mmapplypolicy $1 -A 600 \ -a 10 \ -P $base/policy.txt \ -I prepare \ -f $base/candidates \ -L 1 \ -N some_nodes \ --choice-algorithm fast \ --max-sort-bytes 1G \ --split-margin 2.0 The filesystem holds about 1.2G inodes. I tried to speed-up sorting with the last three arguments. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== [attachment "smime.p7s" deleted by Huzefa H Pancha/India/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From jonathan.b.mills at nasa.gov Tue Feb 9 15:07:21 2021 From: jonathan.b.mills at nasa.gov (Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC]) Date: Tue, 9 Feb 2021 15:07:21 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Re: [gpfs][gui] Change document root in server.xml In-Reply-To: References: , Message-ID: Oh, without a doubt, and we considered it. However, the agency where I work makes it rather difficult to register new DNS names, and they actively discourage standing up new websites. But yes, I agree that is the simplest solution, and we may be forced to attempt it and wade through the red tape. Thank you for your response. Jonathan ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Jon Diprose Sent: Tuesday, February 9, 2021 4:13 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Might it be easier to add gpfs1. and gpfs2. as CNAMEs to your web proxy / firewall , and then configure the web proxy to select the backend on the basis of the requested hostname? https://gpfs1./ redirects to https://gpfs1./gui/ and there's no mucking about with gpfs stuff required at all. Jon -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC] Sent: 08 February 2021 16:00 To: gpfsug main discussion list Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Dear Community, We have multiple GPFS clusters within our internal networks. We want to expose more than one GPFS GUI interface, from behind a shared web proxy / firewall setup, which has SAML2 integration with our organization's IDP. (The SAML2 is not part of the question, it's just a rationale for doing it this way.) Essentially, we're trying to do something like https:///gpfs/gui1 and https:///gpfs/gui2 The websphere config in the GPFS GUI packages basically assume a document root of "/". This poses additional problems when it forms relative URLs...for example, immediately after login, it tries to redirect from "/" to "/gui". Through our web proxy, this ends up being https:///gui which is a 404. While problems might be addressable with some complex mod_rewrite in our web proxy, I've been looking for a more straightforward solution. In /opt/ibm/wlp/usr/servers/gpfsgui/server.xml, it's possible to change the document root in line 34: Changing the document root would allow me to mod_proxy from https:///gpfs/gui1 to https:///gpfs/gui1, and in theory that makes it easy to support many GPFS GUIs, and should fix the relative URL / redirect issue. However, while it seems like this ought to simplify matters, in fact it just breaks other parts of the GPFS GUI, because it calls URLs that aren't part of "ROOT.war" and those also assume a document root of "/". So, my basic question for the community is this: Has anyone out there successfully mucked with the GPFS GUI document root? Or am I just barking up the wrong tree? Thanks in advance, Jonathan -- Jonathan Mills / jonathan.mills at nasa.gov NASA GSFC / NCCS HPC (606.2) Bldg 28, Rm. S230 / c. 252-412-5710 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Cjonathan.b.mills%40nasa.gov%7Cb4b847626cd347c347f408d8ccdb21f2%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637484588838656778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=d9r02MnZDlXpfiMTY5dxNIR0FGBORNKMiHUh8MS5NRs%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Cjonathan.b.mills%40nasa.gov%7Cb4b847626cd347c347f408d8ccdb21f2%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637484588838656778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=d9r02MnZDlXpfiMTY5dxNIR0FGBORNKMiHUh8MS5NRs%3D&reserved=0 From helge.hauglin at usit.uio.no Tue Feb 9 15:46:39 2021 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Tue, 09 Feb 2021 16:46:39 +0100 Subject: [gpfsug-discuss] Limiting CES SMB shares to specific subnets Message-ID: Hi. We have an ESS 5.0.4.3 cluster with a CES cluster serving files with NFSv4 ACLs to NFS and SMB clients. This system is used for sensitive research data, and will the next years house thousands of research projects, which will have to be strictly separated. Each project has its own subnet for the project linux and windows hosts. Project directories are independent filesets in file systems, each project directory has NFSv4 ACLs giving acces to only the project group. Project NFS shares are limited to each project's subnet. Project SMB shares have export ACLs (as in "mmsmb exportacl ..") limiting share access to the project's member group, in addition to the NFSv4 ACLs. We also want to limit access to SMB shares to project subnets. There is no way to specify that with "mmsmb", but we have found /usr/lpp/mmfs/bin/net conf setparm "hosts allow" to be working, at least with some limited testing: share access is actually limited to the specified subnets. The additional settings seems to be stored in CTDB under /var/lib/ctdb/persistent. We assume that the "net conf setparm" method is not officially supported by IBM. Although it seems to be working, we wonder if it is a good idea to implement it. For instance, we are wondering if the additional settings will survive later ESS code upgrades, and if it will scale to thousands of SMB shares. We are considering doing the SMB subnet limiting outside CES, but that would add complexity and overhead, so we are not very keen on that. What do other IBM ESS customers do, do you have any advice for us? Yea or nay? Regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From christof.schmitt at us.ibm.com Tue Feb 9 18:06:14 2021 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 9 Feb 2021 18:06:14 +0000 Subject: [gpfsug-discuss] Limiting CES SMB shares to specific subnets In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From helge.hauglin at usit.uio.no Wed Feb 10 09:09:01 2021 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Wed, 10 Feb 2021 10:09:01 +0100 Subject: [gpfsug-discuss] Limiting CES SMB shares to specific subnets In-Reply-To: (Christof Schmitt's message of "Tue, 9 Feb 2021 18:06:14 +0000") References: Message-ID: Hi Christof, thanks for your answer. I have added our vote for the RFE, and put us on the watchlist. Is it possible to say anything about when the RFE might be implemented? >> Project SMB shares have export ACLs (as in "mmsmb exportacl ..")> limiting share access to the project's member group, in addition to the> NFSv4 ACLs.>> We also want to limit access to SMB shares to project subnets.> There is no way to specify that with "mmsmb", but we have found>> /usr/lpp/mmfs/bin/net conf setparm "hosts allow" >> to be working, at least with some limited testing: share access is> actually limited to the specified subnets. The additional settings> seems to be stored in CTDB under /var/lib/ctdb/persistent.>> We assume that the "net conf setparm" method is not officially supported> by IBM. Although it seems to be working, we wonder if it is a good idea> to implement it. For instance, we are wondering if the additional> settings will survive later ESS code upgrades, and if it will scale to> thousands of SMB shares. > > Officially Scale only supports Samba options that can be set throughthe GUI or the mmsmb CLI. Everything else set through 'net conf' hasnot been tested and is not supported. In this specific case, this islikely to work, and it should also be preserved across code upgrades,but again, this is not an official support statement. > > This is also not a new request, there is also a pending RFE to makethis an official Scale feature:https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=141534 > > Regards, > > Christof Schmitt > Software Engineer > IBM Systems, Spectrum Scale Development > +1 520 799 2469 > christof.schmitt at us.ibm.com > @chsc Twitter > > IBM > > > ----- Original message -----From: Helge Hauglin Sent by: gpfsug-discuss-bounces at spectrumscale.orgTo: gpfsug-discuss at spectrumscale.orgCc:Subject: [EXTERNAL] [gpfsug-discuss] Limiting CES SMB shares to specific subnetsDate: Tue, Feb 9, 2021 9:10 AM > Hi.We have an ESS 5.0.4.3 cluster with a CES cluster serving files withNFSv4 ACLs to NFS and SMB clients. This system is used forsensitive research data, and will the next years house thousands ofresearch projects, which will have to be strictly separated. Eachproject has its own subnet for the project linux and windows hosts.Project directories are independent filesets in file systems, eachproject directory has NFSv4 ACLs giving acces to only the project group.Project NFS shares are limited to each project's subnet.Project SMB shares have export ACLs (as in "mmsmb exportacl ..")limiting share access to the project's member group, in addition to theNFSv4 ACLs.We also want to limit access to SMB shares to project subnets.There is no way to specify that with "mmsmb", but we have found /usr/lpp/mmfs/bin/net conf setparm "hosts allow" to be working, at least with some limited testing: share access isactually limited to the specified subnets. The additional settingsseems to be stored in CTDB under /var/lib/ctdb/persistent.We assume that the "net conf setparm" method is not officially supportedby IBM. Although it seems to be working, we wonder if it is a good ideato implement it. For instance, we are wondering if the additionalsettings will survive later ESS code upgrades, and if it will scale tothousands of SMB shares.We are considering doing the SMB subnet limiting outside CES, but that wouldadd complexity and overhead, so we are not very keen on that.What do other IBM ESS customers do, do you have any advice for us?Yea or nay?Regards,Helge Hauglin----------------------------------------------------------------Mr. Helge Hauglin, Senior EngineerSystem administratorCenter for Information Technology, University of Oslo, Norway_______________________________________________gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From janfrode at tanso.net Fri Feb 19 12:50:08 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 19 Feb 2021 13:50:08 +0100 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180119163803.79fddbeb@osc.edu> References: <20180119163803.79fddbeb@osc.edu> Message-ID: We just discussed this a bit internally, and I found "something* that might help... There's a mmrestripefs --inode-criteria command that can be used to identify files with these unknown-to-ILM flags set. Something like: # echo illreplicated > criteria # mmrestripefs gpfs01 -p --inode-criteria criteria -o result Scanning file system metadata, phase 1 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning file system metadata, phase 2 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning file system metadata, phase 3 ... Scan completed successfully. Scanning file system metadata, phase 4 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning file system metadata, phase 5 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning user file metadata ... 100.00 % complete on Thu Feb 18 15:30:14 2021 ( 835584 inodes with total 4271 MB data processed) Scan completed successfully. Check file '/mnt/gpfs01/result' on scale-dev-01 for inodes with broken disk addresses or failures. # cat /mnt/gpfs01/result This inode list was generated in the Parallel Inode Traverse on Thu Feb 18 15:30:10 2021 INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR]) 55559 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE Unclear to me if "-p" is too much work for this task though, or maybe it can be limited with --metadata-only or other options.. HTH On Sat, Jan 20, 2018 at 12:10 AM Edward Wahl wrote: > > This one has been on my list a long time so I figured I'd ask here first > before > I open an apar or request an enhancement (most likely). > > Is there a way using the policy engine to determine the following? > > -metadata replication total/current > -unbalanced file > > Looking to catch things like this that stand out on my filesystem without > having to run several hundred million 'mmlsattr's. > > metadata replication: 1 max 2 > flags: unbalanced > > Ed > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Fri Feb 19 16:29:01 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 19 Feb 2021 17:29:01 +0100 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: References: <20180119163803.79fddbeb@osc.edu> Message-ID: I also see that /usr/lpp/mmfs/samples/util/tsinode will list these flags: [root at scale-dev-01 util]# ./tsinode /mnt/gpfs01/|grep 55559 55559 631878862 0 0 1048576000 104857600 -rw-r--r-- 2 1610965529.234470000 1610965531.035007326 1612952829.394919000 0 blk=1048576 modsnap=1 extperms=0x2,xa replmeta *illReplicated unbalanced* dev=3824,150 archive compressed crtime 1610965529.234470000 That program can probably easily be modified to only list these files.. -jf On Fri, Feb 19, 2021 at 1:50 PM Jan-Frode Myklebust wrote: > We just discussed this a bit internally, and I found "something* that > might help... There's a mmrestripefs --inode-criteria command that can be > used to identify files with these unknown-to-ILM flags set. Something like: > > # echo illreplicated > criteria > # mmrestripefs gpfs01 -p --inode-criteria criteria -o result > Scanning file system metadata, phase 1 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning file system metadata, phase 2 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning file system metadata, phase 3 ... > Scan completed successfully. > Scanning file system metadata, phase 4 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning file system metadata, phase 5 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning user file metadata ... > 100.00 % complete on Thu Feb 18 15:30:14 2021 ( 835584 inodes with total 4271 MB data processed) > Scan completed successfully. > Check file '/mnt/gpfs01/result' on scale-dev-01 for inodes with broken disk addresses or failures. > # cat /mnt/gpfs01/result > This inode list was generated in the Parallel Inode Traverse on Thu Feb 18 15:30:10 2021 > INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR]) > 55559 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE > > Unclear to me if "-p" is too much work for this task though, or maybe it can be limited with --metadata-only or other options.. > > > HTH > > > > > On Sat, Jan 20, 2018 at 12:10 AM Edward Wahl wrote: > >> >> This one has been on my list a long time so I figured I'd ask here first >> before >> I open an apar or request an enhancement (most likely). >> >> Is there a way using the policy engine to determine the following? >> >> -metadata replication total/current >> -unbalanced file >> >> Looking to catch things like this that stand out on my filesystem without >> having to run several hundred million 'mmlsattr's. >> >> metadata replication: 1 max 2 >> flags: unbalanced >> >> Ed >> >> >> >> -- >> >> Ed Wahl >> Ohio Supercomputer Center >> 614-292-9302 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.trafford at yale.edu Tue Feb 23 00:39:36 2021 From: tyler.trafford at yale.edu (Trafford, Tyler) Date: Tue, 23 Feb 2021 00:39:36 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu>, <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu>, Message-ID: My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn?t really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 From TROPPENS at de.ibm.com Wed Feb 24 16:17:05 2021 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Wed, 24 Feb 2021 16:17:05 +0000 Subject: [gpfsug-discuss] Digital German Spectrum Scale User Meeting on March 3&4, 2021 Message-ID: An HTML attachment was scrubbed... URL: From jon at well.ox.ac.uk Thu Feb 25 10:13:09 2021 From: jon at well.ox.ac.uk (Jon Diprose) Date: Thu, 25 Feb 2021 10:13:09 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu>, <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu>, Message-ID: I have written an xCAT plugin such that the client can ask the master to do it, in preparation for some shiny new kit. Not in a public repo yet, so please email me direct if you want further info. -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Trafford, Tyler Sent: 23 February 2021 00:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn't really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jpr9c at virginia.edu Thu Feb 25 15:47:41 2021 From: jpr9c at virginia.edu (Ruffner, Scott (jpr9c)) Date: Thu, 25 Feb 2021 15:47:41 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu> <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> Message-ID: I appreciate the solutions; at the moment, I'm making do with having a key for just this specific copy, and doing our own rsync. If we had local disk on our nodes, that would be helpful, but /var/mmfs is mounted on tmpfs and gets flushed at every reboot (not just upgrades to hardware or software stack. I'm testing with invoking the mmsdrrestore from the client only using a key. Yeah, not ideal leaving that laying around. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu ?On 2/25/21, 5:13 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jon Diprose" wrote: I have written an xCAT plugin such that the client can ask the master to do it, in preparation for some shiny new kit. Not in a public repo yet, so please email me direct if you want further info. -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Trafford, Tyler Sent: 23 February 2021 00:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn't really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jon at well.ox.ac.uk Thu Feb 25 16:18:47 2021 From: jon at well.ox.ac.uk (Jon Diprose) Date: Thu, 25 Feb 2021 16:18:47 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu> <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> Message-ID: The call to the plugin can be invoked directly from the client at any time, not just as a result of an install or an updatenode. I see no reason it couldn't go in a script that's run on client startup, though you'd have to hard-code the address+port the xcatd is listening on as that would normally be provided by xCAT. I started from the remoteshell postscript and worked backwards through the getcredentials.awk script to the credentials.pm plugin, and re-used what I needed for a plugin that invokes relevant mmgetstate, mmaddnode and/or mmsdrrestore calls on the master. Alternatively, just look at those existing postscripts/plugins for a mechanism to dynamically obtain the key, which can then be ephemeral on the client. That is the code path xCAT uses to get keys and passwords to the clients, though the comments at the top of credentials.pm (in /opt/xcat/lib/perl/xCAT_plugins/) give a fairly brutal assessment of the security situation. Jon -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Ruffner, Scott (jpr9c) Sent: 25 February 2021 15:48 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. I appreciate the solutions; at the moment, I'm making do with having a key for just this specific copy, and doing our own rsync. If we had local disk on our nodes, that would be helpful, but /var/mmfs is mounted on tmpfs and gets flushed at every reboot (not just upgrades to hardware or software stack. I'm testing with invoking the mmsdrrestore from the client only using a key. Yeah, not ideal leaving that laying around. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu ?On 2/25/21, 5:13 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jon Diprose" wrote: I have written an xCAT plugin such that the client can ask the master to do it, in preparation for some shiny new kit. Not in a public repo yet, so please email me direct if you want further info. -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Trafford, Tyler Sent: 23 February 2021 00:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn't really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Fri Feb 26 01:49:42 2021 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 26 Feb 2021 01:49:42 +0000 Subject: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Message-ID: <4397068B-AD6C-445E-AA07-390FBA7B743F@nuance.com> I have 2 nodes in a new ESS cluster that won?t display data via the GUI, but I can pull up data using the command line. Node is healthy, pmsensors is running and shows connected to the collector. Command line shows this: [root at ems1 IBM]# mmperfmon query compareNodes cpu_user -b 3600 -n 2 Legend: 1: afmgw03|CPU|cpu_user 2: afmgw04|CPU|cpu_user 3: ems1.gpfs.net|CPU|cpu_user 4: essio1.gpfs.net|CPU|cpu_user 5: essio2.gpfs.net|CPU|cpu_user Row Timestamp afmgw03 afmgw04 ems1 essio1 essio2 1 2021-02-25-19:00:00 0.069542 0.059736 0.231 0.196292 0.213047 2 2021-02-25-20:00:00 0.076463 0.067958 0.270258 0.221003 0.230289 But when I try and view afmgw03 or afmgw04 on the GUI, no data. What?s up? Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Fri Feb 26 12:18:24 2021 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 26 Feb 2021 07:18:24 -0500 Subject: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line In-Reply-To: <4397068B-AD6C-445E-AA07-390FBA7B743F@nuance.com> References: <4397068B-AD6C-445E-AA07-390FBA7B743F@nuance.com> Message-ID: Bob, could you please provide the version of ESS/Scale you have installed? Also, could you please provide information about the exact GUI screen you are using that is not providing the data? Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 02/25/2021 09:41 PM Subject: [EXTERNAL] [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Sent by: gpfsug-discuss-bounces at spectrumscale.org I have 2 nodes in a new ESS cluster that won?t display data via the GUI, but I can pull up data using the command line. Node is healthy, pmsensors is running and shows connected to the collector. Command line shows this: [root at ems1 IBM]# mmperfmon query compareNodes cpu_user -b 3600 -n 2 Legend: 1: afmgw03|CPU|cpu_user 2: afmgw04|CPU|cpu_user 3: ems1.gpfs.net|CPU|cpu_user 4: essio1.gpfs.net|CPU|cpu_user 5: essio2.gpfs.net|CPU|cpu_user Row Timestamp afmgw03 afmgw04 ems1 essio1 essio2 1 2021-02-25-19:00:00 0.069542 0.059736 0.231 0.196292 0.213047 2 2021-02-25-20:00:00 0.076463 0.067958 0.270258 0.221003 0.230289 But when I try and view afmgw03 or afmgw04 on the GUI, no data. What?s up? Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=_afEd51Its1IKJqf-evUvjCHSBCrdDMxTJWn0juEBBI&s=Vn6zN83tuKly6fA8xw128AYfimkX9PZFDY9TRBEQ45s&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Feb 26 13:36:57 2021 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 26 Feb 2021 13:36:57 +0000 Subject: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Message-ID: <5F3EDDA0-28C2-460E-8E99-77E728E4618D@nuance.com> ESS 6.0.1.2 with GPFS 5.0.5.4 What?s even stranger is that I logged in this morning and it was working so? ? Not sure if we should/need to follow-up. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of IBM Spectrum Scale Reply-To: gpfsug main discussion list Date: Friday, February 26, 2021 at 6:18 AM To: gpfsug main discussion list Cc: "gpfsug-discuss-bounces at spectrumscale.org" Subject: [EXTERNAL] Re: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line CAUTION: This Email is from an EXTERNAL source. Ensure you trust this sender before clicking on any links or attachments. ________________________________ Bob, could you please provide the version of ESS/Scale you have installed? Also, could you please provide information about the exact GUI screen you are using that is not providing the data? Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 02/25/2021 09:41 PM Subject: [EXTERNAL] [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I have 2 nodes in a new ESS cluster that won?t display data via the GUI, but I can pull up data using the command line. Node is healthy, pmsensors is running and shows connected to the collector. Command line shows this: [root at ems1 IBM]# mmperfmon query compareNodes cpu_user -b 3600 -n 2 Legend: 1: afmgw03|CPU|cpu_user 2: afmgw04|CPU|cpu_user 3: ems1.gpfs.net|CPU|cpu_user 4: essio1.gpfs.net|CPU|cpu_user 5: essio2.gpfs.net|CPU|cpu_user Row Timestamp afmgw03 afmgw04 ems1 essio1 essio2 1 2021-02-25-19:00:00 0.069542 0.059736 0.231 0.196292 0.213047 2 2021-02-25-20:00:00 0.076463 0.067958 0.270258 0.221003 0.230289 But when I try and view afmgw03 or afmgw04 on the GUI, no data. What?s up? Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=_afEd51Its1IKJqf-evUvjCHSBCrdDMxTJWn0juEBBI&s=Vn6zN83tuKly6fA8xw128AYfimkX9PZFDY9TRBEQ45s&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Sat Feb 27 18:01:02 2021 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sat, 27 Feb 2021 18:01:02 +0000 Subject: [gpfsug-discuss] dssgmkfs.mmvdisk number of NSD's Message-ID: Doing an upgrade on our storage which involved replacing all the 4TB disks with 16TB disks. Some hiccups with five of the disks being dead when inserted but that is all sorted. So the system was originally installed with DSS-G 2.0a so with "legacy" commands for vdisks etc. We had 10 metadata NSD's and 10 data NSD's per draw aka recovery group of the D3284 enclosures. The dssgmkfs.mmvdisk has created exactly one data and one metadata NSD per draw of a DS3284 leading to a really small number of NSD's in the file system. All my instincts tell me that this is going to lead to horrible performance on the file system. Historically you wanted a reasonable number of NSD's in a system for decent performance. Taking what the ddsgmkfs.mmvdisk has give me even with a DSS-G260 you would get only 12 NSD's of each type, which for a potentially ~5PB file system seems on the really low side to me. Is there any way to tell ddsgmkfs.mmvdisk to create more NSD's than the one per recovery group or is this no longer relevant and performance with really low numbers of NSD's is fine these days? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From janfrode at tanso.net Sun Feb 28 09:31:57 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 28 Feb 2021 10:31:57 +0100 Subject: [gpfsug-discuss] dssgmkfs.mmvdisk number of NSD's In-Reply-To: References: Message-ID: I?ve tried benchmarking many vs. few vdisks per RG, and never could see any performance difference. Usually we create 1 vdisk per enclosure per RG, thinking this will allow us to grow with same size vdisks when adding additional enclosures in the future. Don?t think mmvdisk can be told to create multiple vdisks per RG directly, so you have to manually create multiple vdisk sets each with the apropriate size. -jf l?r. 27. feb. 2021 kl. 19:01 skrev Jonathan Buzzard < jonathan.buzzard at strath.ac.uk>: > > Doing an upgrade on our storage which involved replacing all the 4TB > disks with 16TB disks. Some hiccups with five of the disks being dead > when inserted but that is all sorted. > > So the system was originally installed with DSS-G 2.0a so with "legacy" > commands for vdisks etc. We had 10 metadata NSD's and 10 data NSD's per > draw aka recovery group of the D3284 enclosures. > > The dssgmkfs.mmvdisk has created exactly one data and one metadata NSD > per draw of a DS3284 leading to a really small number of NSD's in the > file system. > > All my instincts tell me that this is going to lead to horrible > performance on the file system. Historically you wanted a reasonable > number of NSD's in a system for decent performance. > > Taking what the ddsgmkfs.mmvdisk has give me even with a DSS-G260 you > would get only 12 NSD's of each type, which for a potentially ~5PB file > system seems on the really low side to me. > > Is there any way to tell ddsgmkfs.mmvdisk to create more NSD's than the > one per recovery group or is this no longer relevant and performance > with really low numbers of NSD's is fine these days? > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Mon Feb 1 15:08:19 2021 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 1 Feb 2021 15:08:19 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu>, <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> Message-ID: We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn?t really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From daniel.kidger at uk.ibm.com Mon Feb 1 16:25:11 2021 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Mon, 1 Feb 2021 16:25:11 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From Adam.Willis at ibm.com Mon Feb 1 17:15:18 2021 From: Adam.Willis at ibm.com (Adam Willis) Date: Mon, 1 Feb 2021 17:15:18 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From janfrode at tanso.net Mon Feb 1 18:11:26 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 1 Feb 2021 19:11:26 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: Agree.. Write a policy that takes a "mmapplypolicy -M var=val" argument, and figure out the workdays outside of the policy. Something like: # cat test.poilcy define( access_age, (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) /* list migrated files */ RULE EXTERNAL LIST 'oldFiles' EXEC '' RULE 'oldFiles' LIST 'oldFiles' WHERE (access_age > MINAGE) # mmapplypolicy gpfs01 -P test.policy -I defer -f ./filelist -M MINAGE=5 On Mon, Feb 1, 2021 at 5:29 PM Daniel Kidger wrote: > To me, it feels like you need to do the search the other way around. > > Firstly, from the target of say "4 weekdays ago", work out how many real > days ago that was. > Then use that as a criteria in mmfind or mmapplypolicy. > > Daniel > > _________________________________________________________ > *Daniel Kidger Ph.D.* > IBM Technical Sales Specialist > Spectrum Scale, Spectrum Discover and IBM Cloud Object Storage > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > > > > > > > > > ----- Original message ----- > From: Jonathan Buzzard > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, > Issue 18 > Date: Sat, Jan 30, 2021 10:29 > > On 30/01/2021 00:31, Owen Morgan wrote: > > [SNIP] > > > > > I would prefer to stay in the bounds of the SQL policy rule setup as > > that is the framework I have created and started to implement.. > > > In general SQL is Turing complete. Though I have not checked in detail I > believe the SQL of the policy engine is too. > > I would also note that SQL has a whole bunch of time/date functions. > > So something like > > define(offset, 4) > define(day, DAYOFWEEK(CURRENT_TIMESTAMP)) > define(age,(DAYS(CURRENT_TIMESTAMP)-DAYS(ACCESS_TIME))) > define(workingdays, > CASE > WHEN day=1 > THEN offest+1 > WHEN day=6 > THEN offset > WHEN day=7 > THEN offset+1 > ELSE > offset+2 > ) > > /* delete all files from files older than 4 working days */ > RULE purge4 DELETE > WHERE (age>workingdays) FOR FILESET dummies > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From giovanni.bracco at enea.it Mon Feb 1 19:41:43 2021 From: giovanni.bracco at enea.it (Giovanni Bracco) Date: Mon, 1 Feb 2021 20:41:43 +0100 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> Message-ID: <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> On 30/01/21 21:01, Walter Sklenka wrote: > Hi Giovanni! > Thats great! Many thanks for your fast and detailed answer!!!! > So this is the way we will go too! > > Have a nice weekend and keep healthy! > Best regards > Walter > I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works! If yu have any issues I am ready to discuss! Regards Giovanni > -----Original Message----- > From: Giovanni Bracco > Sent: Samstag, 30. J?nner 2021 18:08 > To: gpfsug main discussion list ; Walter Sklenka > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > See > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a multifabric GPFS Spectrum Scale layout," 2019 International Conference on High Performance Computing & Simulation (HPCS), Dublin, Ireland, 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > When setting up the system the main trick has been: > just use CentOS drivers and do not install OFED We do not use IPoIB. > > Giovanni > > On 30/01/21 06:45, Walter Sklenka wrote: >> Hi! >> >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? >> >> In the faq >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. >> html#rdma >> >> >> They talk about RDMA : >> >> "RDMA is NOT ?supported on a node when both Mellanox HCAs and Intel >> Omni-Path HFIs are ENABLED for RDMA." >> >> So do I understand right: When we do NOT enable ?the opa interface we >> can still enable IB ? >> >> The reason I ask ?is, that we have a gpfs cluster of 6 NSD Servers >> (wih access to storage) ?with opa interfaces which provide access to >> remote cluster ?also via OPA. >> >> A new cluster with HDR interfaces will be implemented soon >> >> They shell have access to the same filesystems >> >> When we add HDR interfaces to? NSD servers? and enable rdma on this >> network ?while disabling rdma on opa we would accept the worse >> performance via opa . We hope that this provides ?still better perf >> and less technical overhead ?than using routers >> >> Or am I totally wrong? >> >> Thank you very much and keep healthy! >> >> Best regards >> >> Walter >> >> Mit freundlichen Gr??en >> */Walter Sklenka/* >> */Technical Consultant/* >> >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 >> Wien >> Tel: +43 1 29 22 165-31 >> Fax: +43 1 29 22 165-90 >> E-Mail: sklenka at edv-design.at >> Internet: www.edv-design.at >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Giovanni Bracco > phone +39 351 8804788 > E-mail giovanni.bracco at enea.it > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco From jonathan.buzzard at strath.ac.uk Mon Feb 1 20:17:34 2021 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 1 Feb 2021 20:17:34 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: On 01/02/2021 18:11, Jan-Frode Myklebust wrote: > CAUTION: This email originated outside the University. Check before > clicking links or attachments. > Agree.. Write a policy that takes a "mmapplypolicy -M var=val" argument, > and figure out the workdays outside of the policy. Something like: > > # cat test.poilcy > define( access_age, ? ? (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) > /* list migrated files */ > RULE EXTERNAL LIST 'oldFiles' EXEC '' > RULE 'oldFiles' LIST 'oldFiles' > ? ? WHERE (access_age > MINAGE) > > #?mmapplypolicy gpfs01 ?-P test.policy -I defer -f ./filelist -M MINAGE=5 > Why bother when you can do it all in the policy? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From owen.morgan at motionpicturesolutions.com Mon Feb 1 21:09:16 2021 From: owen.morgan at motionpicturesolutions.com (Owen Morgan) Date: Mon, 1 Feb 2021 21:09:16 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 Message-ID: Jonathan, If I have a single policy file with all the related department rules and each time they want to add additional rules with different working day thresholds maybe using this -M method is easier. Its clear that the 'maths' and date/timestamp manipulation is easier in shell (my preferred is bash) than in the SQL of the policy (your example is succinct but needs to be repeated everytime a new rule is added with a different working day threshold, which is what I'm trying (if possiblr) to avoid. It seems to me the IBM SQL engine is perhaps missing more 'SQL' in built date/time functions like DateAdd and DateDiff etc.. as this would be a moot point. Its a shame I can't make one function that given a working day input as an argument spits out how many 'real' days exist between them for the file age comparison all in the SQL. It can be done for 1 specific input argument, but needs the whole function repeated manually for a different input argument, and further repeated for a different argument etc.. Maybe I'm also compounding the issue by trying to make the policy file as concise as possible (for sake of clarity as to what the rules are trying to achieve, and easy expandability), and demanding too much of the SQL-like syntax that IBM have created. I have options for mmfind or even (as suggested) -M inoput to mmapplypolicy where I us bash to create a small function that does what I need, spits out 'real days' given a working day input, and using arrays and for-loop create a dynamic calling of the mmapplypolicy command (which I'm kinda half doing anyways for other reasons in my launcher script. As always, I'm seriously amazed at people with soo much experience and knowledge taking time out to help, guide, and offer input like everyone has been doing!! I'm relatively early in my career, so being able to interact and learn from experienced persons is giving me such a wider insight! Thanks! Owen. [Sent from Front] On 1 February 2021, 20:17 GMT jonathan.buzzard at strath.ac.uk wrote: On 01/02/2021 18:11, Jan-Frode Myklebust wrote: > CAUTION: This email originated outside the University. Check before > clicking links or attachments. > Agree.. Write a policy that takes a "mmapplypolicy -M var=val" argument, > and figure out the workdays outside of the policy. Something like: > > # cat test.poilcy > define( access_age, (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) > /* list migrated files */ > RULE EXTERNAL LIST 'oldFiles' EXEC '' > RULE 'oldFiles' LIST 'oldFiles' > WHERE (access_age > MINAGE) > > # mmapplypolicy gpfs01 -P test.policy -I defer -f ./filelist -M MINAGE=5 > Why bother when you can do it all in the policy? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG Owen Morgan Data Wrangler Motion Picture Solutions Ltd T: E: owen.morgan at motionpicturesolutions.com | W: motionpicturesolutions.com A: Mission Hall, 9-11 North End Road, London, W14 8ST Motion Picture Solutions Ltd is a company registered in England and Wales under number 5388229, VAT number 201330482 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Mon Feb 1 21:43:43 2021 From: jtucker at pixitmedia.com (Jez Tucker) Date: Mon, 1 Feb 2021 21:43:43 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: <02351047-3b02-352f-48bc-b4b3e0e03918@pixitmedia.com> Hi Owen, ? This is a great thread and raises as is usual with GPFS that there is more often than not many tools in the bag and more than one way to achieve a requirement. If you stick with policy (indeed worthwhile aspect of GPFS to learn well) you can also build up libraries of centralised macros and reference them into any policy file alike so at the top of the file: include(/mmfs1/policies/macros/general_macros.m4) include(/mmfs1/policies/macros/pixit_excludes.m4) Check your PixStor system under /mmfs1/policies/.. The .m4 extension is not required, but a nod to https://www.gnu.org/savannah-checkouts/gnu/m4/manual/m4-1.4.18/m4.html which the policy engine also encapsulates.? Marc Kaplan once told me "You can do just about anything with m4 and cleverness and patience...", though he failed to mention the required amounts of coffee and pizza ;-)? Search the list digests for his valuable insights over the years. Alternatively you can achieve all this and more using the Python API https://www.arcapix.com/pixstorapi/index.html (especially your date processing). If you'd like direct support with any this, ping us over an email via support@ - alternatively there are indeed many great minds a wealth of experience and views on this list (and it's nice to meet the community too). SSUG also has a Slack channel too.. ssug-poweraiug.slack.com Most of all - have fun learning. Kind regards, Jez p.s. yes the threading has all gone wonky. ah well :-) On 01/02/2021 21:09, Owen Morgan wrote: > Jonathan, > > If I have a single policy file with all the related department rules > and each time they want to add additional rules with different working > day thresholds maybe using this -M method is easier. Its clear that > the 'maths' and date/timestamp manipulation is easier in shell (my > preferred is bash) than in the SQL of the policy (your example is > succinct but needs to be repeated everytime a new rule is added with a > different working day threshold, which is what I'm trying (if > possiblr) to avoid. > > It seems to me the IBM SQL engine is perhaps missing more 'SQL' in > built date/time functions like DateAdd and DateDiff etc..? as this > would be a moot point. Its a shame I can't make one function that > given a working day input as an argument spits out how many 'real' > days exist between them for the file age comparison all in the SQL. It > can be done for 1 specific input argument, but needs the whole > function repeated manually for a different input argument, and further > repeated for a different argument etc.. > > Maybe I'm also compounding the issue by trying to make the policy file > as concise as possible (for sake of clarity as to what the rules are > trying to achieve, and easy expandability), and demanding too much of > the SQL-like syntax that IBM have created. > > I have options for mmfind or even (as suggested) -M inoput to > mmapplypolicy where I us bash to create a small function that does > what I need, spits out 'real days' given a working day input, and > using arrays and for-loop create a dynamic calling of the > mmapplypolicy command (which I'm kinda half doing anyways for other > reasons in my launcher script. > > > As always, I'm seriously amazed at people with soo much experience and > knowledge taking time out to help, guide, and offer input like > everyone has been doing!! I'm relatively early in my career, so being > able to interact and learn from experienced persons is giving me such > a wider insight! > > Thanks! > > Owen. > Sent from Front > ? ? > > Owen?Morgan? > Data?Wrangler > Motion?Picture?Solutions?Ltd > T:?** > > E:?*owen.morgan at motionpicturesolutions.com* > ?|? > W:?*motionpicturesolutions.com* > > A:? Mission?Hall,?9?11?North?End?Road ,? London ,? W14?8ST > > Motion Picture Solutions Ltd is a company registered in England and > Wales under number 5388229, VAT number 201330482 > >> On 1 February 2021, 20:17 GMT jonathan.buzzard at strath.ac.uk >> wrote: >> >> On 01/02/2021 18:11, Jan-Frode Myklebust wrote: >>> >>> > CAUTION: This email originated outside the University. Check before >>> > clicking links or attachments. >>> > Agree.. Write a policy that takes a "mmapplypolicy -M var=val" >>> argument, >>> > and figure out the workdays outside of the policy. Something like: >>> > >>> > # cat test.poilcy >>> > define( access_age, ? ? (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) >>> > /* list migrated files */ >>> > RULE EXTERNAL LIST 'oldFiles' EXEC '' >>> > RULE 'oldFiles' LIST 'oldFiles' >>> > ? ? WHERE (access_age > MINAGE) >>> > >>> > #?mmapplypolicy gpfs01 ?-P test.policy -I defer -f ./filelist -M >>> MINAGE=5 >>> > >>> >>> Why bother when you can do it all in the policy? >>> >>> JAB. >>> >>> -- >>> Jonathan A. Buzzard Tel: +44141-5483420 >>> HPC System Administrator, ARCHIE-WeSt. >>> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG >>> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- *Jez Tucker* VP Research and Development | Pixit Media m: +44 (0) 776 419 3820 e: jtucker at pixitmedia.com Visit www.pixitmedia.com -- This email is confidential in that it is?intended for the exclusive attention of?the addressee(s) indicated. If you are?not the intended recipient, this email?should not be read or disclosed to?any other person. Please notify the?sender immediately and delete this?email from your computer system.?Any opinions expressed are not?necessarily those of the company?from which this email was sent and,?whilst to the best of our knowledge no?viruses or defects exist, no?responsibility can be accepted for any?loss or damage arising from its?receipt or subsequent use of this? email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon Feb 1 21:58:34 2021 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 1 Feb 2021 21:58:34 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: <3e2d8556-72a3-930f-dc41-c15ffac1e495@strath.ac.uk> On 01/02/2021 21:09, Owen Morgan wrote: > CAUTION: This email originated outside the University. Check before > clicking links or attachments. > Jonathan, > > If I have a single policy file with all the related department rules and > each time they want to add additional rules with different working day > thresholds maybe using this -M method is easier. Its clear that the > 'maths' and date/timestamp manipulation is easier in shell (my preferred > is bash) than in the SQL of the policy (your example is succinct but > needs to be repeated everytime a new rule is added with a different > working day threshold, which is what I'm trying (if possiblr) to avoid. > I actually think there is a much better way to achieve it than the case statement, but that would have required me to do some more thinking and testing to make sure it worked ;-) Anyway how many different "working" days old do you need? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From anacreo at gmail.com Mon Feb 1 22:14:39 2021 From: anacreo at gmail.com (Alec) Date: Mon, 1 Feb 2021 14:14:39 -0800 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: <3e2d8556-72a3-930f-dc41-c15ffac1e495@strath.ac.uk> References: <3e2d8556-72a3-930f-dc41-c15ffac1e495@strath.ac.uk> Message-ID: In our environment managing the GPFS policies (or anything GPFS) are a bit of an Art form and really only a few people out of a sea of engineers can/will do it. So I like to keep the GPFS policies simple and durable. If you really wanted to you could generate a list of every US (or whatever country) holiday you wanted and use Python, Awk, whatever to produce an exact number of business days versus not; You can also use directory structure, GPFS attributes in parent directory, or a config file to maintain the requirements, which may be more manageable by other team members. Anyhow my point being don't just look at the technical of how to implement, but what is the most maintainable, .documentable, and recoverable solution. Some things I would consider in my design? * Supportability - What is the skill level of the engineer required to maintain the system. * Serviceability - What component would change most frequently, what's the complexity and impact of a change. * Fail Safe - If a component fails, what will the negative impact be. * Trackability - Are all the activities logged. * Sanity Checking - Can you define safety mechanisms to prevent a failure. For instance if your purge script detects that there are 10k+ files to purge, can you interrupt the process? For GPFS we try to keep the service components simple, and the logic in shell scripting where there is a plethora of knowledgeable engineers. So my attack would be like this: * Policy runs, purges any file with purgeafter beyond the current date (maybe validate that the purge.date and modify/access date agree still). <-- is simple * I would have it run the logic twice, and bomb if a safety threshhold is exceeded. Should log the results. * Config in /gpfs/configs/purge_configuration /some/path 10bd /some/scratchspace 30d /some/reallybigfiles 1d * mmfind $(awk '{print $1}' < gpfs/configs/purge_configuration) -type f --exec /gpfs/scripts/setpurgedates * Could also be a regular mmapplypolicy command as well... * /gpfs/scripts/setpurgedates can rescan /gpfs/configs/purge_configuration and read in the date, check against the path passed, and then perform the necessary date logic, and exec the mmchattr, preserve the file's modify/access time to a purge.date time.. * Could log its decisions to a log file. Happy Hunting, I've already learned quite a few things from monitoring this list in a very short period of time. * If a candidate ever tells me they know everything about Unix... I pass. We're all very experienced noobies in this ever progressing field/platform. * Alec On Mon, Feb 1, 2021 at 1:58 PM Jonathan Buzzard < jonathan.buzzard at strath.ac.uk> wrote: > On 01/02/2021 21:09, Owen Morgan wrote: > > CAUTION: This email originated outside the University. Check before > > clicking links or attachments. > > Jonathan, > > > > If I have a single policy file with all the related department rules and > > each time they want to add additional rules with different working day > > thresholds maybe using this -M method is easier. Its clear that the > > 'maths' and date/timestamp manipulation is easier in shell (my preferred > > is bash) than in the SQL of the policy (your example is succinct but > > needs to be repeated everytime a new rule is added with a different > > working day threshold, which is what I'm trying (if possiblr) to avoid. > > > > I actually think there is a much better way to achieve it than the case > statement, but that would have required me to do some more thinking and > testing to make sure it worked ;-) > > Anyway how many different "working" days old do you need? > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Walter.Sklenka at EDV-Design.at Tue Feb 2 13:10:05 2021 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Tue, 2 Feb 2021 13:10:05 +0000 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> Message-ID: <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> Hi Giovanni! Thank you for your offer! ? it is planned to be implemented in June or so We will use RHEL 8.x and newest gpfs version available Only one question for this moment if I am allowed: Did you ever ran into any problems with IBM support? I mean they say in the FAQ shortly "not supported" , but do they in your environment or do you accept that rdma problems would be needed to be fixed without IBM Thank you very much and have great days! And keep healthy! Best regards walter -----Original Message----- From: Giovanni Bracco Sent: Montag, 1. Februar 2021 20:42 To: Walter Sklenka Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled On 30/01/21 21:01, Walter Sklenka wrote: > Hi Giovanni! > Thats great! Many thanks for your fast and detailed answer!!!! > So this is the way we will go too! > > Have a nice weekend and keep healthy! > Best regards > Walter > I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works! If yu have any issues I am ready to discuss! Regards Giovanni > -----Original Message----- > From: Giovanni Bracco > > Sent: Samstag, 30. J?nner 2021 18:08 > To: gpfsug main discussion list >; > Walter Sklenka > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > See > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a > multifabric GPFS Spectrum Scale layout," 2019 International Conference > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > When setting up the system the main trick has been: > just use CentOS drivers and do not install OFED We do not use IPoIB. > > Giovanni > > On 30/01/21 06:45, Walter Sklenka wrote: >> Hi! >> >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? >> >> In the faq >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. >> html#rdma >> >> >> They talk about RDMA : >> >> "RDMA is NOT supported on a node when both Mellanox HCAs and Intel >> Omni-Path HFIs are ENABLED for RDMA." >> >> So do I understand right: When we do NOT enable the opa interface we >> can still enable IB ? >> >> The reason I ask is, that we have a gpfs cluster of 6 NSD Servers >> (wih access to storage) with opa interfaces which provide access to >> remote cluster also via OPA. >> >> A new cluster with HDR interfaces will be implemented soon >> >> They shell have access to the same filesystems >> >> When we add HDR interfaces to NSD servers and enable rdma on this >> network while disabling rdma on opa we would accept the worse >> performance via opa . We hope that this provides still better perf >> and less technical overhead than using routers >> >> Or am I totally wrong? >> >> Thank you very much and keep healthy! >> >> Best regards >> >> Walter >> >> Mit freundlichen Gr??en >> */Walter Sklenka/* >> */Technical Consultant/* >> >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 >> Wien >> Tel: +43 1 29 22 165-31 >> Fax: +43 1 29 22 165-90 >> E-Mail: sklenka at edv-design.at >> Internet: www.edv-design.at >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Giovanni Bracco > phone +39 351 8804788 > E-mail giovanni.bracco at enea.it > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco -------------- next part -------------- An HTML attachment was scrubbed... URL: From Walter.Sklenka at EDV-Design.at Tue Feb 2 13:19:37 2021 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Tue, 2 Feb 2021 13:19:37 +0000 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> Message-ID: Hi Giovanni! Thank you very much for your offer , we really would be very grateful to be allowed to come if we run into troubles! Well, the implementation will not happen before June or later, but may I ask only one question meanwhile? Did you ever run into problems with IBM support or did you get a special ?OK? from them? Or do you accept to sove any rdma specific problems without support ? (it?s only because of the FAQ ?not supported? ) Have a great day and keep healthy! Best regards walter -----Original Message----- From: Giovanni Bracco Sent: Montag, 1. Februar 2021 20:42 To: Walter Sklenka Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled On 30/01/21 21:01, Walter Sklenka wrote: > Hi Giovanni! > Thats great! Many thanks for your fast and detailed answer!!!! > So this is the way we will go too! > > Have a nice weekend and keep healthy! > Best regards > Walter > I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works! If yu have any issues I am ready to discuss! Regards Giovanni > -----Original Message----- > From: Giovanni Bracco > > Sent: Samstag, 30. J?nner 2021 18:08 > To: gpfsug main discussion list >; > Walter Sklenka > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > See > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a > multifabric GPFS Spectrum Scale layout," 2019 International Conference > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > When setting up the system the main trick has been: > just use CentOS drivers and do not install OFED We do not use IPoIB. > > Giovanni > > On 30/01/21 06:45, Walter Sklenka wrote: >> Hi! >> >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? >> >> In the faq >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. >> html#rdma >> >> >> They talk about RDMA : >> >> "RDMA is NOT supported on a node when both Mellanox HCAs and Intel >> Omni-Path HFIs are ENABLED for RDMA." >> >> So do I understand right: When we do NOT enable the opa interface we >> can still enable IB ? >> >> The reason I ask is, that we have a gpfs cluster of 6 NSD Servers >> (wih access to storage) with opa interfaces which provide access to >> remote cluster also via OPA. >> >> A new cluster with HDR interfaces will be implemented soon >> >> They shell have access to the same filesystems >> >> When we add HDR interfaces to NSD servers and enable rdma on this >> network while disabling rdma on opa we would accept the worse >> performance via opa . We hope that this provides still better perf >> and less technical overhead than using routers >> >> Or am I totally wrong? >> >> Thank you very much and keep healthy! >> >> Best regards >> >> Walter >> >> Mit freundlichen Gr??en >> */Walter Sklenka/* >> */Technical Consultant/* >> >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 >> Wien >> Tel: +43 1 29 22 165-31 >> Fax: +43 1 29 22 165-90 >> E-Mail: sklenka at edv-design.at >> Internet: www.edv-design.at >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Giovanni Bracco > phone +39 351 8804788 > E-mail giovanni.bracco at enea.it > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Tue Feb 2 16:08:47 2021 From: ewahl at osc.edu (Wahl, Edward) Date: Tue, 2 Feb 2021 16:08:47 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180202152336.03e8bab7@osc.edu> References: <20180119163803.79fddbeb@osc.edu> , <20180202152336.03e8bab7@osc.edu> Message-ID: Replying to a 3 year old message I sent, hoping that in the last couple of years that Scale has added some ILM extensions into the policy engine that I have missed, or somehow didn't notice? Just ran into a file with an 'unbalanced' flag and I REALLY don't want to have to mmlsattr everything. AGAIN. /facepalm IBM? Bueller? Bueller? When everyone answers: "No", I'm guessing this needs to be a request for improvement/enhancement? Ed Wahl Ohio Supercomputer Center ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Edward Wahl Sent: Friday, February 2, 2018 3:23 PM To: John Hearns Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] policy ilm features? Thanks John, this was the path I was HOPING to go down as I do similar things already, but there appears to be no extended attribute in ILM for what I want. Data block replication flag exists in the ILM, but not MetaData, or balance. Yet these states ARE reported by mmlsattr, so there must be a flag somewhere. bad MD replication & balance example: mmlsattr -L /fs/scratch/sysp/ed/180days.pol file name: /fs/scratch/sysp/ed/180days.pol metadata replication: 1 max 2 data replication: 1 max 2 flags: illreplicated,unbalanced Encrypted: yes File next to it for comparison. note proper MD replication and balance. mmlsattr -L /fs/scratch/sysp/ed/120days.pol file name: /fs/scratch/sysp/ed/120days.pol metadata replication: 2 max 2 data replication: 1 max 2 flags: Encrypted: yes misc_attributes flags from a policy run showing no difference in status: FJAEu -- /fs/scratch/sysp/ed/180days.pol FJAEu -- /fs/scratch/sysp/ed/120days.pol File system has MD replication enabled, but not Data, so ALL files show "J" ilm flag mmlsfs scratch -m flag value description ------------------- ------------------------ ----------------------------------- -m 2 Default number of metadata replicas mmlsfs scratch -r flag value description ------------------- ------------------------ ----------------------------------- -r 1 Default number of data replicas I poked around a little trying to find out if perhaps using GetXattr would work and show me what I wanted, it does not. All I sem to be able to get is the File Encryption Key. I was hoping perhaps someone had found a cheaper way for this to work rather than hundreds of millions of 'mmlsattr' execs. :-( On the plus side, I've only run across a few of these and all appear to be from before we did the MD replication and re-striping. On the minus, I have NO idea where they are, and they appears to be on both of our filesystems. So several hundred million files to check. Ed On Mon, 22 Jan 2018 08:29:42 +0000 John Hearns wrote: > Ed, > This is not a perfect answer. You need to look at policies for this. I have > been doing something similar recently. > > Something like: > > RULE 'list_file' EXTERNAL LIST 'all-files' EXEC > '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files' > SHOW( varchar(kb_allocated) || ' ' || varchar(file_size) || ' ' || > varchar(misc_attributes) || ' ' || name || ' ' || fileset_name ) WHERE > REGEX(misc_attributes,'[J]') > > > So this policy shows the kbytes allocates, file size, the miscellaneous > attributes, name and fileset name For all files with miscellaneous > attributes of 'J' which means 'Some data blocks might be ill replicated' > > > > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Edward Wahl > Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] policy ilm features? > > > This one has been on my list a long time so I figured I'd ask here first > before I open an apar or request an enhancement (most likely). > > Is there a way using the policy engine to determine the following? > > -metadata replication total/current > -unbalanced file > > Looking to catch things like this that stand out on my filesystem without > having to run several hundred million 'mmlsattr's. > > metadata replication: 1 max 2 > flags: unbalanced > > Ed > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D&reserved=0 > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is provided > on an AS-IS basis without any express or implied warranties or liabilities. > To the extent you are relying on this information, you are doing so at your > own risk. If you are not the intended recipient, please notify the sender > immediately by replying to this message and destroy all copies of this > message and any attachments. Neither the sender nor the company/group of > companies he or she represents shall be liable for the proper and complete > transmission of the information contained in this communication, or for any > delay in its receipt. _______________________________________________ > gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Ed Wahl Ohio Supercomputer Center 614-292-9302 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Tue Feb 2 18:09:28 2021 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 2 Feb 2021 18:09:28 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: References: , <20180119163803.79fddbeb@osc.edu>, <20180202152336.03e8bab7@osc.edu> Message-ID: An HTML attachment was scrubbed... URL: From ewahl at osc.edu Tue Feb 2 21:26:42 2021 From: ewahl at osc.edu (Wahl, Edward) Date: Tue, 2 Feb 2021 21:26:42 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: References: , <20180119163803.79fddbeb@osc.edu>, <20180202152336.03e8bab7@osc.edu>, Message-ID: My issues were never specifically the Unbalanced flag, originally I ran into ilm/policy issues with our metadata and illreplicated files. We were working to expand our Metadata storage at the time uner 4.2.3.xx and had added a number of SSDs to the array. But AFTER a restripe files were still illreplicated. I then discovered that the policy engine had no way to tell me what files were and were not replicated. Just hoping that more ILM info had been added to the policy engine since I ran into this, and seeing the unbalanced files jogged my memory to look at the docs again, where I didn't see anything in 5.x and ask the list as well. This isn't a big deal, more curiousity on my part. I see you have some of the original thread attached, so perhaps take a glance and see if it makes sense? I Now if you REALLY want to step into a mine field, go find my thread on SKLM usage and GPFS/SS. Every single respondent to my question about filling up their SKLM logs with errors was a positive. And SKLM L2 and L3 support swears GPFS/SS is using SKLM wrong... Ed Wahl OSC ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederick Stock Sent: Tuesday, February 2, 2021 1:09 PM To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] policy ilm features? Hello Ed. Jordan contacted me about the question you are posing so I am responding to your message. Could you please provide clarification as to why the existence of the unbalanced flag is of a concern, or why you would want to know all the files that have this flag set? The flag would be cleared once the file was rebalanced either through normal access or through the execution of the mmrestripefs/mmrestripefile commands. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com ----- Original message ----- From: "Wahl, Edward" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] policy ilm features? Date: Tue, Feb 2, 2021 11:52 AM Replying to a 3 year old message I sent, hoping that in the last couple of years that Scale has added some ILM extensions into the policy engine that I have missed, or somehow didn't notice? Just ran into a file with an 'unbalanced' flag and I REALLY don't want to have to mmlsattr everything. AGAIN. /facepalm IBM? Bueller? Bueller? When everyone answers: "No", I'm guessing this needs to be a request for improvement/enhancement? Ed Wahl Ohio Supercomputer Center ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Edward Wahl Sent: Friday, February 2, 2018 3:23 PM To: John Hearns Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] policy ilm features? Thanks John, this was the path I was HOPING to go down as I do similar things already, but there appears to be no extended attribute in ILM for what I want. Data block replication flag exists in the ILM, but not MetaData, or balance. Yet these states ARE reported by mmlsattr, so there must be a flag somewhere. bad MD replication & balance example: mmlsattr -L /fs/scratch/sysp/ed/180days.pol file name: /fs/scratch/sysp/ed/180days.pol metadata replication: 1 max 2 data replication: 1 max 2 flags: illreplicated,unbalanced Encrypted: yes File next to it for comparison. note proper MD replication and balance. mmlsattr -L /fs/scratch/sysp/ed/120days.pol file name: /fs/scratch/sysp/ed/120days.pol metadata replication: 2 max 2 data replication: 1 max 2 flags: Encrypted: yes misc_attributes flags from a policy run showing no difference in status: FJAEu -- /fs/scratch/sysp/ed/180days.pol FJAEu -- /fs/scratch/sysp/ed/120days.pol File system has MD replication enabled, but not Data, so ALL files show "J" ilm flag mmlsfs scratch -m flag value description ------------------- ------------------------ ----------------------------------- -m 2 Default number of metadata replicas mmlsfs scratch -r flag value description ------------------- ------------------------ ----------------------------------- -r 1 Default number of data replicas I poked around a little trying to find out if perhaps using GetXattr would work and show me what I wanted, it does not. All I sem to be able to get is the File Encryption Key. I was hoping perhaps someone had found a cheaper way for this to work rather than hundreds of millions of 'mmlsattr' execs. :-( On the plus side, I've only run across a few of these and all appear to be from before we did the MD replication and re-striping. On the minus, I have NO idea where they are, and they appears to be on both of our filesystems. So several hundred million files to check. Ed On Mon, 22 Jan 2018 08:29:42 +0000 John Hearns wrote: > Ed, > This is not a perfect answer. You need to look at policies for this. I have > been doing something similar recently. > > Something like: > > RULE 'list_file' EXTERNAL LIST 'all-files' EXEC > '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files' > SHOW( varchar(kb_allocated) || ' ' || varchar(file_size) || ' ' || > varchar(misc_attributes) || ' ' || name || ' ' || fileset_name ) WHERE > REGEX(misc_attributes,'[J]') > > > So this policy shows the kbytes allocates, file size, the miscellaneous > attributes, name and fileset name For all files with miscellaneous > attributes of 'J' which means 'Some data blocks might be ill replicated' > > > > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Edward Wahl > Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] policy ilm features? > > > This one has been on my list a long time so I figured I'd ask here first > before I open an apar or request an enhancement (most likely). > > Is there a way using the policy engine to determine the following? > > -metadata replication total/current > -unbalanced file > > Looking to catch things like this that stand out on my filesystem without > having to run several hundred million 'mmlsattr's. > > metadata replication: 1 max 2 > flags: unbalanced > > Ed > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D&reserved=0 > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is provided > on an AS-IS basis without any express or implied warranties or liabilities. > To the extent you are relying on this information, you are doing so at your > own risk. If you are not the intended recipient, please notify the sender > immediately by replying to this message and destroy all copies of this > message and any attachments. Neither the sender nor the company/group of > companies he or she represents shall be liable for the proper and complete > transmission of the information contained in this communication, or for any > delay in its receipt. _______________________________________________ > gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Ed Wahl Ohio Supercomputer Center 614-292-9302 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From giovanni.bracco at enea.it Wed Feb 3 08:58:37 2021 From: giovanni.bracco at enea.it (Giovanni Bracco) Date: Wed, 3 Feb 2021 09:58:37 +0100 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> Message-ID: <0ecee41c-9a9b-83d4-59ae-2b0fbda526fb@enea.it> We did not explore the issue of the IBM support and for budget limitation and for the mandatory integration of the data space between the two clusters, we decided to try the setup of the multi-fabric infrastructure and up to now it has been working without problems. Giovanni On 02/02/21 14:10, Walter Sklenka wrote: > Hi Giovanni! > > Thank you for your offer! ? > > it is planned to be implemented in June or so > > We will use RHEL 8.x and newest gpfs version available > > Only one question for this moment if I am allowed: > > Did you ever ran into any problems with IBM support? I mean they say in > the FAQ shortly "not supported" , but do they in your environment or do > you accept that rdma problems would be needed to be fixed without IBM > > Thank you very much and have great days! And keep healthy! > > Best regards walter > > -----Original Message----- > From: Giovanni Bracco > Sent: Montag, 1. Februar 2021 20:42 > To: Walter Sklenka > Cc: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > On 30/01/21 21:01, Walter Sklenka wrote: > > > Hi Giovanni! > > > Thats great! Many thanks for your fast and detailed answer!!!! > > > So this is the way we will go too! > > > > > > Have a nice weekend and keep healthy! > > > Best regards > > > Walter > > > > > I suppose you will implement the solution with more recent versions of > the software components, so please let me know if everything works! > > If yu have any issues I am ready to discuss! > > Regards > > Giovanni > > > -----Original Message----- > > > From: Giovanni Bracco > > > > Sent: Samstag, 30. J?nner 2021 18:08 > > > To: gpfsug main discussion list >; > > > Walter Sklenka > > > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > > > Server with only ib rdma enabled > > > > > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, > each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes > SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA > Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN > storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > > > See > > > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a > > > multifabric GPFS Spectrum Scale layout," 2019 International Conference > > > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > > > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > > > > > When setting up the system the main trick has been: > > > just use CentOS drivers and do not install OFED We do not use IPoIB. > > > > > > Giovanni > > > > > > On 30/01/21 06:45, Walter Sklenka wrote: > > >> Hi! > > >> > > >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? > > >> > > >> In the faq > > >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. > > >> html#rdma > > >> > > >> > > >> They talk about RDMA : > > >> > > >> "RDMA is NOT ?supported on a node when both Mellanox HCAs and Intel > > >> Omni-Path HFIs are ENABLED for RDMA." > > >> > > >> So do I understand right: When we do NOT enable ?the opa interface we > > >> can still enable IB ? > > >> > > >> The reason I ask ?is, that we have a gpfs cluster of 6 NSD Servers > > >> (wih access to storage) ?with opa interfaces which provide access to > > >> remote cluster ?also via OPA. > > >> > > >> A new cluster with HDR interfaces will be implemented soon > > >> > > >> They shell have access to the same filesystems > > >> > > >> When we add HDR interfaces to? NSD servers? and enable rdma on this > > >> network ?while disabling rdma on opa we would accept the worse > > >> performance via opa . We hope that this provides ?still better perf > > >> and less technical overhead ?than using routers > > >> > > >> Or am I totally wrong? > > >> > > >> Thank you very much and keep healthy! > > >> > > >> Best regards > > >> > > >> Walter > > >> > > >> Mit freundlichen Gr??en > > >> */Walter Sklenka/* > > >> */Technical Consultant/* > > >> > > >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 > > >> Wien > > >> Tel: +43 1 29 22 165-31 > > >> Fax: +43 1 29 22 165-90 > > >> E-Mail: sklenka at edv-design.at > > > >> Internet: www.edv-design.at > > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > -- > > > Giovanni Bracco > > > phone? +39 351 8804788 > > > E-mail giovanni.bracco at enea.it > > > WWW http://www.afs.enea.it/bracco > > > > > -- > > Giovanni Bracco > > phone? +39 351 8804788 > > E-mail giovanni.bracco at enea.it > > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco From Walter.Sklenka at EDV-Design.at Wed Feb 3 15:21:18 2021 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Wed, 3 Feb 2021 15:21:18 +0000 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <0ecee41c-9a9b-83d4-59ae-2b0fbda526fb@enea.it> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> <0ecee41c-9a9b-83d4-59ae-2b0fbda526fb@enea.it> Message-ID: Hi Givanni ! I understand and am convinced that the is an excellent solution !! Thank you very much! -----Original Message----- From: Giovanni Bracco Sent: Mittwoch, 3. Februar 2021 09:59 To: Walter Sklenka Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled We did not explore the issue of the IBM support and for budget limitation and for the mandatory integration of the data space between the two clusters, we decided to try the setup of the multi-fabric infrastructure and up to now it has been working without problems. Giovanni On 02/02/21 14:10, Walter Sklenka wrote: > Hi Giovanni! > > Thank you for your offer! ? > > it is planned to be implemented in June or so > > We will use RHEL 8.x and newest gpfs version available > > Only one question for this moment if I am allowed: > > Did you ever ran into any problems with IBM support? I mean they say > in the FAQ shortly "not supported" , but do they in your environment > or do you accept that rdma problems would be needed to be fixed > without IBM > > Thank you very much and have great days! And keep healthy! > > Best regards walter > > -----Original Message----- > From: Giovanni Bracco > Sent: Montag, 1. Februar 2021 20:42 > To: Walter Sklenka > Cc: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > On 30/01/21 21:01, Walter Sklenka wrote: > > > Hi Giovanni! > > > Thats great! Many thanks for your fast and detailed answer!!!! > > > So this is the way we will go too! > > > > > > Have a nice weekend and keep healthy! > > > Best regards > > > Walter > > > > > I suppose you will implement the solution with more recent versions of > the software components, so please let me know if everything works! > > If yu have any issues I am ready to discuss! > > Regards > > Giovanni > > > -----Original Message----- > > > From: Giovanni Bracco > > > > Sent: Samstag, 30. J?nner 2021 18:08 > > > To: gpfsug main discussion list >; > > > Walter Sklenka > > > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > > > Server with only ib rdma enabled > > > > > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, > each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes > SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main > OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to > DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > > > See > > > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of > a > > > multifabric GPFS Spectrum Scale layout," 2019 International > Conference > > > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > > > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > > > > > When setting up the system the main trick has been: > > > just use CentOS drivers and do not install OFED We do not use IPoIB. > > > > > > Giovanni > > > > > > On 30/01/21 06:45, Walter Sklenka wrote: > > >> Hi! > > >> > > >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? > > >> > > >> In the faq > > >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. > > >> html#rdma > > >> > > >> > > >> They talk about RDMA : > > >> > > >> "RDMA is NOT ?supported on a node when both Mellanox HCAs and > Intel > > >> Omni-Path HFIs are ENABLED for RDMA." > > >> > > >> So do I understand right: When we do NOT enable ?the opa interface > we > > >> can still enable IB ? > > >> > > >> The reason I ask ?is, that we have a gpfs cluster of 6 NSD Servers > > >> (wih access to storage) ?with opa interfaces which provide access > to > > >> remote cluster ?also via OPA. > > >> > > >> A new cluster with HDR interfaces will be implemented soon > > >> > > >> They shell have access to the same filesystems > > >> > > >> When we add HDR interfaces to? NSD servers? and enable rdma on > this > > >> network ?while disabling rdma on opa we would accept the worse > > >> performance via opa . We hope that this provides ?still better > perf > > >> and less technical overhead ?than using routers > > >> > > >> Or am I totally wrong? > > >> > > >> Thank you very much and keep healthy! > > >> > > >> Best regards > > >> > > >> Walter > > >> > > >> Mit freundlichen Gr??en > > >> */Walter Sklenka/* > > >> */Technical Consultant/* > > >> > > >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 > > >> Wien > > >> Tel: +43 1 29 22 165-31 > > >> Fax: +43 1 29 22 165-90 > > >> E-Mail: sklenka at edv-design.at > > > >> Internet: www.edv-design.at > > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > -- > > > Giovanni Bracco > > > phone? +39 351 8804788 > > > E-mail giovanni.bracco at enea.it > > > WWW http://www.afs.enea.it/bracco > > > > > -- > > Giovanni Bracco > > phone? +39 351 8804788 > > E-mail giovanni.bracco at enea.it > > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco From henrik at morsing.cc Wed Feb 3 16:18:44 2021 From: henrik at morsing.cc (Henrik Morsing) Date: Wed, 3 Feb 2021 16:18:44 +0000 Subject: [gpfsug-discuss] Intro Message-ID: <20210203161844.GO14457@morsing.cc> Hi all, I live in Buckinghamshire in the UK, but work in portsmouth for the energy company SSE. Some years ago, we bought in two 840TB Spectrum Scale systems to replace our tape libraries. I was put on the project to get it up and running and the tape library phased out, but the project has stalled multiple times. I am back on the project now, every time it feels like I am starting over, and on top I need to patch it again which was a massive undertaking last time, so I am not looking forward to that. The clusters are connected via a six link 10Gb/s etherchannel to our Spectrum Protect backup servers. Two of these have been setup, including cross-site replication job, so we do have some idea of it working, but performance monitoring is another aspect I need to look at. I am planning to install njmon+Grafana to see how that looks. Hope to get lots of tips from this forum! Regards, Henrik Morsing From jonathan.b.mills at nasa.gov Mon Feb 8 16:00:20 2021 From: jonathan.b.mills at nasa.gov (Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC]) Date: Mon, 8 Feb 2021 16:00:20 +0000 Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Message-ID: Dear Community, We have multiple GPFS clusters within our internal networks. We want to expose more than one GPFS GUI interface, from behind a shared web proxy / firewall setup, which has SAML2 integration with our organization's IDP. (The SAML2 is not part of the question, it's just a rationale for doing it this way.) Essentially, we're trying to do something like https:///gpfs/gui1 and https:///gpfs/gui2 The websphere config in the GPFS GUI packages basically assume a document root of "/". This poses additional problems when it forms relative URLs...for example, immediately after login, it tries to redirect from "/" to "/gui". Through our web proxy, this ends up being https:///gui which is a 404. While problems might be addressable with some complex mod_rewrite in our web proxy, I've been looking for a more straightforward solution. In /opt/ibm/wlp/usr/servers/gpfsgui/server.xml, it's possible to change the document root in line 34: Changing the document root would allow me to mod_proxy from https:///gpfs/gui1 to https:///gpfs/gui1, and in theory that makes it easy to support many GPFS GUIs, and should fix the relative URL / redirect issue. However, while it seems like this ought to simplify matters, in fact it just breaks other parts of the GPFS GUI, because it calls URLs that aren't part of "ROOT.war" and those also assume a document root of "/". So, my basic question for the community is this: Has anyone out there successfully mucked with the GPFS GUI document root? Or am I just barking up the wrong tree? Thanks in advance, Jonathan -- Jonathan Mills / jonathan.mills at nasa.gov NASA GSFC / NCCS HPC (606.2) Bldg 28, Rm. S230 / c. 252-412-5710 From heinrich.billich at id.ethz.ch Mon Feb 8 21:27:27 2021 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Mon, 8 Feb 2021 21:27:27 +0000 Subject: [gpfsug-discuss] Mmapplypolicy with -I defer doesn't sort resulting list? Is this by intention? Message-ID: <9B403CA7-7785-480E-ACC1-38561F0797CD@id.ethz.ch> Hello, I want to migrate data with mmapplypolicy to a different pool: 1. create a file list with "-I defer -f /some/path" 2. execute with "-I yes -r /some/path" I noted that the file list created in 1. Is not sorted. I asked to sort by kb_allocated, the ideas is to migrate the largest files first and to stop once we freed enough space. Is this intended, or should the file list created in 1. be sorted? I want to be able to review the list before I start the migration. The list holds size/kb_allocated in hex numbers, I didn't manage to sort it with 'sort'. Hence I would prefer to get a sorted list from mmapplypolicy instead of writing some code to do this. The list holds files with sizes between 32K and 250G and 4.5M lines, hence it makes sense to sort to move the largest files on the top. Thank you and best regards, Heiner The policy is RULE 'migrate' MIGRATE FROM POOL 'data' THRESHOLD (50,50) WEIGHT (kb_allocated) TO POOL 'capacity' LIMIT (98) WHERE kb_allocated > 0 AND (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) > 90 And the command line mmapplypolicy $1 -A 600 \ -a 10 \ -P $base/policy.txt \ -I prepare \ -f $base/candidates \ -L 1 \ -N some_nodes \ --choice-algorithm fast \ --max-sort-bytes 1G \ --split-margin 2.0 The filesystem holds about 1.2G inodes. I tried to speed-up sorting with the last three arguments. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5254 bytes Desc: not available URL: From jon at well.ox.ac.uk Tue Feb 9 09:13:55 2021 From: jon at well.ox.ac.uk (Jon Diprose) Date: Tue, 9 Feb 2021 09:13:55 +0000 Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml In-Reply-To: References: Message-ID: Might it be easier to add gpfs1. and gpfs2. as CNAMEs to your web proxy / firewall , and then configure the web proxy to select the backend on the basis of the requested hostname? https://gpfs1./ redirects to https://gpfs1./gui/ and there's no mucking about with gpfs stuff required at all. Jon -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC] Sent: 08 February 2021 16:00 To: gpfsug main discussion list Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Dear Community, We have multiple GPFS clusters within our internal networks. We want to expose more than one GPFS GUI interface, from behind a shared web proxy / firewall setup, which has SAML2 integration with our organization's IDP. (The SAML2 is not part of the question, it's just a rationale for doing it this way.) Essentially, we're trying to do something like https:///gpfs/gui1 and https:///gpfs/gui2 The websphere config in the GPFS GUI packages basically assume a document root of "/". This poses additional problems when it forms relative URLs...for example, immediately after login, it tries to redirect from "/" to "/gui". Through our web proxy, this ends up being https:///gui which is a 404. While problems might be addressable with some complex mod_rewrite in our web proxy, I've been looking for a more straightforward solution. In /opt/ibm/wlp/usr/servers/gpfsgui/server.xml, it's possible to change the document root in line 34: Changing the document root would allow me to mod_proxy from https:///gpfs/gui1 to https:///gpfs/gui1, and in theory that makes it easy to support many GPFS GUIs, and should fix the relative URL / redirect issue. However, while it seems like this ought to simplify matters, in fact it just breaks other parts of the GPFS GUI, because it calls URLs that aren't part of "ROOT.war" and those also assume a document root of "/". So, my basic question for the community is this: Has anyone out there successfully mucked with the GPFS GUI document root? Or am I just barking up the wrong tree? Thanks in advance, Jonathan -- Jonathan Mills / jonathan.mills at nasa.gov NASA GSFC / NCCS HPC (606.2) Bldg 28, Rm. S230 / c. 252-412-5710 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From scale at us.ibm.com Tue Feb 9 10:55:45 2021 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 9 Feb 2021 16:25:45 +0530 Subject: [gpfsug-discuss] Mmapplypolicy with -I defer doesn't sort resulting list? Is this by intention? In-Reply-To: <9B403CA7-7785-480E-ACC1-38561F0797CD@id.ethz.ch> References: <9B403CA7-7785-480E-ACC1-38561F0797CD@id.ethz.ch> Message-ID: Hi Billich, I think the problem is that you are specifying --choice-algorithm fast and as per documentation "The fast choice method does not completely sort the candidates by weight." To sort the list you can try specifying --choice-algorithm exact which is also the default. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 09-02-2021 03.13 AM Subject: [EXTERNAL] [gpfsug-discuss] Mmapplypolicy with -I defer doesn't sort resulting list? Is this by intention? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I want to migrate data with mmapplypolicy to a different pool: 1. create a file list with "-I defer -f /some/path" 2. execute with "-I yes -r /some/path" I noted that the file list created in 1. Is not sorted. I asked to sort by kb_allocated, the ideas is to migrate the largest files first and to stop once we freed enough space. Is this intended, or should the file list created in 1. be sorted? I want to be able to review the list before I start the migration. The list holds size/kb_allocated in hex numbers, I didn't manage to sort it with 'sort'. Hence I would prefer to get a sorted list from mmapplypolicy instead of writing some code to do this. The list holds files with sizes between 32K and 250G and 4.5M lines, hence it makes sense to sort to move the largest files on the top. Thank you and best regards, Heiner The policy is RULE 'migrate' MIGRATE FROM POOL 'data' THRESHOLD (50,50) WEIGHT (kb_allocated) TO POOL 'capacity' LIMIT (98) WHERE kb_allocated > 0 AND (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) > 90 And the command line mmapplypolicy $1 -A 600 \ -a 10 \ -P $base/policy.txt \ -I prepare \ -f $base/candidates \ -L 1 \ -N some_nodes \ --choice-algorithm fast \ --max-sort-bytes 1G \ --split-margin 2.0 The filesystem holds about 1.2G inodes. I tried to speed-up sorting with the last three arguments. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== [attachment "smime.p7s" deleted by Huzefa H Pancha/India/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From jonathan.b.mills at nasa.gov Tue Feb 9 15:07:21 2021 From: jonathan.b.mills at nasa.gov (Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC]) Date: Tue, 9 Feb 2021 15:07:21 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Re: [gpfs][gui] Change document root in server.xml In-Reply-To: References: , Message-ID: Oh, without a doubt, and we considered it. However, the agency where I work makes it rather difficult to register new DNS names, and they actively discourage standing up new websites. But yes, I agree that is the simplest solution, and we may be forced to attempt it and wade through the red tape. Thank you for your response. Jonathan ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Jon Diprose Sent: Tuesday, February 9, 2021 4:13 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Might it be easier to add gpfs1. and gpfs2. as CNAMEs to your web proxy / firewall , and then configure the web proxy to select the backend on the basis of the requested hostname? https://gpfs1./ redirects to https://gpfs1./gui/ and there's no mucking about with gpfs stuff required at all. Jon -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC] Sent: 08 February 2021 16:00 To: gpfsug main discussion list Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Dear Community, We have multiple GPFS clusters within our internal networks. We want to expose more than one GPFS GUI interface, from behind a shared web proxy / firewall setup, which has SAML2 integration with our organization's IDP. (The SAML2 is not part of the question, it's just a rationale for doing it this way.) Essentially, we're trying to do something like https:///gpfs/gui1 and https:///gpfs/gui2 The websphere config in the GPFS GUI packages basically assume a document root of "/". This poses additional problems when it forms relative URLs...for example, immediately after login, it tries to redirect from "/" to "/gui". Through our web proxy, this ends up being https:///gui which is a 404. While problems might be addressable with some complex mod_rewrite in our web proxy, I've been looking for a more straightforward solution. In /opt/ibm/wlp/usr/servers/gpfsgui/server.xml, it's possible to change the document root in line 34: Changing the document root would allow me to mod_proxy from https:///gpfs/gui1 to https:///gpfs/gui1, and in theory that makes it easy to support many GPFS GUIs, and should fix the relative URL / redirect issue. However, while it seems like this ought to simplify matters, in fact it just breaks other parts of the GPFS GUI, because it calls URLs that aren't part of "ROOT.war" and those also assume a document root of "/". So, my basic question for the community is this: Has anyone out there successfully mucked with the GPFS GUI document root? Or am I just barking up the wrong tree? Thanks in advance, Jonathan -- Jonathan Mills / jonathan.mills at nasa.gov NASA GSFC / NCCS HPC (606.2) Bldg 28, Rm. S230 / c. 252-412-5710 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Cjonathan.b.mills%40nasa.gov%7Cb4b847626cd347c347f408d8ccdb21f2%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637484588838656778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=d9r02MnZDlXpfiMTY5dxNIR0FGBORNKMiHUh8MS5NRs%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Cjonathan.b.mills%40nasa.gov%7Cb4b847626cd347c347f408d8ccdb21f2%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637484588838656778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=d9r02MnZDlXpfiMTY5dxNIR0FGBORNKMiHUh8MS5NRs%3D&reserved=0 From helge.hauglin at usit.uio.no Tue Feb 9 15:46:39 2021 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Tue, 09 Feb 2021 16:46:39 +0100 Subject: [gpfsug-discuss] Limiting CES SMB shares to specific subnets Message-ID: Hi. We have an ESS 5.0.4.3 cluster with a CES cluster serving files with NFSv4 ACLs to NFS and SMB clients. This system is used for sensitive research data, and will the next years house thousands of research projects, which will have to be strictly separated. Each project has its own subnet for the project linux and windows hosts. Project directories are independent filesets in file systems, each project directory has NFSv4 ACLs giving acces to only the project group. Project NFS shares are limited to each project's subnet. Project SMB shares have export ACLs (as in "mmsmb exportacl ..") limiting share access to the project's member group, in addition to the NFSv4 ACLs. We also want to limit access to SMB shares to project subnets. There is no way to specify that with "mmsmb", but we have found /usr/lpp/mmfs/bin/net conf setparm "hosts allow" to be working, at least with some limited testing: share access is actually limited to the specified subnets. The additional settings seems to be stored in CTDB under /var/lib/ctdb/persistent. We assume that the "net conf setparm" method is not officially supported by IBM. Although it seems to be working, we wonder if it is a good idea to implement it. For instance, we are wondering if the additional settings will survive later ESS code upgrades, and if it will scale to thousands of SMB shares. We are considering doing the SMB subnet limiting outside CES, but that would add complexity and overhead, so we are not very keen on that. What do other IBM ESS customers do, do you have any advice for us? Yea or nay? Regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From christof.schmitt at us.ibm.com Tue Feb 9 18:06:14 2021 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 9 Feb 2021 18:06:14 +0000 Subject: [gpfsug-discuss] Limiting CES SMB shares to specific subnets In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From helge.hauglin at usit.uio.no Wed Feb 10 09:09:01 2021 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Wed, 10 Feb 2021 10:09:01 +0100 Subject: [gpfsug-discuss] Limiting CES SMB shares to specific subnets In-Reply-To: (Christof Schmitt's message of "Tue, 9 Feb 2021 18:06:14 +0000") References: Message-ID: Hi Christof, thanks for your answer. I have added our vote for the RFE, and put us on the watchlist. Is it possible to say anything about when the RFE might be implemented? >> Project SMB shares have export ACLs (as in "mmsmb exportacl ..")> limiting share access to the project's member group, in addition to the> NFSv4 ACLs.>> We also want to limit access to SMB shares to project subnets.> There is no way to specify that with "mmsmb", but we have found>> /usr/lpp/mmfs/bin/net conf setparm "hosts allow" >> to be working, at least with some limited testing: share access is> actually limited to the specified subnets. The additional settings> seems to be stored in CTDB under /var/lib/ctdb/persistent.>> We assume that the "net conf setparm" method is not officially supported> by IBM. Although it seems to be working, we wonder if it is a good idea> to implement it. For instance, we are wondering if the additional> settings will survive later ESS code upgrades, and if it will scale to> thousands of SMB shares. > > Officially Scale only supports Samba options that can be set throughthe GUI or the mmsmb CLI. Everything else set through 'net conf' hasnot been tested and is not supported. In this specific case, this islikely to work, and it should also be preserved across code upgrades,but again, this is not an official support statement. > > This is also not a new request, there is also a pending RFE to makethis an official Scale feature:https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=141534 > > Regards, > > Christof Schmitt > Software Engineer > IBM Systems, Spectrum Scale Development > +1 520 799 2469 > christof.schmitt at us.ibm.com > @chsc Twitter > > IBM > > > ----- Original message -----From: Helge Hauglin Sent by: gpfsug-discuss-bounces at spectrumscale.orgTo: gpfsug-discuss at spectrumscale.orgCc:Subject: [EXTERNAL] [gpfsug-discuss] Limiting CES SMB shares to specific subnetsDate: Tue, Feb 9, 2021 9:10 AM > Hi.We have an ESS 5.0.4.3 cluster with a CES cluster serving files withNFSv4 ACLs to NFS and SMB clients. This system is used forsensitive research data, and will the next years house thousands ofresearch projects, which will have to be strictly separated. Eachproject has its own subnet for the project linux and windows hosts.Project directories are independent filesets in file systems, eachproject directory has NFSv4 ACLs giving acces to only the project group.Project NFS shares are limited to each project's subnet.Project SMB shares have export ACLs (as in "mmsmb exportacl ..")limiting share access to the project's member group, in addition to theNFSv4 ACLs.We also want to limit access to SMB shares to project subnets.There is no way to specify that with "mmsmb", but we have found /usr/lpp/mmfs/bin/net conf setparm "hosts allow" to be working, at least with some limited testing: share access isactually limited to the specified subnets. The additional settingsseems to be stored in CTDB under /var/lib/ctdb/persistent.We assume that the "net conf setparm" method is not officially supportedby IBM. Although it seems to be working, we wonder if it is a good ideato implement it. For instance, we are wondering if the additionalsettings will survive later ESS code upgrades, and if it will scale tothousands of SMB shares.We are considering doing the SMB subnet limiting outside CES, but that wouldadd complexity and overhead, so we are not very keen on that.What do other IBM ESS customers do, do you have any advice for us?Yea or nay?Regards,Helge Hauglin----------------------------------------------------------------Mr. Helge Hauglin, Senior EngineerSystem administratorCenter for Information Technology, University of Oslo, Norway_______________________________________________gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From janfrode at tanso.net Fri Feb 19 12:50:08 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 19 Feb 2021 13:50:08 +0100 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180119163803.79fddbeb@osc.edu> References: <20180119163803.79fddbeb@osc.edu> Message-ID: We just discussed this a bit internally, and I found "something* that might help... There's a mmrestripefs --inode-criteria command that can be used to identify files with these unknown-to-ILM flags set. Something like: # echo illreplicated > criteria # mmrestripefs gpfs01 -p --inode-criteria criteria -o result Scanning file system metadata, phase 1 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning file system metadata, phase 2 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning file system metadata, phase 3 ... Scan completed successfully. Scanning file system metadata, phase 4 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning file system metadata, phase 5 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning user file metadata ... 100.00 % complete on Thu Feb 18 15:30:14 2021 ( 835584 inodes with total 4271 MB data processed) Scan completed successfully. Check file '/mnt/gpfs01/result' on scale-dev-01 for inodes with broken disk addresses or failures. # cat /mnt/gpfs01/result This inode list was generated in the Parallel Inode Traverse on Thu Feb 18 15:30:10 2021 INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR]) 55559 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE Unclear to me if "-p" is too much work for this task though, or maybe it can be limited with --metadata-only or other options.. HTH On Sat, Jan 20, 2018 at 12:10 AM Edward Wahl wrote: > > This one has been on my list a long time so I figured I'd ask here first > before > I open an apar or request an enhancement (most likely). > > Is there a way using the policy engine to determine the following? > > -metadata replication total/current > -unbalanced file > > Looking to catch things like this that stand out on my filesystem without > having to run several hundred million 'mmlsattr's. > > metadata replication: 1 max 2 > flags: unbalanced > > Ed > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Fri Feb 19 16:29:01 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 19 Feb 2021 17:29:01 +0100 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: References: <20180119163803.79fddbeb@osc.edu> Message-ID: I also see that /usr/lpp/mmfs/samples/util/tsinode will list these flags: [root at scale-dev-01 util]# ./tsinode /mnt/gpfs01/|grep 55559 55559 631878862 0 0 1048576000 104857600 -rw-r--r-- 2 1610965529.234470000 1610965531.035007326 1612952829.394919000 0 blk=1048576 modsnap=1 extperms=0x2,xa replmeta *illReplicated unbalanced* dev=3824,150 archive compressed crtime 1610965529.234470000 That program can probably easily be modified to only list these files.. -jf On Fri, Feb 19, 2021 at 1:50 PM Jan-Frode Myklebust wrote: > We just discussed this a bit internally, and I found "something* that > might help... There's a mmrestripefs --inode-criteria command that can be > used to identify files with these unknown-to-ILM flags set. Something like: > > # echo illreplicated > criteria > # mmrestripefs gpfs01 -p --inode-criteria criteria -o result > Scanning file system metadata, phase 1 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning file system metadata, phase 2 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning file system metadata, phase 3 ... > Scan completed successfully. > Scanning file system metadata, phase 4 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning file system metadata, phase 5 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning user file metadata ... > 100.00 % complete on Thu Feb 18 15:30:14 2021 ( 835584 inodes with total 4271 MB data processed) > Scan completed successfully. > Check file '/mnt/gpfs01/result' on scale-dev-01 for inodes with broken disk addresses or failures. > # cat /mnt/gpfs01/result > This inode list was generated in the Parallel Inode Traverse on Thu Feb 18 15:30:10 2021 > INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR]) > 55559 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE > > Unclear to me if "-p" is too much work for this task though, or maybe it can be limited with --metadata-only or other options.. > > > HTH > > > > > On Sat, Jan 20, 2018 at 12:10 AM Edward Wahl wrote: > >> >> This one has been on my list a long time so I figured I'd ask here first >> before >> I open an apar or request an enhancement (most likely). >> >> Is there a way using the policy engine to determine the following? >> >> -metadata replication total/current >> -unbalanced file >> >> Looking to catch things like this that stand out on my filesystem without >> having to run several hundred million 'mmlsattr's. >> >> metadata replication: 1 max 2 >> flags: unbalanced >> >> Ed >> >> >> >> -- >> >> Ed Wahl >> Ohio Supercomputer Center >> 614-292-9302 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.trafford at yale.edu Tue Feb 23 00:39:36 2021 From: tyler.trafford at yale.edu (Trafford, Tyler) Date: Tue, 23 Feb 2021 00:39:36 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu>, <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu>, Message-ID: My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn?t really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 From TROPPENS at de.ibm.com Wed Feb 24 16:17:05 2021 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Wed, 24 Feb 2021 16:17:05 +0000 Subject: [gpfsug-discuss] Digital German Spectrum Scale User Meeting on March 3&4, 2021 Message-ID: An HTML attachment was scrubbed... URL: From jon at well.ox.ac.uk Thu Feb 25 10:13:09 2021 From: jon at well.ox.ac.uk (Jon Diprose) Date: Thu, 25 Feb 2021 10:13:09 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu>, <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu>, Message-ID: I have written an xCAT plugin such that the client can ask the master to do it, in preparation for some shiny new kit. Not in a public repo yet, so please email me direct if you want further info. -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Trafford, Tyler Sent: 23 February 2021 00:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn't really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jpr9c at virginia.edu Thu Feb 25 15:47:41 2021 From: jpr9c at virginia.edu (Ruffner, Scott (jpr9c)) Date: Thu, 25 Feb 2021 15:47:41 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu> <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> Message-ID: I appreciate the solutions; at the moment, I'm making do with having a key for just this specific copy, and doing our own rsync. If we had local disk on our nodes, that would be helpful, but /var/mmfs is mounted on tmpfs and gets flushed at every reboot (not just upgrades to hardware or software stack. I'm testing with invoking the mmsdrrestore from the client only using a key. Yeah, not ideal leaving that laying around. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu ?On 2/25/21, 5:13 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jon Diprose" wrote: I have written an xCAT plugin such that the client can ask the master to do it, in preparation for some shiny new kit. Not in a public repo yet, so please email me direct if you want further info. -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Trafford, Tyler Sent: 23 February 2021 00:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn't really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jon at well.ox.ac.uk Thu Feb 25 16:18:47 2021 From: jon at well.ox.ac.uk (Jon Diprose) Date: Thu, 25 Feb 2021 16:18:47 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu> <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> Message-ID: The call to the plugin can be invoked directly from the client at any time, not just as a result of an install or an updatenode. I see no reason it couldn't go in a script that's run on client startup, though you'd have to hard-code the address+port the xcatd is listening on as that would normally be provided by xCAT. I started from the remoteshell postscript and worked backwards through the getcredentials.awk script to the credentials.pm plugin, and re-used what I needed for a plugin that invokes relevant mmgetstate, mmaddnode and/or mmsdrrestore calls on the master. Alternatively, just look at those existing postscripts/plugins for a mechanism to dynamically obtain the key, which can then be ephemeral on the client. That is the code path xCAT uses to get keys and passwords to the clients, though the comments at the top of credentials.pm (in /opt/xcat/lib/perl/xCAT_plugins/) give a fairly brutal assessment of the security situation. Jon -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Ruffner, Scott (jpr9c) Sent: 25 February 2021 15:48 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. I appreciate the solutions; at the moment, I'm making do with having a key for just this specific copy, and doing our own rsync. If we had local disk on our nodes, that would be helpful, but /var/mmfs is mounted on tmpfs and gets flushed at every reboot (not just upgrades to hardware or software stack. I'm testing with invoking the mmsdrrestore from the client only using a key. Yeah, not ideal leaving that laying around. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu ?On 2/25/21, 5:13 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jon Diprose" wrote: I have written an xCAT plugin such that the client can ask the master to do it, in preparation for some shiny new kit. Not in a public repo yet, so please email me direct if you want further info. -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Trafford, Tyler Sent: 23 February 2021 00:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn't really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Fri Feb 26 01:49:42 2021 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 26 Feb 2021 01:49:42 +0000 Subject: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Message-ID: <4397068B-AD6C-445E-AA07-390FBA7B743F@nuance.com> I have 2 nodes in a new ESS cluster that won?t display data via the GUI, but I can pull up data using the command line. Node is healthy, pmsensors is running and shows connected to the collector. Command line shows this: [root at ems1 IBM]# mmperfmon query compareNodes cpu_user -b 3600 -n 2 Legend: 1: afmgw03|CPU|cpu_user 2: afmgw04|CPU|cpu_user 3: ems1.gpfs.net|CPU|cpu_user 4: essio1.gpfs.net|CPU|cpu_user 5: essio2.gpfs.net|CPU|cpu_user Row Timestamp afmgw03 afmgw04 ems1 essio1 essio2 1 2021-02-25-19:00:00 0.069542 0.059736 0.231 0.196292 0.213047 2 2021-02-25-20:00:00 0.076463 0.067958 0.270258 0.221003 0.230289 But when I try and view afmgw03 or afmgw04 on the GUI, no data. What?s up? Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Fri Feb 26 12:18:24 2021 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 26 Feb 2021 07:18:24 -0500 Subject: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line In-Reply-To: <4397068B-AD6C-445E-AA07-390FBA7B743F@nuance.com> References: <4397068B-AD6C-445E-AA07-390FBA7B743F@nuance.com> Message-ID: Bob, could you please provide the version of ESS/Scale you have installed? Also, could you please provide information about the exact GUI screen you are using that is not providing the data? Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 02/25/2021 09:41 PM Subject: [EXTERNAL] [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Sent by: gpfsug-discuss-bounces at spectrumscale.org I have 2 nodes in a new ESS cluster that won?t display data via the GUI, but I can pull up data using the command line. Node is healthy, pmsensors is running and shows connected to the collector. Command line shows this: [root at ems1 IBM]# mmperfmon query compareNodes cpu_user -b 3600 -n 2 Legend: 1: afmgw03|CPU|cpu_user 2: afmgw04|CPU|cpu_user 3: ems1.gpfs.net|CPU|cpu_user 4: essio1.gpfs.net|CPU|cpu_user 5: essio2.gpfs.net|CPU|cpu_user Row Timestamp afmgw03 afmgw04 ems1 essio1 essio2 1 2021-02-25-19:00:00 0.069542 0.059736 0.231 0.196292 0.213047 2 2021-02-25-20:00:00 0.076463 0.067958 0.270258 0.221003 0.230289 But when I try and view afmgw03 or afmgw04 on the GUI, no data. What?s up? Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=_afEd51Its1IKJqf-evUvjCHSBCrdDMxTJWn0juEBBI&s=Vn6zN83tuKly6fA8xw128AYfimkX9PZFDY9TRBEQ45s&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Feb 26 13:36:57 2021 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 26 Feb 2021 13:36:57 +0000 Subject: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Message-ID: <5F3EDDA0-28C2-460E-8E99-77E728E4618D@nuance.com> ESS 6.0.1.2 with GPFS 5.0.5.4 What?s even stranger is that I logged in this morning and it was working so? ? Not sure if we should/need to follow-up. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of IBM Spectrum Scale Reply-To: gpfsug main discussion list Date: Friday, February 26, 2021 at 6:18 AM To: gpfsug main discussion list Cc: "gpfsug-discuss-bounces at spectrumscale.org" Subject: [EXTERNAL] Re: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line CAUTION: This Email is from an EXTERNAL source. Ensure you trust this sender before clicking on any links or attachments. ________________________________ Bob, could you please provide the version of ESS/Scale you have installed? Also, could you please provide information about the exact GUI screen you are using that is not providing the data? Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 02/25/2021 09:41 PM Subject: [EXTERNAL] [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I have 2 nodes in a new ESS cluster that won?t display data via the GUI, but I can pull up data using the command line. Node is healthy, pmsensors is running and shows connected to the collector. Command line shows this: [root at ems1 IBM]# mmperfmon query compareNodes cpu_user -b 3600 -n 2 Legend: 1: afmgw03|CPU|cpu_user 2: afmgw04|CPU|cpu_user 3: ems1.gpfs.net|CPU|cpu_user 4: essio1.gpfs.net|CPU|cpu_user 5: essio2.gpfs.net|CPU|cpu_user Row Timestamp afmgw03 afmgw04 ems1 essio1 essio2 1 2021-02-25-19:00:00 0.069542 0.059736 0.231 0.196292 0.213047 2 2021-02-25-20:00:00 0.076463 0.067958 0.270258 0.221003 0.230289 But when I try and view afmgw03 or afmgw04 on the GUI, no data. What?s up? Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=_afEd51Its1IKJqf-evUvjCHSBCrdDMxTJWn0juEBBI&s=Vn6zN83tuKly6fA8xw128AYfimkX9PZFDY9TRBEQ45s&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Sat Feb 27 18:01:02 2021 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sat, 27 Feb 2021 18:01:02 +0000 Subject: [gpfsug-discuss] dssgmkfs.mmvdisk number of NSD's Message-ID: Doing an upgrade on our storage which involved replacing all the 4TB disks with 16TB disks. Some hiccups with five of the disks being dead when inserted but that is all sorted. So the system was originally installed with DSS-G 2.0a so with "legacy" commands for vdisks etc. We had 10 metadata NSD's and 10 data NSD's per draw aka recovery group of the D3284 enclosures. The dssgmkfs.mmvdisk has created exactly one data and one metadata NSD per draw of a DS3284 leading to a really small number of NSD's in the file system. All my instincts tell me that this is going to lead to horrible performance on the file system. Historically you wanted a reasonable number of NSD's in a system for decent performance. Taking what the ddsgmkfs.mmvdisk has give me even with a DSS-G260 you would get only 12 NSD's of each type, which for a potentially ~5PB file system seems on the really low side to me. Is there any way to tell ddsgmkfs.mmvdisk to create more NSD's than the one per recovery group or is this no longer relevant and performance with really low numbers of NSD's is fine these days? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From janfrode at tanso.net Sun Feb 28 09:31:57 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 28 Feb 2021 10:31:57 +0100 Subject: [gpfsug-discuss] dssgmkfs.mmvdisk number of NSD's In-Reply-To: References: Message-ID: I?ve tried benchmarking many vs. few vdisks per RG, and never could see any performance difference. Usually we create 1 vdisk per enclosure per RG, thinking this will allow us to grow with same size vdisks when adding additional enclosures in the future. Don?t think mmvdisk can be told to create multiple vdisks per RG directly, so you have to manually create multiple vdisk sets each with the apropriate size. -jf l?r. 27. feb. 2021 kl. 19:01 skrev Jonathan Buzzard < jonathan.buzzard at strath.ac.uk>: > > Doing an upgrade on our storage which involved replacing all the 4TB > disks with 16TB disks. Some hiccups with five of the disks being dead > when inserted but that is all sorted. > > So the system was originally installed with DSS-G 2.0a so with "legacy" > commands for vdisks etc. We had 10 metadata NSD's and 10 data NSD's per > draw aka recovery group of the D3284 enclosures. > > The dssgmkfs.mmvdisk has created exactly one data and one metadata NSD > per draw of a DS3284 leading to a really small number of NSD's in the > file system. > > All my instincts tell me that this is going to lead to horrible > performance on the file system. Historically you wanted a reasonable > number of NSD's in a system for decent performance. > > Taking what the ddsgmkfs.mmvdisk has give me even with a DSS-G260 you > would get only 12 NSD's of each type, which for a potentially ~5PB file > system seems on the really low side to me. > > Is there any way to tell ddsgmkfs.mmvdisk to create more NSD's than the > one per recovery group or is this no longer relevant and performance > with really low numbers of NSD's is fine these days? > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Mon Feb 1 15:08:19 2021 From: p.childs at qmul.ac.uk (Peter Childs) Date: Mon, 1 Feb 2021 15:08:19 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu>, <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> Message-ID: We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn?t really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From daniel.kidger at uk.ibm.com Mon Feb 1 16:25:11 2021 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Mon, 1 Feb 2021 16:25:11 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From Adam.Willis at ibm.com Mon Feb 1 17:15:18 2021 From: Adam.Willis at ibm.com (Adam Willis) Date: Mon, 1 Feb 2021 17:15:18 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From janfrode at tanso.net Mon Feb 1 18:11:26 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 1 Feb 2021 19:11:26 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: Agree.. Write a policy that takes a "mmapplypolicy -M var=val" argument, and figure out the workdays outside of the policy. Something like: # cat test.poilcy define( access_age, (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) /* list migrated files */ RULE EXTERNAL LIST 'oldFiles' EXEC '' RULE 'oldFiles' LIST 'oldFiles' WHERE (access_age > MINAGE) # mmapplypolicy gpfs01 -P test.policy -I defer -f ./filelist -M MINAGE=5 On Mon, Feb 1, 2021 at 5:29 PM Daniel Kidger wrote: > To me, it feels like you need to do the search the other way around. > > Firstly, from the target of say "4 weekdays ago", work out how many real > days ago that was. > Then use that as a criteria in mmfind or mmapplypolicy. > > Daniel > > _________________________________________________________ > *Daniel Kidger Ph.D.* > IBM Technical Sales Specialist > Spectrum Scale, Spectrum Discover and IBM Cloud Object Storage > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > > > > > > > > > ----- Original message ----- > From: Jonathan Buzzard > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, > Issue 18 > Date: Sat, Jan 30, 2021 10:29 > > On 30/01/2021 00:31, Owen Morgan wrote: > > [SNIP] > > > > > I would prefer to stay in the bounds of the SQL policy rule setup as > > that is the framework I have created and started to implement.. > > > In general SQL is Turing complete. Though I have not checked in detail I > believe the SQL of the policy engine is too. > > I would also note that SQL has a whole bunch of time/date functions. > > So something like > > define(offset, 4) > define(day, DAYOFWEEK(CURRENT_TIMESTAMP)) > define(age,(DAYS(CURRENT_TIMESTAMP)-DAYS(ACCESS_TIME))) > define(workingdays, > CASE > WHEN day=1 > THEN offest+1 > WHEN day=6 > THEN offset > WHEN day=7 > THEN offset+1 > ELSE > offset+2 > ) > > /* delete all files from files older than 4 working days */ > RULE purge4 DELETE > WHERE (age>workingdays) FOR FILESET dummies > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From giovanni.bracco at enea.it Mon Feb 1 19:41:43 2021 From: giovanni.bracco at enea.it (Giovanni Bracco) Date: Mon, 1 Feb 2021 20:41:43 +0100 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> Message-ID: <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> On 30/01/21 21:01, Walter Sklenka wrote: > Hi Giovanni! > Thats great! Many thanks for your fast and detailed answer!!!! > So this is the way we will go too! > > Have a nice weekend and keep healthy! > Best regards > Walter > I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works! If yu have any issues I am ready to discuss! Regards Giovanni > -----Original Message----- > From: Giovanni Bracco > Sent: Samstag, 30. J?nner 2021 18:08 > To: gpfsug main discussion list ; Walter Sklenka > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > See > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a multifabric GPFS Spectrum Scale layout," 2019 International Conference on High Performance Computing & Simulation (HPCS), Dublin, Ireland, 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > When setting up the system the main trick has been: > just use CentOS drivers and do not install OFED We do not use IPoIB. > > Giovanni > > On 30/01/21 06:45, Walter Sklenka wrote: >> Hi! >> >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? >> >> In the faq >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. >> html#rdma >> >> >> They talk about RDMA : >> >> "RDMA is NOT ?supported on a node when both Mellanox HCAs and Intel >> Omni-Path HFIs are ENABLED for RDMA." >> >> So do I understand right: When we do NOT enable ?the opa interface we >> can still enable IB ? >> >> The reason I ask ?is, that we have a gpfs cluster of 6 NSD Servers >> (wih access to storage) ?with opa interfaces which provide access to >> remote cluster ?also via OPA. >> >> A new cluster with HDR interfaces will be implemented soon >> >> They shell have access to the same filesystems >> >> When we add HDR interfaces to? NSD servers? and enable rdma on this >> network ?while disabling rdma on opa we would accept the worse >> performance via opa . We hope that this provides ?still better perf >> and less technical overhead ?than using routers >> >> Or am I totally wrong? >> >> Thank you very much and keep healthy! >> >> Best regards >> >> Walter >> >> Mit freundlichen Gr??en >> */Walter Sklenka/* >> */Technical Consultant/* >> >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 >> Wien >> Tel: +43 1 29 22 165-31 >> Fax: +43 1 29 22 165-90 >> E-Mail: sklenka at edv-design.at >> Internet: www.edv-design.at >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Giovanni Bracco > phone +39 351 8804788 > E-mail giovanni.bracco at enea.it > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco From jonathan.buzzard at strath.ac.uk Mon Feb 1 20:17:34 2021 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 1 Feb 2021 20:17:34 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: On 01/02/2021 18:11, Jan-Frode Myklebust wrote: > CAUTION: This email originated outside the University. Check before > clicking links or attachments. > Agree.. Write a policy that takes a "mmapplypolicy -M var=val" argument, > and figure out the workdays outside of the policy. Something like: > > # cat test.poilcy > define( access_age, ? ? (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) > /* list migrated files */ > RULE EXTERNAL LIST 'oldFiles' EXEC '' > RULE 'oldFiles' LIST 'oldFiles' > ? ? WHERE (access_age > MINAGE) > > #?mmapplypolicy gpfs01 ?-P test.policy -I defer -f ./filelist -M MINAGE=5 > Why bother when you can do it all in the policy? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From owen.morgan at motionpicturesolutions.com Mon Feb 1 21:09:16 2021 From: owen.morgan at motionpicturesolutions.com (Owen Morgan) Date: Mon, 1 Feb 2021 21:09:16 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 Message-ID: Jonathan, If I have a single policy file with all the related department rules and each time they want to add additional rules with different working day thresholds maybe using this -M method is easier. Its clear that the 'maths' and date/timestamp manipulation is easier in shell (my preferred is bash) than in the SQL of the policy (your example is succinct but needs to be repeated everytime a new rule is added with a different working day threshold, which is what I'm trying (if possiblr) to avoid. It seems to me the IBM SQL engine is perhaps missing more 'SQL' in built date/time functions like DateAdd and DateDiff etc.. as this would be a moot point. Its a shame I can't make one function that given a working day input as an argument spits out how many 'real' days exist between them for the file age comparison all in the SQL. It can be done for 1 specific input argument, but needs the whole function repeated manually for a different input argument, and further repeated for a different argument etc.. Maybe I'm also compounding the issue by trying to make the policy file as concise as possible (for sake of clarity as to what the rules are trying to achieve, and easy expandability), and demanding too much of the SQL-like syntax that IBM have created. I have options for mmfind or even (as suggested) -M inoput to mmapplypolicy where I us bash to create a small function that does what I need, spits out 'real days' given a working day input, and using arrays and for-loop create a dynamic calling of the mmapplypolicy command (which I'm kinda half doing anyways for other reasons in my launcher script. As always, I'm seriously amazed at people with soo much experience and knowledge taking time out to help, guide, and offer input like everyone has been doing!! I'm relatively early in my career, so being able to interact and learn from experienced persons is giving me such a wider insight! Thanks! Owen. [Sent from Front] On 1 February 2021, 20:17 GMT jonathan.buzzard at strath.ac.uk wrote: On 01/02/2021 18:11, Jan-Frode Myklebust wrote: > CAUTION: This email originated outside the University. Check before > clicking links or attachments. > Agree.. Write a policy that takes a "mmapplypolicy -M var=val" argument, > and figure out the workdays outside of the policy. Something like: > > # cat test.poilcy > define( access_age, (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) > /* list migrated files */ > RULE EXTERNAL LIST 'oldFiles' EXEC '' > RULE 'oldFiles' LIST 'oldFiles' > WHERE (access_age > MINAGE) > > # mmapplypolicy gpfs01 -P test.policy -I defer -f ./filelist -M MINAGE=5 > Why bother when you can do it all in the policy? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG Owen Morgan Data Wrangler Motion Picture Solutions Ltd T: E: owen.morgan at motionpicturesolutions.com | W: motionpicturesolutions.com A: Mission Hall, 9-11 North End Road, London, W14 8ST Motion Picture Solutions Ltd is a company registered in England and Wales under number 5388229, VAT number 201330482 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Mon Feb 1 21:43:43 2021 From: jtucker at pixitmedia.com (Jez Tucker) Date: Mon, 1 Feb 2021 21:43:43 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: <02351047-3b02-352f-48bc-b4b3e0e03918@pixitmedia.com> Hi Owen, ? This is a great thread and raises as is usual with GPFS that there is more often than not many tools in the bag and more than one way to achieve a requirement. If you stick with policy (indeed worthwhile aspect of GPFS to learn well) you can also build up libraries of centralised macros and reference them into any policy file alike so at the top of the file: include(/mmfs1/policies/macros/general_macros.m4) include(/mmfs1/policies/macros/pixit_excludes.m4) Check your PixStor system under /mmfs1/policies/.. The .m4 extension is not required, but a nod to https://www.gnu.org/savannah-checkouts/gnu/m4/manual/m4-1.4.18/m4.html which the policy engine also encapsulates.? Marc Kaplan once told me "You can do just about anything with m4 and cleverness and patience...", though he failed to mention the required amounts of coffee and pizza ;-)? Search the list digests for his valuable insights over the years. Alternatively you can achieve all this and more using the Python API https://www.arcapix.com/pixstorapi/index.html (especially your date processing). If you'd like direct support with any this, ping us over an email via support@ - alternatively there are indeed many great minds a wealth of experience and views on this list (and it's nice to meet the community too). SSUG also has a Slack channel too.. ssug-poweraiug.slack.com Most of all - have fun learning. Kind regards, Jez p.s. yes the threading has all gone wonky. ah well :-) On 01/02/2021 21:09, Owen Morgan wrote: > Jonathan, > > If I have a single policy file with all the related department rules > and each time they want to add additional rules with different working > day thresholds maybe using this -M method is easier. Its clear that > the 'maths' and date/timestamp manipulation is easier in shell (my > preferred is bash) than in the SQL of the policy (your example is > succinct but needs to be repeated everytime a new rule is added with a > different working day threshold, which is what I'm trying (if > possiblr) to avoid. > > It seems to me the IBM SQL engine is perhaps missing more 'SQL' in > built date/time functions like DateAdd and DateDiff etc..? as this > would be a moot point. Its a shame I can't make one function that > given a working day input as an argument spits out how many 'real' > days exist between them for the file age comparison all in the SQL. It > can be done for 1 specific input argument, but needs the whole > function repeated manually for a different input argument, and further > repeated for a different argument etc.. > > Maybe I'm also compounding the issue by trying to make the policy file > as concise as possible (for sake of clarity as to what the rules are > trying to achieve, and easy expandability), and demanding too much of > the SQL-like syntax that IBM have created. > > I have options for mmfind or even (as suggested) -M inoput to > mmapplypolicy where I us bash to create a small function that does > what I need, spits out 'real days' given a working day input, and > using arrays and for-loop create a dynamic calling of the > mmapplypolicy command (which I'm kinda half doing anyways for other > reasons in my launcher script. > > > As always, I'm seriously amazed at people with soo much experience and > knowledge taking time out to help, guide, and offer input like > everyone has been doing!! I'm relatively early in my career, so being > able to interact and learn from experienced persons is giving me such > a wider insight! > > Thanks! > > Owen. > Sent from Front > ? ? > > Owen?Morgan? > Data?Wrangler > Motion?Picture?Solutions?Ltd > T:?** > > E:?*owen.morgan at motionpicturesolutions.com* > ?|? > W:?*motionpicturesolutions.com* > > A:? Mission?Hall,?9?11?North?End?Road ,? London ,? W14?8ST > > Motion Picture Solutions Ltd is a company registered in England and > Wales under number 5388229, VAT number 201330482 > >> On 1 February 2021, 20:17 GMT jonathan.buzzard at strath.ac.uk >> wrote: >> >> On 01/02/2021 18:11, Jan-Frode Myklebust wrote: >>> >>> > CAUTION: This email originated outside the University. Check before >>> > clicking links or attachments. >>> > Agree.. Write a policy that takes a "mmapplypolicy -M var=val" >>> argument, >>> > and figure out the workdays outside of the policy. Something like: >>> > >>> > # cat test.poilcy >>> > define( access_age, ? ? (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME))) >>> > /* list migrated files */ >>> > RULE EXTERNAL LIST 'oldFiles' EXEC '' >>> > RULE 'oldFiles' LIST 'oldFiles' >>> > ? ? WHERE (access_age > MINAGE) >>> > >>> > #?mmapplypolicy gpfs01 ?-P test.policy -I defer -f ./filelist -M >>> MINAGE=5 >>> > >>> >>> Why bother when you can do it all in the policy? >>> >>> JAB. >>> >>> -- >>> Jonathan A. Buzzard Tel: +44141-5483420 >>> HPC System Administrator, ARCHIE-WeSt. >>> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG >>> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- *Jez Tucker* VP Research and Development | Pixit Media m: +44 (0) 776 419 3820 e: jtucker at pixitmedia.com Visit www.pixitmedia.com -- This email is confidential in that it is?intended for the exclusive attention of?the addressee(s) indicated. If you are?not the intended recipient, this email?should not be read or disclosed to?any other person. Please notify the?sender immediately and delete this?email from your computer system.?Any opinions expressed are not?necessarily those of the company?from which this email was sent and,?whilst to the best of our knowledge no?viruses or defects exist, no?responsibility can be accepted for any?loss or damage arising from its?receipt or subsequent use of this? email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon Feb 1 21:58:34 2021 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 1 Feb 2021 21:58:34 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: References: Message-ID: <3e2d8556-72a3-930f-dc41-c15ffac1e495@strath.ac.uk> On 01/02/2021 21:09, Owen Morgan wrote: > CAUTION: This email originated outside the University. Check before > clicking links or attachments. > Jonathan, > > If I have a single policy file with all the related department rules and > each time they want to add additional rules with different working day > thresholds maybe using this -M method is easier. Its clear that the > 'maths' and date/timestamp manipulation is easier in shell (my preferred > is bash) than in the SQL of the policy (your example is succinct but > needs to be repeated everytime a new rule is added with a different > working day threshold, which is what I'm trying (if possiblr) to avoid. > I actually think there is a much better way to achieve it than the case statement, but that would have required me to do some more thinking and testing to make sure it worked ;-) Anyway how many different "working" days old do you need? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From anacreo at gmail.com Mon Feb 1 22:14:39 2021 From: anacreo at gmail.com (Alec) Date: Mon, 1 Feb 2021 14:14:39 -0800 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18 In-Reply-To: <3e2d8556-72a3-930f-dc41-c15ffac1e495@strath.ac.uk> References: <3e2d8556-72a3-930f-dc41-c15ffac1e495@strath.ac.uk> Message-ID: In our environment managing the GPFS policies (or anything GPFS) are a bit of an Art form and really only a few people out of a sea of engineers can/will do it. So I like to keep the GPFS policies simple and durable. If you really wanted to you could generate a list of every US (or whatever country) holiday you wanted and use Python, Awk, whatever to produce an exact number of business days versus not; You can also use directory structure, GPFS attributes in parent directory, or a config file to maintain the requirements, which may be more manageable by other team members. Anyhow my point being don't just look at the technical of how to implement, but what is the most maintainable, .documentable, and recoverable solution. Some things I would consider in my design? * Supportability - What is the skill level of the engineer required to maintain the system. * Serviceability - What component would change most frequently, what's the complexity and impact of a change. * Fail Safe - If a component fails, what will the negative impact be. * Trackability - Are all the activities logged. * Sanity Checking - Can you define safety mechanisms to prevent a failure. For instance if your purge script detects that there are 10k+ files to purge, can you interrupt the process? For GPFS we try to keep the service components simple, and the logic in shell scripting where there is a plethora of knowledgeable engineers. So my attack would be like this: * Policy runs, purges any file with purgeafter beyond the current date (maybe validate that the purge.date and modify/access date agree still). <-- is simple * I would have it run the logic twice, and bomb if a safety threshhold is exceeded. Should log the results. * Config in /gpfs/configs/purge_configuration /some/path 10bd /some/scratchspace 30d /some/reallybigfiles 1d * mmfind $(awk '{print $1}' < gpfs/configs/purge_configuration) -type f --exec /gpfs/scripts/setpurgedates * Could also be a regular mmapplypolicy command as well... * /gpfs/scripts/setpurgedates can rescan /gpfs/configs/purge_configuration and read in the date, check against the path passed, and then perform the necessary date logic, and exec the mmchattr, preserve the file's modify/access time to a purge.date time.. * Could log its decisions to a log file. Happy Hunting, I've already learned quite a few things from monitoring this list in a very short period of time. * If a candidate ever tells me they know everything about Unix... I pass. We're all very experienced noobies in this ever progressing field/platform. * Alec On Mon, Feb 1, 2021 at 1:58 PM Jonathan Buzzard < jonathan.buzzard at strath.ac.uk> wrote: > On 01/02/2021 21:09, Owen Morgan wrote: > > CAUTION: This email originated outside the University. Check before > > clicking links or attachments. > > Jonathan, > > > > If I have a single policy file with all the related department rules and > > each time they want to add additional rules with different working day > > thresholds maybe using this -M method is easier. Its clear that the > > 'maths' and date/timestamp manipulation is easier in shell (my preferred > > is bash) than in the SQL of the policy (your example is succinct but > > needs to be repeated everytime a new rule is added with a different > > working day threshold, which is what I'm trying (if possiblr) to avoid. > > > > I actually think there is a much better way to achieve it than the case > statement, but that would have required me to do some more thinking and > testing to make sure it worked ;-) > > Anyway how many different "working" days old do you need? > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Walter.Sklenka at EDV-Design.at Tue Feb 2 13:10:05 2021 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Tue, 2 Feb 2021 13:10:05 +0000 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> Message-ID: <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> Hi Giovanni! Thank you for your offer! ? it is planned to be implemented in June or so We will use RHEL 8.x and newest gpfs version available Only one question for this moment if I am allowed: Did you ever ran into any problems with IBM support? I mean they say in the FAQ shortly "not supported" , but do they in your environment or do you accept that rdma problems would be needed to be fixed without IBM Thank you very much and have great days! And keep healthy! Best regards walter -----Original Message----- From: Giovanni Bracco Sent: Montag, 1. Februar 2021 20:42 To: Walter Sklenka Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled On 30/01/21 21:01, Walter Sklenka wrote: > Hi Giovanni! > Thats great! Many thanks for your fast and detailed answer!!!! > So this is the way we will go too! > > Have a nice weekend and keep healthy! > Best regards > Walter > I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works! If yu have any issues I am ready to discuss! Regards Giovanni > -----Original Message----- > From: Giovanni Bracco > > Sent: Samstag, 30. J?nner 2021 18:08 > To: gpfsug main discussion list >; > Walter Sklenka > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > See > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a > multifabric GPFS Spectrum Scale layout," 2019 International Conference > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > When setting up the system the main trick has been: > just use CentOS drivers and do not install OFED We do not use IPoIB. > > Giovanni > > On 30/01/21 06:45, Walter Sklenka wrote: >> Hi! >> >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? >> >> In the faq >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. >> html#rdma >> >> >> They talk about RDMA : >> >> "RDMA is NOT supported on a node when both Mellanox HCAs and Intel >> Omni-Path HFIs are ENABLED for RDMA." >> >> So do I understand right: When we do NOT enable the opa interface we >> can still enable IB ? >> >> The reason I ask is, that we have a gpfs cluster of 6 NSD Servers >> (wih access to storage) with opa interfaces which provide access to >> remote cluster also via OPA. >> >> A new cluster with HDR interfaces will be implemented soon >> >> They shell have access to the same filesystems >> >> When we add HDR interfaces to NSD servers and enable rdma on this >> network while disabling rdma on opa we would accept the worse >> performance via opa . We hope that this provides still better perf >> and less technical overhead than using routers >> >> Or am I totally wrong? >> >> Thank you very much and keep healthy! >> >> Best regards >> >> Walter >> >> Mit freundlichen Gr??en >> */Walter Sklenka/* >> */Technical Consultant/* >> >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 >> Wien >> Tel: +43 1 29 22 165-31 >> Fax: +43 1 29 22 165-90 >> E-Mail: sklenka at edv-design.at >> Internet: www.edv-design.at >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Giovanni Bracco > phone +39 351 8804788 > E-mail giovanni.bracco at enea.it > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco -------------- next part -------------- An HTML attachment was scrubbed... URL: From Walter.Sklenka at EDV-Design.at Tue Feb 2 13:19:37 2021 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Tue, 2 Feb 2021 13:19:37 +0000 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> Message-ID: Hi Giovanni! Thank you very much for your offer , we really would be very grateful to be allowed to come if we run into troubles! Well, the implementation will not happen before June or later, but may I ask only one question meanwhile? Did you ever run into problems with IBM support or did you get a special ?OK? from them? Or do you accept to sove any rdma specific problems without support ? (it?s only because of the FAQ ?not supported? ) Have a great day and keep healthy! Best regards walter -----Original Message----- From: Giovanni Bracco Sent: Montag, 1. Februar 2021 20:42 To: Walter Sklenka Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled On 30/01/21 21:01, Walter Sklenka wrote: > Hi Giovanni! > Thats great! Many thanks for your fast and detailed answer!!!! > So this is the way we will go too! > > Have a nice weekend and keep healthy! > Best regards > Walter > I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works! If yu have any issues I am ready to discuss! Regards Giovanni > -----Original Message----- > From: Giovanni Bracco > > Sent: Samstag, 30. J?nner 2021 18:08 > To: gpfsug main discussion list >; > Walter Sklenka > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > See > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a > multifabric GPFS Spectrum Scale layout," 2019 International Conference > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > When setting up the system the main trick has been: > just use CentOS drivers and do not install OFED We do not use IPoIB. > > Giovanni > > On 30/01/21 06:45, Walter Sklenka wrote: >> Hi! >> >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? >> >> In the faq >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. >> html#rdma >> >> >> They talk about RDMA : >> >> "RDMA is NOT supported on a node when both Mellanox HCAs and Intel >> Omni-Path HFIs are ENABLED for RDMA." >> >> So do I understand right: When we do NOT enable the opa interface we >> can still enable IB ? >> >> The reason I ask is, that we have a gpfs cluster of 6 NSD Servers >> (wih access to storage) with opa interfaces which provide access to >> remote cluster also via OPA. >> >> A new cluster with HDR interfaces will be implemented soon >> >> They shell have access to the same filesystems >> >> When we add HDR interfaces to NSD servers and enable rdma on this >> network while disabling rdma on opa we would accept the worse >> performance via opa . We hope that this provides still better perf >> and less technical overhead than using routers >> >> Or am I totally wrong? >> >> Thank you very much and keep healthy! >> >> Best regards >> >> Walter >> >> Mit freundlichen Gr??en >> */Walter Sklenka/* >> */Technical Consultant/* >> >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 >> Wien >> Tel: +43 1 29 22 165-31 >> Fax: +43 1 29 22 165-90 >> E-Mail: sklenka at edv-design.at >> Internet: www.edv-design.at >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Giovanni Bracco > phone +39 351 8804788 > E-mail giovanni.bracco at enea.it > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Tue Feb 2 16:08:47 2021 From: ewahl at osc.edu (Wahl, Edward) Date: Tue, 2 Feb 2021 16:08:47 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180202152336.03e8bab7@osc.edu> References: <20180119163803.79fddbeb@osc.edu> , <20180202152336.03e8bab7@osc.edu> Message-ID: Replying to a 3 year old message I sent, hoping that in the last couple of years that Scale has added some ILM extensions into the policy engine that I have missed, or somehow didn't notice? Just ran into a file with an 'unbalanced' flag and I REALLY don't want to have to mmlsattr everything. AGAIN. /facepalm IBM? Bueller? Bueller? When everyone answers: "No", I'm guessing this needs to be a request for improvement/enhancement? Ed Wahl Ohio Supercomputer Center ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Edward Wahl Sent: Friday, February 2, 2018 3:23 PM To: John Hearns Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] policy ilm features? Thanks John, this was the path I was HOPING to go down as I do similar things already, but there appears to be no extended attribute in ILM for what I want. Data block replication flag exists in the ILM, but not MetaData, or balance. Yet these states ARE reported by mmlsattr, so there must be a flag somewhere. bad MD replication & balance example: mmlsattr -L /fs/scratch/sysp/ed/180days.pol file name: /fs/scratch/sysp/ed/180days.pol metadata replication: 1 max 2 data replication: 1 max 2 flags: illreplicated,unbalanced Encrypted: yes File next to it for comparison. note proper MD replication and balance. mmlsattr -L /fs/scratch/sysp/ed/120days.pol file name: /fs/scratch/sysp/ed/120days.pol metadata replication: 2 max 2 data replication: 1 max 2 flags: Encrypted: yes misc_attributes flags from a policy run showing no difference in status: FJAEu -- /fs/scratch/sysp/ed/180days.pol FJAEu -- /fs/scratch/sysp/ed/120days.pol File system has MD replication enabled, but not Data, so ALL files show "J" ilm flag mmlsfs scratch -m flag value description ------------------- ------------------------ ----------------------------------- -m 2 Default number of metadata replicas mmlsfs scratch -r flag value description ------------------- ------------------------ ----------------------------------- -r 1 Default number of data replicas I poked around a little trying to find out if perhaps using GetXattr would work and show me what I wanted, it does not. All I sem to be able to get is the File Encryption Key. I was hoping perhaps someone had found a cheaper way for this to work rather than hundreds of millions of 'mmlsattr' execs. :-( On the plus side, I've only run across a few of these and all appear to be from before we did the MD replication and re-striping. On the minus, I have NO idea where they are, and they appears to be on both of our filesystems. So several hundred million files to check. Ed On Mon, 22 Jan 2018 08:29:42 +0000 John Hearns wrote: > Ed, > This is not a perfect answer. You need to look at policies for this. I have > been doing something similar recently. > > Something like: > > RULE 'list_file' EXTERNAL LIST 'all-files' EXEC > '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files' > SHOW( varchar(kb_allocated) || ' ' || varchar(file_size) || ' ' || > varchar(misc_attributes) || ' ' || name || ' ' || fileset_name ) WHERE > REGEX(misc_attributes,'[J]') > > > So this policy shows the kbytes allocates, file size, the miscellaneous > attributes, name and fileset name For all files with miscellaneous > attributes of 'J' which means 'Some data blocks might be ill replicated' > > > > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Edward Wahl > Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] policy ilm features? > > > This one has been on my list a long time so I figured I'd ask here first > before I open an apar or request an enhancement (most likely). > > Is there a way using the policy engine to determine the following? > > -metadata replication total/current > -unbalanced file > > Looking to catch things like this that stand out on my filesystem without > having to run several hundred million 'mmlsattr's. > > metadata replication: 1 max 2 > flags: unbalanced > > Ed > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D&reserved=0 > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is provided > on an AS-IS basis without any express or implied warranties or liabilities. > To the extent you are relying on this information, you are doing so at your > own risk. If you are not the intended recipient, please notify the sender > immediately by replying to this message and destroy all copies of this > message and any attachments. Neither the sender nor the company/group of > companies he or she represents shall be liable for the proper and complete > transmission of the information contained in this communication, or for any > delay in its receipt. _______________________________________________ > gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Ed Wahl Ohio Supercomputer Center 614-292-9302 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Tue Feb 2 18:09:28 2021 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 2 Feb 2021 18:09:28 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: References: , <20180119163803.79fddbeb@osc.edu>, <20180202152336.03e8bab7@osc.edu> Message-ID: An HTML attachment was scrubbed... URL: From ewahl at osc.edu Tue Feb 2 21:26:42 2021 From: ewahl at osc.edu (Wahl, Edward) Date: Tue, 2 Feb 2021 21:26:42 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: References: , <20180119163803.79fddbeb@osc.edu>, <20180202152336.03e8bab7@osc.edu>, Message-ID: My issues were never specifically the Unbalanced flag, originally I ran into ilm/policy issues with our metadata and illreplicated files. We were working to expand our Metadata storage at the time uner 4.2.3.xx and had added a number of SSDs to the array. But AFTER a restripe files were still illreplicated. I then discovered that the policy engine had no way to tell me what files were and were not replicated. Just hoping that more ILM info had been added to the policy engine since I ran into this, and seeing the unbalanced files jogged my memory to look at the docs again, where I didn't see anything in 5.x and ask the list as well. This isn't a big deal, more curiousity on my part. I see you have some of the original thread attached, so perhaps take a glance and see if it makes sense? I Now if you REALLY want to step into a mine field, go find my thread on SKLM usage and GPFS/SS. Every single respondent to my question about filling up their SKLM logs with errors was a positive. And SKLM L2 and L3 support swears GPFS/SS is using SKLM wrong... Ed Wahl OSC ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederick Stock Sent: Tuesday, February 2, 2021 1:09 PM To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] policy ilm features? Hello Ed. Jordan contacted me about the question you are posing so I am responding to your message. Could you please provide clarification as to why the existence of the unbalanced flag is of a concern, or why you would want to know all the files that have this flag set? The flag would be cleared once the file was rebalanced either through normal access or through the execution of the mmrestripefs/mmrestripefile commands. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com ----- Original message ----- From: "Wahl, Edward" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] policy ilm features? Date: Tue, Feb 2, 2021 11:52 AM Replying to a 3 year old message I sent, hoping that in the last couple of years that Scale has added some ILM extensions into the policy engine that I have missed, or somehow didn't notice? Just ran into a file with an 'unbalanced' flag and I REALLY don't want to have to mmlsattr everything. AGAIN. /facepalm IBM? Bueller? Bueller? When everyone answers: "No", I'm guessing this needs to be a request for improvement/enhancement? Ed Wahl Ohio Supercomputer Center ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Edward Wahl Sent: Friday, February 2, 2018 3:23 PM To: John Hearns Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] policy ilm features? Thanks John, this was the path I was HOPING to go down as I do similar things already, but there appears to be no extended attribute in ILM for what I want. Data block replication flag exists in the ILM, but not MetaData, or balance. Yet these states ARE reported by mmlsattr, so there must be a flag somewhere. bad MD replication & balance example: mmlsattr -L /fs/scratch/sysp/ed/180days.pol file name: /fs/scratch/sysp/ed/180days.pol metadata replication: 1 max 2 data replication: 1 max 2 flags: illreplicated,unbalanced Encrypted: yes File next to it for comparison. note proper MD replication and balance. mmlsattr -L /fs/scratch/sysp/ed/120days.pol file name: /fs/scratch/sysp/ed/120days.pol metadata replication: 2 max 2 data replication: 1 max 2 flags: Encrypted: yes misc_attributes flags from a policy run showing no difference in status: FJAEu -- /fs/scratch/sysp/ed/180days.pol FJAEu -- /fs/scratch/sysp/ed/120days.pol File system has MD replication enabled, but not Data, so ALL files show "J" ilm flag mmlsfs scratch -m flag value description ------------------- ------------------------ ----------------------------------- -m 2 Default number of metadata replicas mmlsfs scratch -r flag value description ------------------- ------------------------ ----------------------------------- -r 1 Default number of data replicas I poked around a little trying to find out if perhaps using GetXattr would work and show me what I wanted, it does not. All I sem to be able to get is the File Encryption Key. I was hoping perhaps someone had found a cheaper way for this to work rather than hundreds of millions of 'mmlsattr' execs. :-( On the plus side, I've only run across a few of these and all appear to be from before we did the MD replication and re-striping. On the minus, I have NO idea where they are, and they appears to be on both of our filesystems. So several hundred million files to check. Ed On Mon, 22 Jan 2018 08:29:42 +0000 John Hearns wrote: > Ed, > This is not a perfect answer. You need to look at policies for this. I have > been doing something similar recently. > > Something like: > > RULE 'list_file' EXTERNAL LIST 'all-files' EXEC > '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files' > SHOW( varchar(kb_allocated) || ' ' || varchar(file_size) || ' ' || > varchar(misc_attributes) || ' ' || name || ' ' || fileset_name ) WHERE > REGEX(misc_attributes,'[J]') > > > So this policy shows the kbytes allocates, file size, the miscellaneous > attributes, name and fileset name For all files with miscellaneous > attributes of 'J' which means 'Some data blocks might be ill replicated' > > > > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Edward Wahl > Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] policy ilm features? > > > This one has been on my list a long time so I figured I'd ask here first > before I open an apar or request an enhancement (most likely). > > Is there a way using the policy engine to determine the following? > > -metadata replication total/current > -unbalanced file > > Looking to catch things like this that stand out on my filesystem without > having to run several hundred million 'mmlsattr's. > > metadata replication: 1 max 2 > flags: unbalanced > > Ed > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D&reserved=0 > -- The information contained in this communication and any attachments is > confidential and may be privileged, and is for the sole use of the intended > recipient(s). Any unauthorized review, use, disclosure or distribution is > prohibited. Unless explicitly stated otherwise in the body of this > communication or the attachment thereto (if any), the information is provided > on an AS-IS basis without any express or implied warranties or liabilities. > To the extent you are relying on this information, you are doing so at your > own risk. If you are not the intended recipient, please notify the sender > immediately by replying to this message and destroy all copies of this > message and any attachments. Neither the sender nor the company/group of > companies he or she represents shall be liable for the proper and complete > transmission of the information contained in this communication, or for any > delay in its receipt. _______________________________________________ > gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Ed Wahl Ohio Supercomputer Center 614-292-9302 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From giovanni.bracco at enea.it Wed Feb 3 08:58:37 2021 From: giovanni.bracco at enea.it (Giovanni Bracco) Date: Wed, 3 Feb 2021 09:58:37 +0100 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> Message-ID: <0ecee41c-9a9b-83d4-59ae-2b0fbda526fb@enea.it> We did not explore the issue of the IBM support and for budget limitation and for the mandatory integration of the data space between the two clusters, we decided to try the setup of the multi-fabric infrastructure and up to now it has been working without problems. Giovanni On 02/02/21 14:10, Walter Sklenka wrote: > Hi Giovanni! > > Thank you for your offer! ? > > it is planned to be implemented in June or so > > We will use RHEL 8.x and newest gpfs version available > > Only one question for this moment if I am allowed: > > Did you ever ran into any problems with IBM support? I mean they say in > the FAQ shortly "not supported" , but do they in your environment or do > you accept that rdma problems would be needed to be fixed without IBM > > Thank you very much and have great days! And keep healthy! > > Best regards walter > > -----Original Message----- > From: Giovanni Bracco > Sent: Montag, 1. Februar 2021 20:42 > To: Walter Sklenka > Cc: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > On 30/01/21 21:01, Walter Sklenka wrote: > > > Hi Giovanni! > > > Thats great! Many thanks for your fast and detailed answer!!!! > > > So this is the way we will go too! > > > > > > Have a nice weekend and keep healthy! > > > Best regards > > > Walter > > > > > I suppose you will implement the solution with more recent versions of > the software components, so please let me know if everything works! > > If yu have any issues I am ready to discuss! > > Regards > > Giovanni > > > -----Original Message----- > > > From: Giovanni Bracco > > > > Sent: Samstag, 30. J?nner 2021 18:08 > > > To: gpfsug main discussion list >; > > > Walter Sklenka > > > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > > > Server with only ib rdma enabled > > > > > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, > each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes > SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA > Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN > storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > > > See > > > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a > > > multifabric GPFS Spectrum Scale layout," 2019 International Conference > > > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > > > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > > > > > When setting up the system the main trick has been: > > > just use CentOS drivers and do not install OFED We do not use IPoIB. > > > > > > Giovanni > > > > > > On 30/01/21 06:45, Walter Sklenka wrote: > > >> Hi! > > >> > > >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? > > >> > > >> In the faq > > >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. > > >> html#rdma > > >> > > >> > > >> They talk about RDMA : > > >> > > >> "RDMA is NOT ?supported on a node when both Mellanox HCAs and Intel > > >> Omni-Path HFIs are ENABLED for RDMA." > > >> > > >> So do I understand right: When we do NOT enable ?the opa interface we > > >> can still enable IB ? > > >> > > >> The reason I ask ?is, that we have a gpfs cluster of 6 NSD Servers > > >> (wih access to storage) ?with opa interfaces which provide access to > > >> remote cluster ?also via OPA. > > >> > > >> A new cluster with HDR interfaces will be implemented soon > > >> > > >> They shell have access to the same filesystems > > >> > > >> When we add HDR interfaces to? NSD servers? and enable rdma on this > > >> network ?while disabling rdma on opa we would accept the worse > > >> performance via opa . We hope that this provides ?still better perf > > >> and less technical overhead ?than using routers > > >> > > >> Or am I totally wrong? > > >> > > >> Thank you very much and keep healthy! > > >> > > >> Best regards > > >> > > >> Walter > > >> > > >> Mit freundlichen Gr??en > > >> */Walter Sklenka/* > > >> */Technical Consultant/* > > >> > > >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 > > >> Wien > > >> Tel: +43 1 29 22 165-31 > > >> Fax: +43 1 29 22 165-90 > > >> E-Mail: sklenka at edv-design.at > > > >> Internet: www.edv-design.at > > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > -- > > > Giovanni Bracco > > > phone? +39 351 8804788 > > > E-mail giovanni.bracco at enea.it > > > WWW http://www.afs.enea.it/bracco > > > > > -- > > Giovanni Bracco > > phone? +39 351 8804788 > > E-mail giovanni.bracco at enea.it > > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco From Walter.Sklenka at EDV-Design.at Wed Feb 3 15:21:18 2021 From: Walter.Sklenka at EDV-Design.at (Walter Sklenka) Date: Wed, 3 Feb 2021 15:21:18 +0000 Subject: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In-Reply-To: <0ecee41c-9a9b-83d4-59ae-2b0fbda526fb@enea.it> References: <14218088180e4613847984c44e0321d8@Mail.EDVDesign.cloudia> <3bb0f4ca-f6ee-6013-45a0-e783470089f0@enea.it> <871067b8-3cd0-f8dd-aecd-8f599b1d4986@enea.it> <5c9860aa1a9244b191c279432367d257@Mail.EDVDesign.cloudia> <0ecee41c-9a9b-83d4-59ae-2b0fbda526fb@enea.it> Message-ID: Hi Givanni ! I understand and am convinced that the is an excellent solution !! Thank you very much! -----Original Message----- From: Giovanni Bracco Sent: Mittwoch, 3. Februar 2021 09:59 To: Walter Sklenka Cc: gpfsug main discussion list Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled We did not explore the issue of the IBM support and for budget limitation and for the mandatory integration of the data space between the two clusters, we decided to try the setup of the multi-fabric infrastructure and up to now it has been working without problems. Giovanni On 02/02/21 14:10, Walter Sklenka wrote: > Hi Giovanni! > > Thank you for your offer! ? > > it is planned to be implemented in June or so > > We will use RHEL 8.x and newest gpfs version available > > Only one question for this moment if I am allowed: > > Did you ever ran into any problems with IBM support? I mean they say > in the FAQ shortly "not supported" , but do they in your environment > or do you accept that rdma problems would be needed to be fixed > without IBM > > Thank you very much and have great days! And keep healthy! > > Best regards walter > > -----Original Message----- > From: Giovanni Bracco > Sent: Montag, 1. Februar 2021 20:42 > To: Walter Sklenka > Cc: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > On 30/01/21 21:01, Walter Sklenka wrote: > > > Hi Giovanni! > > > Thats great! Many thanks for your fast and detailed answer!!!! > > > So this is the way we will go too! > > > > > > Have a nice weekend and keep healthy! > > > Best regards > > > Walter > > > > > I suppose you will implement the solution with more recent versions of > the software components, so please let me know if everything works! > > If yu have any issues I am ready to discuss! > > Regards > > Giovanni > > > -----Original Message----- > > > From: Giovanni Bracco > > > > Sent: Samstag, 30. J?nner 2021 18:08 > > > To: gpfsug main discussion list >; > > > Walter Sklenka > > > > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > > > Server with only ib rdma enabled > > > > > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, > each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes > SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main > OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to > DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. > > > See > > > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of > a > > > multifabric GPFS Spectrum Scale layout," 2019 International > Conference > > > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > > > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > > > > > When setting up the system the main trick has been: > > > just use CentOS drivers and do not install OFED We do not use IPoIB. > > > > > > Giovanni > > > > > > On 30/01/21 06:45, Walter Sklenka wrote: > > >> Hi! > > >> > > >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? > > >> > > >> In the faq > > >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. > > >> html#rdma > > >> > > >> > > >> They talk about RDMA : > > >> > > >> "RDMA is NOT ?supported on a node when both Mellanox HCAs and > Intel > > >> Omni-Path HFIs are ENABLED for RDMA." > > >> > > >> So do I understand right: When we do NOT enable ?the opa interface > we > > >> can still enable IB ? > > >> > > >> The reason I ask ?is, that we have a gpfs cluster of 6 NSD Servers > > >> (wih access to storage) ?with opa interfaces which provide access > to > > >> remote cluster ?also via OPA. > > >> > > >> A new cluster with HDR interfaces will be implemented soon > > >> > > >> They shell have access to the same filesystems > > >> > > >> When we add HDR interfaces to? NSD servers? and enable rdma on > this > > >> network ?while disabling rdma on opa we would accept the worse > > >> performance via opa . We hope that this provides ?still better > perf > > >> and less technical overhead ?than using routers > > >> > > >> Or am I totally wrong? > > >> > > >> Thank you very much and keep healthy! > > >> > > >> Best regards > > >> > > >> Walter > > >> > > >> Mit freundlichen Gr??en > > >> */Walter Sklenka/* > > >> */Technical Consultant/* > > >> > > >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 > > >> Wien > > >> Tel: +43 1 29 22 165-31 > > >> Fax: +43 1 29 22 165-90 > > >> E-Mail: sklenka at edv-design.at > > > >> Internet: www.edv-design.at > > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > >> > > > > > > -- > > > Giovanni Bracco > > > phone? +39 351 8804788 > > > E-mail giovanni.bracco at enea.it > > > WWW http://www.afs.enea.it/bracco > > > > > -- > > Giovanni Bracco > > phone? +39 351 8804788 > > E-mail giovanni.bracco at enea.it > > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bracco at enea.it WWW http://www.afs.enea.it/bracco From henrik at morsing.cc Wed Feb 3 16:18:44 2021 From: henrik at morsing.cc (Henrik Morsing) Date: Wed, 3 Feb 2021 16:18:44 +0000 Subject: [gpfsug-discuss] Intro Message-ID: <20210203161844.GO14457@morsing.cc> Hi all, I live in Buckinghamshire in the UK, but work in portsmouth for the energy company SSE. Some years ago, we bought in two 840TB Spectrum Scale systems to replace our tape libraries. I was put on the project to get it up and running and the tape library phased out, but the project has stalled multiple times. I am back on the project now, every time it feels like I am starting over, and on top I need to patch it again which was a massive undertaking last time, so I am not looking forward to that. The clusters are connected via a six link 10Gb/s etherchannel to our Spectrum Protect backup servers. Two of these have been setup, including cross-site replication job, so we do have some idea of it working, but performance monitoring is another aspect I need to look at. I am planning to install njmon+Grafana to see how that looks. Hope to get lots of tips from this forum! Regards, Henrik Morsing From jonathan.b.mills at nasa.gov Mon Feb 8 16:00:20 2021 From: jonathan.b.mills at nasa.gov (Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC]) Date: Mon, 8 Feb 2021 16:00:20 +0000 Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Message-ID: Dear Community, We have multiple GPFS clusters within our internal networks. We want to expose more than one GPFS GUI interface, from behind a shared web proxy / firewall setup, which has SAML2 integration with our organization's IDP. (The SAML2 is not part of the question, it's just a rationale for doing it this way.) Essentially, we're trying to do something like https:///gpfs/gui1 and https:///gpfs/gui2 The websphere config in the GPFS GUI packages basically assume a document root of "/". This poses additional problems when it forms relative URLs...for example, immediately after login, it tries to redirect from "/" to "/gui". Through our web proxy, this ends up being https:///gui which is a 404. While problems might be addressable with some complex mod_rewrite in our web proxy, I've been looking for a more straightforward solution. In /opt/ibm/wlp/usr/servers/gpfsgui/server.xml, it's possible to change the document root in line 34: Changing the document root would allow me to mod_proxy from https:///gpfs/gui1 to https:///gpfs/gui1, and in theory that makes it easy to support many GPFS GUIs, and should fix the relative URL / redirect issue. However, while it seems like this ought to simplify matters, in fact it just breaks other parts of the GPFS GUI, because it calls URLs that aren't part of "ROOT.war" and those also assume a document root of "/". So, my basic question for the community is this: Has anyone out there successfully mucked with the GPFS GUI document root? Or am I just barking up the wrong tree? Thanks in advance, Jonathan -- Jonathan Mills / jonathan.mills at nasa.gov NASA GSFC / NCCS HPC (606.2) Bldg 28, Rm. S230 / c. 252-412-5710 From heinrich.billich at id.ethz.ch Mon Feb 8 21:27:27 2021 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Mon, 8 Feb 2021 21:27:27 +0000 Subject: [gpfsug-discuss] Mmapplypolicy with -I defer doesn't sort resulting list? Is this by intention? Message-ID: <9B403CA7-7785-480E-ACC1-38561F0797CD@id.ethz.ch> Hello, I want to migrate data with mmapplypolicy to a different pool: 1. create a file list with "-I defer -f /some/path" 2. execute with "-I yes -r /some/path" I noted that the file list created in 1. Is not sorted. I asked to sort by kb_allocated, the ideas is to migrate the largest files first and to stop once we freed enough space. Is this intended, or should the file list created in 1. be sorted? I want to be able to review the list before I start the migration. The list holds size/kb_allocated in hex numbers, I didn't manage to sort it with 'sort'. Hence I would prefer to get a sorted list from mmapplypolicy instead of writing some code to do this. The list holds files with sizes between 32K and 250G and 4.5M lines, hence it makes sense to sort to move the largest files on the top. Thank you and best regards, Heiner The policy is RULE 'migrate' MIGRATE FROM POOL 'data' THRESHOLD (50,50) WEIGHT (kb_allocated) TO POOL 'capacity' LIMIT (98) WHERE kb_allocated > 0 AND (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) > 90 And the command line mmapplypolicy $1 -A 600 \ -a 10 \ -P $base/policy.txt \ -I prepare \ -f $base/candidates \ -L 1 \ -N some_nodes \ --choice-algorithm fast \ --max-sort-bytes 1G \ --split-margin 2.0 The filesystem holds about 1.2G inodes. I tried to speed-up sorting with the last three arguments. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5254 bytes Desc: not available URL: From jon at well.ox.ac.uk Tue Feb 9 09:13:55 2021 From: jon at well.ox.ac.uk (Jon Diprose) Date: Tue, 9 Feb 2021 09:13:55 +0000 Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml In-Reply-To: References: Message-ID: Might it be easier to add gpfs1. and gpfs2. as CNAMEs to your web proxy / firewall , and then configure the web proxy to select the backend on the basis of the requested hostname? https://gpfs1./ redirects to https://gpfs1./gui/ and there's no mucking about with gpfs stuff required at all. Jon -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC] Sent: 08 February 2021 16:00 To: gpfsug main discussion list Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Dear Community, We have multiple GPFS clusters within our internal networks. We want to expose more than one GPFS GUI interface, from behind a shared web proxy / firewall setup, which has SAML2 integration with our organization's IDP. (The SAML2 is not part of the question, it's just a rationale for doing it this way.) Essentially, we're trying to do something like https:///gpfs/gui1 and https:///gpfs/gui2 The websphere config in the GPFS GUI packages basically assume a document root of "/". This poses additional problems when it forms relative URLs...for example, immediately after login, it tries to redirect from "/" to "/gui". Through our web proxy, this ends up being https:///gui which is a 404. While problems might be addressable with some complex mod_rewrite in our web proxy, I've been looking for a more straightforward solution. In /opt/ibm/wlp/usr/servers/gpfsgui/server.xml, it's possible to change the document root in line 34: Changing the document root would allow me to mod_proxy from https:///gpfs/gui1 to https:///gpfs/gui1, and in theory that makes it easy to support many GPFS GUIs, and should fix the relative URL / redirect issue. However, while it seems like this ought to simplify matters, in fact it just breaks other parts of the GPFS GUI, because it calls URLs that aren't part of "ROOT.war" and those also assume a document root of "/". So, my basic question for the community is this: Has anyone out there successfully mucked with the GPFS GUI document root? Or am I just barking up the wrong tree? Thanks in advance, Jonathan -- Jonathan Mills / jonathan.mills at nasa.gov NASA GSFC / NCCS HPC (606.2) Bldg 28, Rm. S230 / c. 252-412-5710 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From scale at us.ibm.com Tue Feb 9 10:55:45 2021 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 9 Feb 2021 16:25:45 +0530 Subject: [gpfsug-discuss] Mmapplypolicy with -I defer doesn't sort resulting list? Is this by intention? In-Reply-To: <9B403CA7-7785-480E-ACC1-38561F0797CD@id.ethz.ch> References: <9B403CA7-7785-480E-ACC1-38561F0797CD@id.ethz.ch> Message-ID: Hi Billich, I think the problem is that you are specifying --choice-algorithm fast and as per documentation "The fast choice method does not completely sort the candidates by weight." To sort the list you can try specifying --choice-algorithm exact which is also the default. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 09-02-2021 03.13 AM Subject: [EXTERNAL] [gpfsug-discuss] Mmapplypolicy with -I defer doesn't sort resulting list? Is this by intention? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I want to migrate data with mmapplypolicy to a different pool: 1. create a file list with "-I defer -f /some/path" 2. execute with "-I yes -r /some/path" I noted that the file list created in 1. Is not sorted. I asked to sort by kb_allocated, the ideas is to migrate the largest files first and to stop once we freed enough space. Is this intended, or should the file list created in 1. be sorted? I want to be able to review the list before I start the migration. The list holds size/kb_allocated in hex numbers, I didn't manage to sort it with 'sort'. Hence I would prefer to get a sorted list from mmapplypolicy instead of writing some code to do this. The list holds files with sizes between 32K and 250G and 4.5M lines, hence it makes sense to sort to move the largest files on the top. Thank you and best regards, Heiner The policy is RULE 'migrate' MIGRATE FROM POOL 'data' THRESHOLD (50,50) WEIGHT (kb_allocated) TO POOL 'capacity' LIMIT (98) WHERE kb_allocated > 0 AND (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) > 90 And the command line mmapplypolicy $1 -A 600 \ -a 10 \ -P $base/policy.txt \ -I prepare \ -f $base/candidates \ -L 1 \ -N some_nodes \ --choice-algorithm fast \ --max-sort-bytes 1G \ --split-margin 2.0 The filesystem holds about 1.2G inodes. I tried to speed-up sorting with the last three arguments. -- ======================= Heinrich Billich ETH Z?rich Informatikdienste Tel.: +41 44 632 72 56 heinrich.billich at id.ethz.ch ======================== [attachment "smime.p7s" deleted by Huzefa H Pancha/India/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From jonathan.b.mills at nasa.gov Tue Feb 9 15:07:21 2021 From: jonathan.b.mills at nasa.gov (Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC]) Date: Tue, 9 Feb 2021 15:07:21 +0000 Subject: [gpfsug-discuss] [EXTERNAL] Re: [gpfs][gui] Change document root in server.xml In-Reply-To: References: , Message-ID: Oh, without a doubt, and we considered it. However, the agency where I work makes it rather difficult to register new DNS names, and they actively discourage standing up new websites. But yes, I agree that is the simplest solution, and we may be forced to attempt it and wade through the red tape. Thank you for your response. Jonathan ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Jon Diprose Sent: Tuesday, February 9, 2021 4:13 AM To: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Might it be easier to add gpfs1. and gpfs2. as CNAMEs to your web proxy / firewall , and then configure the web proxy to select the backend on the basis of the requested hostname? https://gpfs1./ redirects to https://gpfs1./gui/ and there's no mucking about with gpfs stuff required at all. Jon -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Mills, Jonathan B. (GSFC-606.2)[InuTeq, LLC] Sent: 08 February 2021 16:00 To: gpfsug main discussion list Subject: [gpfsug-discuss] [gpfs][gui] Change document root in server.xml Dear Community, We have multiple GPFS clusters within our internal networks. We want to expose more than one GPFS GUI interface, from behind a shared web proxy / firewall setup, which has SAML2 integration with our organization's IDP. (The SAML2 is not part of the question, it's just a rationale for doing it this way.) Essentially, we're trying to do something like https:///gpfs/gui1 and https:///gpfs/gui2 The websphere config in the GPFS GUI packages basically assume a document root of "/". This poses additional problems when it forms relative URLs...for example, immediately after login, it tries to redirect from "/" to "/gui". Through our web proxy, this ends up being https:///gui which is a 404. While problems might be addressable with some complex mod_rewrite in our web proxy, I've been looking for a more straightforward solution. In /opt/ibm/wlp/usr/servers/gpfsgui/server.xml, it's possible to change the document root in line 34: Changing the document root would allow me to mod_proxy from https:///gpfs/gui1 to https:///gpfs/gui1, and in theory that makes it easy to support many GPFS GUIs, and should fix the relative URL / redirect issue. However, while it seems like this ought to simplify matters, in fact it just breaks other parts of the GPFS GUI, because it calls URLs that aren't part of "ROOT.war" and those also assume a document root of "/". So, my basic question for the community is this: Has anyone out there successfully mucked with the GPFS GUI document root? Or am I just barking up the wrong tree? Thanks in advance, Jonathan -- Jonathan Mills / jonathan.mills at nasa.gov NASA GSFC / NCCS HPC (606.2) Bldg 28, Rm. S230 / c. 252-412-5710 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Cjonathan.b.mills%40nasa.gov%7Cb4b847626cd347c347f408d8ccdb21f2%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637484588838656778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=d9r02MnZDlXpfiMTY5dxNIR0FGBORNKMiHUh8MS5NRs%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Cjonathan.b.mills%40nasa.gov%7Cb4b847626cd347c347f408d8ccdb21f2%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637484588838656778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=d9r02MnZDlXpfiMTY5dxNIR0FGBORNKMiHUh8MS5NRs%3D&reserved=0 From helge.hauglin at usit.uio.no Tue Feb 9 15:46:39 2021 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Tue, 09 Feb 2021 16:46:39 +0100 Subject: [gpfsug-discuss] Limiting CES SMB shares to specific subnets Message-ID: Hi. We have an ESS 5.0.4.3 cluster with a CES cluster serving files with NFSv4 ACLs to NFS and SMB clients. This system is used for sensitive research data, and will the next years house thousands of research projects, which will have to be strictly separated. Each project has its own subnet for the project linux and windows hosts. Project directories are independent filesets in file systems, each project directory has NFSv4 ACLs giving acces to only the project group. Project NFS shares are limited to each project's subnet. Project SMB shares have export ACLs (as in "mmsmb exportacl ..") limiting share access to the project's member group, in addition to the NFSv4 ACLs. We also want to limit access to SMB shares to project subnets. There is no way to specify that with "mmsmb", but we have found /usr/lpp/mmfs/bin/net conf setparm "hosts allow" to be working, at least with some limited testing: share access is actually limited to the specified subnets. The additional settings seems to be stored in CTDB under /var/lib/ctdb/persistent. We assume that the "net conf setparm" method is not officially supported by IBM. Although it seems to be working, we wonder if it is a good idea to implement it. For instance, we are wondering if the additional settings will survive later ESS code upgrades, and if it will scale to thousands of SMB shares. We are considering doing the SMB subnet limiting outside CES, but that would add complexity and overhead, so we are not very keen on that. What do other IBM ESS customers do, do you have any advice for us? Yea or nay? Regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From christof.schmitt at us.ibm.com Tue Feb 9 18:06:14 2021 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 9 Feb 2021 18:06:14 +0000 Subject: [gpfsug-discuss] Limiting CES SMB shares to specific subnets In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From helge.hauglin at usit.uio.no Wed Feb 10 09:09:01 2021 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Wed, 10 Feb 2021 10:09:01 +0100 Subject: [gpfsug-discuss] Limiting CES SMB shares to specific subnets In-Reply-To: (Christof Schmitt's message of "Tue, 9 Feb 2021 18:06:14 +0000") References: Message-ID: Hi Christof, thanks for your answer. I have added our vote for the RFE, and put us on the watchlist. Is it possible to say anything about when the RFE might be implemented? >> Project SMB shares have export ACLs (as in "mmsmb exportacl ..")> limiting share access to the project's member group, in addition to the> NFSv4 ACLs.>> We also want to limit access to SMB shares to project subnets.> There is no way to specify that with "mmsmb", but we have found>> /usr/lpp/mmfs/bin/net conf setparm "hosts allow" >> to be working, at least with some limited testing: share access is> actually limited to the specified subnets. The additional settings> seems to be stored in CTDB under /var/lib/ctdb/persistent.>> We assume that the "net conf setparm" method is not officially supported> by IBM. Although it seems to be working, we wonder if it is a good idea> to implement it. For instance, we are wondering if the additional> settings will survive later ESS code upgrades, and if it will scale to> thousands of SMB shares. > > Officially Scale only supports Samba options that can be set throughthe GUI or the mmsmb CLI. Everything else set through 'net conf' hasnot been tested and is not supported. In this specific case, this islikely to work, and it should also be preserved across code upgrades,but again, this is not an official support statement. > > This is also not a new request, there is also a pending RFE to makethis an official Scale feature:https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=141534 > > Regards, > > Christof Schmitt > Software Engineer > IBM Systems, Spectrum Scale Development > +1 520 799 2469 > christof.schmitt at us.ibm.com > @chsc Twitter > > IBM > > > ----- Original message -----From: Helge Hauglin Sent by: gpfsug-discuss-bounces at spectrumscale.orgTo: gpfsug-discuss at spectrumscale.orgCc:Subject: [EXTERNAL] [gpfsug-discuss] Limiting CES SMB shares to specific subnetsDate: Tue, Feb 9, 2021 9:10 AM > Hi.We have an ESS 5.0.4.3 cluster with a CES cluster serving files withNFSv4 ACLs to NFS and SMB clients. This system is used forsensitive research data, and will the next years house thousands ofresearch projects, which will have to be strictly separated. Eachproject has its own subnet for the project linux and windows hosts.Project directories are independent filesets in file systems, eachproject directory has NFSv4 ACLs giving acces to only the project group.Project NFS shares are limited to each project's subnet.Project SMB shares have export ACLs (as in "mmsmb exportacl ..")limiting share access to the project's member group, in addition to theNFSv4 ACLs.We also want to limit access to SMB shares to project subnets.There is no way to specify that with "mmsmb", but we have found /usr/lpp/mmfs/bin/net conf setparm "hosts allow" to be working, at least with some limited testing: share access isactually limited to the specified subnets. The additional settingsseems to be stored in CTDB under /var/lib/ctdb/persistent.We assume that the "net conf setparm" method is not officially supportedby IBM. Although it seems to be working, we wonder if it is a good ideato implement it. For instance, we are wondering if the additionalsettings will survive later ESS code upgrades, and if it will scale tothousands of SMB shares.We are considering doing the SMB subnet limiting outside CES, but that wouldadd complexity and overhead, so we are not very keen on that.What do other IBM ESS customers do, do you have any advice for us?Yea or nay?Regards,Helge Hauglin----------------------------------------------------------------Mr. Helge Hauglin, Senior EngineerSystem administratorCenter for Information Technology, University of Oslo, Norway_______________________________________________gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From janfrode at tanso.net Fri Feb 19 12:50:08 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 19 Feb 2021 13:50:08 +0100 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180119163803.79fddbeb@osc.edu> References: <20180119163803.79fddbeb@osc.edu> Message-ID: We just discussed this a bit internally, and I found "something* that might help... There's a mmrestripefs --inode-criteria command that can be used to identify files with these unknown-to-ILM flags set. Something like: # echo illreplicated > criteria # mmrestripefs gpfs01 -p --inode-criteria criteria -o result Scanning file system metadata, phase 1 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning file system metadata, phase 2 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning file system metadata, phase 3 ... Scan completed successfully. Scanning file system metadata, phase 4 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning file system metadata, phase 5 ... 100 % complete on Thu Feb 18 15:30:10 2021 Scan completed successfully. Scanning user file metadata ... 100.00 % complete on Thu Feb 18 15:30:14 2021 ( 835584 inodes with total 4271 MB data processed) Scan completed successfully. Check file '/mnt/gpfs01/result' on scale-dev-01 for inodes with broken disk addresses or failures. # cat /mnt/gpfs01/result This inode list was generated in the Parallel Inode Traverse on Thu Feb 18 15:30:10 2021 INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR]) 55559 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE Unclear to me if "-p" is too much work for this task though, or maybe it can be limited with --metadata-only or other options.. HTH On Sat, Jan 20, 2018 at 12:10 AM Edward Wahl wrote: > > This one has been on my list a long time so I figured I'd ask here first > before > I open an apar or request an enhancement (most likely). > > Is there a way using the policy engine to determine the following? > > -metadata replication total/current > -unbalanced file > > Looking to catch things like this that stand out on my filesystem without > having to run several hundred million 'mmlsattr's. > > metadata replication: 1 max 2 > flags: unbalanced > > Ed > > > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Fri Feb 19 16:29:01 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 19 Feb 2021 17:29:01 +0100 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: References: <20180119163803.79fddbeb@osc.edu> Message-ID: I also see that /usr/lpp/mmfs/samples/util/tsinode will list these flags: [root at scale-dev-01 util]# ./tsinode /mnt/gpfs01/|grep 55559 55559 631878862 0 0 1048576000 104857600 -rw-r--r-- 2 1610965529.234470000 1610965531.035007326 1612952829.394919000 0 blk=1048576 modsnap=1 extperms=0x2,xa replmeta *illReplicated unbalanced* dev=3824,150 archive compressed crtime 1610965529.234470000 That program can probably easily be modified to only list these files.. -jf On Fri, Feb 19, 2021 at 1:50 PM Jan-Frode Myklebust wrote: > We just discussed this a bit internally, and I found "something* that > might help... There's a mmrestripefs --inode-criteria command that can be > used to identify files with these unknown-to-ILM flags set. Something like: > > # echo illreplicated > criteria > # mmrestripefs gpfs01 -p --inode-criteria criteria -o result > Scanning file system metadata, phase 1 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning file system metadata, phase 2 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning file system metadata, phase 3 ... > Scan completed successfully. > Scanning file system metadata, phase 4 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning file system metadata, phase 5 ... > 100 % complete on Thu Feb 18 15:30:10 2021 > Scan completed successfully. > Scanning user file metadata ... > 100.00 % complete on Thu Feb 18 15:30:14 2021 ( 835584 inodes with total 4271 MB data processed) > Scan completed successfully. > Check file '/mnt/gpfs01/result' on scale-dev-01 for inodes with broken disk addresses or failures. > # cat /mnt/gpfs01/result > This inode list was generated in the Parallel Inode Traverse on Thu Feb 18 15:30:10 2021 > INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR]) > 55559 0:0 0 1 0 illreplicated unbalanced REGULAR_FILE > > Unclear to me if "-p" is too much work for this task though, or maybe it can be limited with --metadata-only or other options.. > > > HTH > > > > > On Sat, Jan 20, 2018 at 12:10 AM Edward Wahl wrote: > >> >> This one has been on my list a long time so I figured I'd ask here first >> before >> I open an apar or request an enhancement (most likely). >> >> Is there a way using the policy engine to determine the following? >> >> -metadata replication total/current >> -unbalanced file >> >> Looking to catch things like this that stand out on my filesystem without >> having to run several hundred million 'mmlsattr's. >> >> metadata replication: 1 max 2 >> flags: unbalanced >> >> Ed >> >> >> >> -- >> >> Ed Wahl >> Ohio Supercomputer Center >> 614-292-9302 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tyler.trafford at yale.edu Tue Feb 23 00:39:36 2021 From: tyler.trafford at yale.edu (Trafford, Tyler) Date: Tue, 23 Feb 2021 00:39:36 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu>, <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu>, Message-ID: My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn?t really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 From TROPPENS at de.ibm.com Wed Feb 24 16:17:05 2021 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Wed, 24 Feb 2021 16:17:05 +0000 Subject: [gpfsug-discuss] Digital German Spectrum Scale User Meeting on March 3&4, 2021 Message-ID: An HTML attachment was scrubbed... URL: From jon at well.ox.ac.uk Thu Feb 25 10:13:09 2021 From: jon at well.ox.ac.uk (Jon Diprose) Date: Thu, 25 Feb 2021 10:13:09 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu>, <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu>, Message-ID: I have written an xCAT plugin such that the client can ask the master to do it, in preparation for some shiny new kit. Not in a public repo yet, so please email me direct if you want further info. -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Trafford, Tyler Sent: 23 February 2021 00:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn't really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jpr9c at virginia.edu Thu Feb 25 15:47:41 2021 From: jpr9c at virginia.edu (Ruffner, Scott (jpr9c)) Date: Thu, 25 Feb 2021 15:47:41 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu> <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> Message-ID: I appreciate the solutions; at the moment, I'm making do with having a key for just this specific copy, and doing our own rsync. If we had local disk on our nodes, that would be helpful, but /var/mmfs is mounted on tmpfs and gets flushed at every reboot (not just upgrades to hardware or software stack. I'm testing with invoking the mmsdrrestore from the client only using a key. Yeah, not ideal leaving that laying around. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu ?On 2/25/21, 5:13 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jon Diprose" wrote: I have written an xCAT plugin such that the client can ask the master to do it, in preparation for some shiny new kit. Not in a public repo yet, so please email me direct if you want further info. -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Trafford, Tyler Sent: 23 February 2021 00:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn't really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jon at well.ox.ac.uk Thu Feb 25 16:18:47 2021 From: jon at well.ox.ac.uk (Jon Diprose) Date: Thu, 25 Feb 2021 16:18:47 +0000 Subject: [gpfsug-discuss] Adding client nodes using a shared NFS root image. In-Reply-To: References: <4A332838-9D59-477D-AAE2-F79F8AAD143B@virginia.edu> <094EDEFE-4B15-4214-90C4-CD83BC76A10A@brown.edu> <6A72D8F2-65ED-431C-B13F-3D4F189A53DF@virginia.edu> Message-ID: The call to the plugin can be invoked directly from the client at any time, not just as a result of an install or an updatenode. I see no reason it couldn't go in a script that's run on client startup, though you'd have to hard-code the address+port the xcatd is listening on as that would normally be provided by xCAT. I started from the remoteshell postscript and worked backwards through the getcredentials.awk script to the credentials.pm plugin, and re-used what I needed for a plugin that invokes relevant mmgetstate, mmaddnode and/or mmsdrrestore calls on the master. Alternatively, just look at those existing postscripts/plugins for a mechanism to dynamically obtain the key, which can then be ephemeral on the client. That is the code path xCAT uses to get keys and passwords to the clients, though the comments at the top of credentials.pm (in /opt/xcat/lib/perl/xCAT_plugins/) give a fairly brutal assessment of the security situation. Jon -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Ruffner, Scott (jpr9c) Sent: 25 February 2021 15:48 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. I appreciate the solutions; at the moment, I'm making do with having a key for just this specific copy, and doing our own rsync. If we had local disk on our nodes, that would be helpful, but /var/mmfs is mounted on tmpfs and gets flushed at every reboot (not just upgrades to hardware or software stack. I'm testing with invoking the mmsdrrestore from the client only using a key. Yeah, not ideal leaving that laying around. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu ?On 2/25/21, 5:13 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jon Diprose" wrote: I have written an xCAT plugin such that the client can ask the master to do it, in preparation for some shiny new kit. Not in a public repo yet, so please email me direct if you want further info. -- Dr. Jonathan Diprose Tel: 01865 287837 Research Computing Manager Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Trafford, Tyler Sent: 23 February 2021 00:40 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. My solution to this is to have the below defined in an XCAT syncfiles config: /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/ And then in a postscript this happens on the node being (re)installed: # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break done then /usr/lpp/mmfs/bin/mmsdrrestore else rm /var/mmfs/gen/mmsdrfs fi -Tyler Trafford ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Peter Childs Sent: Monday, February 1, 2021 10:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We used to run mmsdrestore -p manager -R /usr/bin/scp in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host, We now use mmsdrestore -N nodename post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons. Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless. My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it. Peter Childs ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Ruffner, Scott (jpr9c) Sent: Friday, January 29, 2021 8:04 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. Thanks David! Slick solution. -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu From: on behalf of "david_johnson at brown.edu" Reply-To: gpfsug main discussion list Date: Friday, January 29, 2021 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image. We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT. The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup. -- ddj Dave Johnson On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) wrote: Hi everyone, We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image. Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn't really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe? Am I going about this the entirely wrong way? -- Scott Ruffner Senior HPC Engineer UVa Research Computing (434)924-6778(o) (434)295-0250(h) sruffner at virginia.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From Robert.Oesterlin at nuance.com Fri Feb 26 01:49:42 2021 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 26 Feb 2021 01:49:42 +0000 Subject: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Message-ID: <4397068B-AD6C-445E-AA07-390FBA7B743F@nuance.com> I have 2 nodes in a new ESS cluster that won?t display data via the GUI, but I can pull up data using the command line. Node is healthy, pmsensors is running and shows connected to the collector. Command line shows this: [root at ems1 IBM]# mmperfmon query compareNodes cpu_user -b 3600 -n 2 Legend: 1: afmgw03|CPU|cpu_user 2: afmgw04|CPU|cpu_user 3: ems1.gpfs.net|CPU|cpu_user 4: essio1.gpfs.net|CPU|cpu_user 5: essio2.gpfs.net|CPU|cpu_user Row Timestamp afmgw03 afmgw04 ems1 essio1 essio2 1 2021-02-25-19:00:00 0.069542 0.059736 0.231 0.196292 0.213047 2 2021-02-25-20:00:00 0.076463 0.067958 0.270258 0.221003 0.230289 But when I try and view afmgw03 or afmgw04 on the GUI, no data. What?s up? Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Fri Feb 26 12:18:24 2021 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 26 Feb 2021 07:18:24 -0500 Subject: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line In-Reply-To: <4397068B-AD6C-445E-AA07-390FBA7B743F@nuance.com> References: <4397068B-AD6C-445E-AA07-390FBA7B743F@nuance.com> Message-ID: Bob, could you please provide the version of ESS/Scale you have installed? Also, could you please provide information about the exact GUI screen you are using that is not providing the data? Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 02/25/2021 09:41 PM Subject: [EXTERNAL] [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Sent by: gpfsug-discuss-bounces at spectrumscale.org I have 2 nodes in a new ESS cluster that won?t display data via the GUI, but I can pull up data using the command line. Node is healthy, pmsensors is running and shows connected to the collector. Command line shows this: [root at ems1 IBM]# mmperfmon query compareNodes cpu_user -b 3600 -n 2 Legend: 1: afmgw03|CPU|cpu_user 2: afmgw04|CPU|cpu_user 3: ems1.gpfs.net|CPU|cpu_user 4: essio1.gpfs.net|CPU|cpu_user 5: essio2.gpfs.net|CPU|cpu_user Row Timestamp afmgw03 afmgw04 ems1 essio1 essio2 1 2021-02-25-19:00:00 0.069542 0.059736 0.231 0.196292 0.213047 2 2021-02-25-20:00:00 0.076463 0.067958 0.270258 0.221003 0.230289 But when I try and view afmgw03 or afmgw04 on the GUI, no data. What?s up? Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=_afEd51Its1IKJqf-evUvjCHSBCrdDMxTJWn0juEBBI&s=Vn6zN83tuKly6fA8xw128AYfimkX9PZFDY9TRBEQ45s&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Feb 26 13:36:57 2021 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 26 Feb 2021 13:36:57 +0000 Subject: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Message-ID: <5F3EDDA0-28C2-460E-8E99-77E728E4618D@nuance.com> ESS 6.0.1.2 with GPFS 5.0.5.4 What?s even stranger is that I logged in this morning and it was working so? ? Not sure if we should/need to follow-up. Bob Oesterlin Sr Principal Storage Engineer, Nuance 507-269-0413 From: on behalf of IBM Spectrum Scale Reply-To: gpfsug main discussion list Date: Friday, February 26, 2021 at 6:18 AM To: gpfsug main discussion list Cc: "gpfsug-discuss-bounces at spectrumscale.org" Subject: [EXTERNAL] Re: [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line CAUTION: This Email is from an EXTERNAL source. Ensure you trust this sender before clicking on any links or attachments. ________________________________ Bob, could you please provide the version of ESS/Scale you have installed? Also, could you please provide information about the exact GUI screen you are using that is not providing the data? Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 02/25/2021 09:41 PM Subject: [EXTERNAL] [gpfsug-discuss] Odd GUI behavior - node will not display in GUI, can pull data via command line Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I have 2 nodes in a new ESS cluster that won?t display data via the GUI, but I can pull up data using the command line. Node is healthy, pmsensors is running and shows connected to the collector. Command line shows this: [root at ems1 IBM]# mmperfmon query compareNodes cpu_user -b 3600 -n 2 Legend: 1: afmgw03|CPU|cpu_user 2: afmgw04|CPU|cpu_user 3: ems1.gpfs.net|CPU|cpu_user 4: essio1.gpfs.net|CPU|cpu_user 5: essio2.gpfs.net|CPU|cpu_user Row Timestamp afmgw03 afmgw04 ems1 essio1 essio2 1 2021-02-25-19:00:00 0.069542 0.059736 0.231 0.196292 0.213047 2 2021-02-25-20:00:00 0.076463 0.067958 0.270258 0.221003 0.230289 But when I try and view afmgw03 or afmgw04 on the GUI, no data. What?s up? Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=_afEd51Its1IKJqf-evUvjCHSBCrdDMxTJWn0juEBBI&s=Vn6zN83tuKly6fA8xw128AYfimkX9PZFDY9TRBEQ45s&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Sat Feb 27 18:01:02 2021 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sat, 27 Feb 2021 18:01:02 +0000 Subject: [gpfsug-discuss] dssgmkfs.mmvdisk number of NSD's Message-ID: Doing an upgrade on our storage which involved replacing all the 4TB disks with 16TB disks. Some hiccups with five of the disks being dead when inserted but that is all sorted. So the system was originally installed with DSS-G 2.0a so with "legacy" commands for vdisks etc. We had 10 metadata NSD's and 10 data NSD's per draw aka recovery group of the D3284 enclosures. The dssgmkfs.mmvdisk has created exactly one data and one metadata NSD per draw of a DS3284 leading to a really small number of NSD's in the file system. All my instincts tell me that this is going to lead to horrible performance on the file system. Historically you wanted a reasonable number of NSD's in a system for decent performance. Taking what the ddsgmkfs.mmvdisk has give me even with a DSS-G260 you would get only 12 NSD's of each type, which for a potentially ~5PB file system seems on the really low side to me. Is there any way to tell ddsgmkfs.mmvdisk to create more NSD's than the one per recovery group or is this no longer relevant and performance with really low numbers of NSD's is fine these days? JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From janfrode at tanso.net Sun Feb 28 09:31:57 2021 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sun, 28 Feb 2021 10:31:57 +0100 Subject: [gpfsug-discuss] dssgmkfs.mmvdisk number of NSD's In-Reply-To: References: Message-ID: I?ve tried benchmarking many vs. few vdisks per RG, and never could see any performance difference. Usually we create 1 vdisk per enclosure per RG, thinking this will allow us to grow with same size vdisks when adding additional enclosures in the future. Don?t think mmvdisk can be told to create multiple vdisks per RG directly, so you have to manually create multiple vdisk sets each with the apropriate size. -jf l?r. 27. feb. 2021 kl. 19:01 skrev Jonathan Buzzard < jonathan.buzzard at strath.ac.uk>: > > Doing an upgrade on our storage which involved replacing all the 4TB > disks with 16TB disks. Some hiccups with five of the disks being dead > when inserted but that is all sorted. > > So the system was originally installed with DSS-G 2.0a so with "legacy" > commands for vdisks etc. We had 10 metadata NSD's and 10 data NSD's per > draw aka recovery group of the D3284 enclosures. > > The dssgmkfs.mmvdisk has created exactly one data and one metadata NSD > per draw of a DS3284 leading to a really small number of NSD's in the > file system. > > All my instincts tell me that this is going to lead to horrible > performance on the file system. Historically you wanted a reasonable > number of NSD's in a system for decent performance. > > Taking what the ddsgmkfs.mmvdisk has give me even with a DSS-G260 you > would get only 12 NSD's of each type, which for a potentially ~5PB file > system seems on the really low side to me. > > Is there any way to tell ddsgmkfs.mmvdisk to create more NSD's than the > one per recovery group or is this no longer relevant and performance > with really low numbers of NSD's is fine these days? > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: