From mail at arif-ali.co.uk Tue Jan 2 10:29:11 2018 From: mail at arif-ali.co.uk (Arif Ali) Date: Tue, 2 Jan 2018 10:29:11 +0000 Subject: [gpfsug-discuss] Testing to see the mailing list is working -- please ignore Message-ID: <42f7a66e-ff42-ef26-8ece-888fdc932f8e@arif-ali.co.uk> Just a quick test message, to ensure mailing list is working, as we've not heard since 22nd December, So please ignore -- regards, Arif Ali -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From r.sobey at imperial.ac.uk Wed Jan 3 10:37:54 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 3 Jan 2018 10:37:54 +0000 Subject: [gpfsug-discuss] mmchfileset syntax Message-ID: Hi all, Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. mmchfileset gpfs filesetname -J /gpfs/new_path mmchfileset: Options -J and FilesetName cannot be specified at the same time. Which leads to what?s the point of the following command as the error suggests it will work: mmchfileset device -J /gpfs/new_path ?as not specifying a FilesetName to the input of mmchfileset seems odd. I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jan 3 11:49:42 2018 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 3 Jan 2018 11:49:42 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: Message-ID: -J is to define which fileset you are talking about. The alternative is give the fileset name straight after the ?device? argument. Once the fileset is unambiguous, you then use other command line option to change things e.g. -j to change the *name* of this fileset. -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. Daniel Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com > On 3 Jan 2018, at 10:38, Sobey, Richard A wrote: > > Hi all, > > Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. > > mmchfileset gpfs filesetname -J /gpfs/new_path > mmchfileset: Options -J and FilesetName cannot be specified at the same time. > > Which leads to what?s the point of the following command as the error suggests it will work: > > mmchfileset device -J /gpfs/new_path > > ?as not specifying a FilesetName to the input of mmchfileset seems odd. > > I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? > > Cheers > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Wed Jan 3 12:37:46 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 3 Jan 2018 12:37:46 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: References: Message-ID: Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and mmchfileset gets the filesetname from that). Ultimately though I cannot change the mount point of the fileset without mmunlink / mmlink, right? Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger Sent: 03 January 2018 11:50 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmchfileset syntax -J is to define which fileset you are talking about. The alternative is give the fileset name straight after the ?device? argument. Once the fileset is unambiguous, you then use other command line option to change things e.g. -j to change the *name* of this fileset. -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. Daniel [/spectrum_storage-banne] [Spectrum Scale Logo] Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com On 3 Jan 2018, at 10:38, Sobey, Richard A > wrote: Hi all, Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. mmchfileset gpfs filesetname -J /gpfs/new_path mmchfileset: Options -J and FilesetName cannot be specified at the same time. Which leads to what?s the point of the following command as the error suggests it will work: mmchfileset device -J /gpfs/new_path ?as not specifying a FilesetName to the input of mmchfileset seems odd. I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? Cheers Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From tconnelly at pixitmedia.com Wed Jan 3 13:27:13 2018 From: tconnelly at pixitmedia.com (Tim Connelly) Date: Wed, 3 Jan 2018 13:27:13 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: References: Message-ID: >From the mmlinkfileset docs; "The user may use the mv command on the directory to move to a new location in the parent fileset, but the mv command is not allowed to move the junction to a different fileset." # mmlsfileset mmfs1 |grep timt sas1-timtest Linked /mmfs1/another_test/timtest # mv /mmfs1/another_test/timtest /mmfs1/another_test/timtest2 # mmlsfileset mmfs1 |grep timt sas1-timtest Linked /mmfs1/another_test/timtest2 Hope this helps! Tim On 3 January 2018 at 12:37, Sobey, Richard A wrote: > Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and > mmchfileset gets the filesetname from that). > > > > Ultimately though I cannot change the mount point of the fileset without > mmunlink / mmlink, right? > > > > Thanks > > Richard > > > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: > gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Daniel Kidger > *Sent:* 03 January 2018 11:50 > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] mmchfileset syntax > > > > -J is to define which fileset you are talking about. > > The alternative is give the fileset name straight after the ?device? > argument. > > > > Once the fileset is unambiguous, you then use other command line option to > change things > > e.g. -j to change the *name* of this fileset. > > -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the > mount point either. > > > > Daniel > > [image: /spectrum_storage-banne] > > > > > [image: Spectrum Scale Logo] > > > > > > *Dr Daniel Kidger* > IBM Technical Sales Specialist > Software Defined Solution Sales > > + <+%2044-7818%20522%20266> 44-(0)7818 522 266 <+%2044-7818%20522%20266> > daniel.kidger at uk.ibm.com > > > On 3 Jan 2018, at 10:38, Sobey, Richard A wrote: > > Hi all, > > > > Attempting to change the junction path of a fileset and I?m getting some > strange output. It may just not be possible but the documentation on > mmchfileset itself doesn?t make it clear. > > > > mmchfileset gpfs filesetname -J /gpfs/new_path > > mmchfileset: Options -J and FilesetName cannot be specified at the same > time. > > > > Which leads to what?s the point of the following command as the error > suggests it will work: > > > > mmchfileset device -J /gpfs/new_path > > > > ?as not specifying a FilesetName to the input of mmchfileset seems odd. > > > > I do of course know that I can just unlink and relink but hey.. trying to > save a few precious seconds ? > > > > Cheers > > Richard > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jan 3 13:13:03 2018 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 3 Jan 2018 13:13:03 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: Message-ID: Richard, The documentation says that that you can simply use ?mv? on a live fileset too. I have just tried this and yes it appears to work. I can move a fileset into another subdirectory of the same filesystem and mmlsfileset confirms the new mount point. Daniel Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com > On 3 Jan 2018, at 12:37, Sobey, Richard A wrote: > > Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and mmchfileset gets the filesetname from that). > > Ultimately though I cannot change the mount point of the fileset without mmunlink / mmlink, right? > > Thanks > Richard > > > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger > Sent: 03 January 2018 11:50 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] mmchfileset syntax > > -J is to define which fileset you are talking about. > The alternative is give the fileset name straight after the ?device? argument. > > Once the fileset is unambiguous, you then use other command line option to change things > e.g. -j to change the *name* of this fileset. > -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. > > > Daniel > > > > > > > Dr Daniel Kidger > IBM Technical Sales Specialist > Software Defined Solution Sales > > + 44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > On 3 Jan 2018, at 10:38, Sobey, Richard A wrote: > > Hi all, > > Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. > > mmchfileset gpfs filesetname -J /gpfs/new_path > mmchfileset: Options -J and FilesetName cannot be specified at the same time. > > Which leads to what?s the point of the following command as the error suggests it will work: > > mmchfileset device -J /gpfs/new_path > > ?as not specifying a FilesetName to the input of mmchfileset seems odd. > > I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? > > Cheers > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Wed Jan 3 13:58:20 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 3 Jan 2018 13:58:20 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: References: Message-ID: Thank you Daniel and Tim, most helpful. Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger Sent: 03 January 2018 13:13 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmchfileset syntax Richard, The documentation says that that you can simply use ?mv? on a live fileset too. I have just tried this and yes it appears to work. I can move a fileset into another subdirectory of the same filesystem and mmlsfileset confirms the new mount point. Daniel [Image removed by sender. /spectrum_storage-banne] [Image removed by sender. Spectrum Scale Logo] Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com On 3 Jan 2018, at 12:37, Sobey, Richard A > wrote: Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and mmchfileset gets the filesetname from that). Ultimately though I cannot change the mount point of the fileset without mmunlink / mmlink, right? Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger Sent: 03 January 2018 11:50 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] mmchfileset syntax -J is to define which fileset you are talking about. The alternative is give the fileset name straight after the ?device? argument. Once the fileset is unambiguous, you then use other command line option to change things e.g. -j to change the *name* of this fileset. -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. Daniel [Image removed by sender. /spectrum_storage-banne] [Image removed by sender. Spectrum Scale Logo] Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com On 3 Jan 2018, at 10:38, Sobey, Richard A > wrote: Hi all, Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. mmchfileset gpfs filesetname -J /gpfs/new_path mmchfileset: Options -J and FilesetName cannot be specified at the same time. Which leads to what?s the point of the following command as the error suggests it will work: mmchfileset device -J /gpfs/new_path ?as not specifying a FilesetName to the input of mmchfileset seems odd. I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? Cheers Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ~WRD029.jpg Type: image/jpeg Size: 823 bytes Desc: ~WRD029.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2519 bytes Desc: image001.jpg URL: From hoov at us.ibm.com Wed Jan 3 17:10:51 2018 From: hoov at us.ibm.com (Theodore Hoover Jr) Date: Wed, 3 Jan 2018 17:10:51 +0000 Subject: [gpfsug-discuss] Fw: Spectrum Scale on AWS - Join Sponsor User Program Message-ID: An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Thu Jan 4 11:16:57 2018 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 4 Jan 2018 11:16:57 +0000 Subject: [gpfsug-discuss] Use AFM for migration of many small files In-Reply-To: References: <467FB293-D33B-46F4-BA1B-A5CB4D28DDE6@psi.ch> Message-ID: <1515064616.28898.32.camel@qmul.ac.uk> We are doing something very similar using 4.2.3-4 and GPFS 4.2.1-1 on the nfs backend. Did you have any success? The plan is to load the new cache from the old GPFS then dump once the cache is full. We've already increase numThreashThreads from 4 to 8 and seen only marginal improvements, we could attempt to increase this further. I'm also wondering if its worth increasing the Refresh Intervals to speed up read of already cache files. At this stage we want to fill the cache and don't care about write back until we switch the target to the new NFS/GPFS from our old GPFS storage to a new box back at our off-site location, (otherwise known as the office) [root at afmgateway1 scratch]# mmlsfileset home home -L --afm Filesets in file system 'home': Attributes for fileset home: ============================= Status Linked Path /data2/home Id 42 Root inode 1343225859 Parent Id 0 Created Wed Jan 3 12:32:33 2018 Comment Inode space 41 Maximum number of inodes 100000000 Allocated inodes 15468544 Permission change flag chmodAndSetacl afm-associated Yes Target nfs://afm1/gpfs/data1/afm/home Mode single-writer File Lookup Refresh Interval 30 (default) File Open Refresh Interval 30 (default) Dir Lookup Refresh Interval 60 (default) Dir Open Refresh Interval 60 (default) Async Delay 15 (default) Last pSnapId 0 Display Home Snapshots no Number of Gateway Flush Threads 8 Prefetch Threshold 0 (default) Eviction Enabled no Thanks in advance. Peter Childs On Tue, 2017-09-05 at 19:57 +0530, Venkateswara R Puvvada wrote: Which version of Spectrum Scale ? What is the fileset mode ? >We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here >I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. How was the performance measured ? If parallel IO is enabled, AFM uses multiple gateway nodes to prefetch the large files (if file size if more than 1GB). Performance difference between small and lager file is huge (1000MB - 150MB = 850MB) here, and generally it is not the case. How many files were present in list file for prefetch ? Could you also share full internaldump from the gateway node ? >I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few >read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. AFM prefetches the files on multiple threads. Default flush threads for prefetch are 36 (fileset.afmNumFlushThreads (default 4) + afmNumIOFlushThreads (default 32)). >Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and >I need to look elsewhere to get better performance for prefetch of many smaller files? See above, AFM reads files on multiple threads parallelly. Try increasing the afmNumFlushThreads on fileset and verify if it improves the performance. ~Venkat (vpuvvada at in.ibm.com) From: "Billich Heinrich Rainer (PSI)" To: "gpfsug-discuss at spectrumscale.org" Date: 09/04/2017 10:18 PM Subject: [gpfsug-discuss] Use AFM for migration of many small files Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and I need to look elsewhere to get better performance for prefetch of many smaller files? We will migrate several filesets in parallel, but still with individual filesets up to 350TB in size 150MB/s isn?t fun. Also just about 150 files/s seconds looks poor. The setup is quite new, hence there may be other places to look at. It?s all RHEL7 an spectrum scale 4.2.2-3 on the afm cache. Thank you, Heiner --, Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://urldefense.proofpoint.com/v2/url?u=https-3A__www.psi.ch&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=eHcVdovN10-m-Qk0Ln2qvol3pkKNFwrzz2wgf1zXVXE&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=LbRyuSM_djs0FDXr27hPottQHAn3OGcivpyRcIDBN3U&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Peter Childs ITS Research Storage Queen Mary, University of London -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Thu Jan 4 17:57:14 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Thu, 4 Jan 2018 17:57:14 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Message-ID: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bipcuds at gmail.com Thu Jan 4 20:34:30 2018 From: bipcuds at gmail.com (Keith Ball) Date: Thu, 4 Jan 2018 15:34:30 -0500 Subject: [gpfsug-discuss] Conflicting RHEL compatibility information in the Spectrum Scale FAQ Message-ID: Hi All, Here is a repeat post of my question on GSS 2.0.7 compatibility with RHEL 6.7. FAQ is contradictory. I know GSS/ESS is usually quite sensitive to use a specific RHEL minor version). And at this point, RHEL 6.5 isn't really supported anymore (no updates provided). On Tue, Dec 19, 2017 at 6:08 PM, Keith Ball wrote: I was recently trying to determine the latest RHEL release that will work > with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code > uses Scale 4.1.0.8. > > A specific X.Y GSS code build, from my experience, is intended to use a > specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), > which no longer has EUS support from RedHat (only 6.7 still does). > > GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be > used, which is a surprising statement given GSS/ESS code's sensitivity to > OS levels (any ESS I have ever seen has never been supported on more than > one version of RHEL). > > According to the Scale FAQ (https://www.ibm.com/support/ > knowledgecenter/STXKQY/gpfsclustersfaq.html#linux), A 2.2, Table 27, > Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and > 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 > is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 > release" ... which contradicts Table 27! > > Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install > GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is > supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option > here). > > Many Thanks, > Keith > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skylar2 at u.washington.edu Fri Jan 5 00:52:31 2018 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Thu, 4 Jan 2018 16:52:31 -0800 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: <85c8b317-a7cb-ce84-81bd-0a7e8b75bc7e@u.washington.edu> While I'm not fully versed on the vulnerability or the proposed fixes, my understanding is that most of the performance impact from the fix is around having kernel memory completely separately from a process's user-space, which means every system call will have cache/TLB misses. This might mean clusters which are using RDMA won't have as big of a performance hit versus ones that aren't able to use RDMA, since they can do a lot more in user-space without involving the kernel. On 01/04/2018 09:57 AM, Buterbaugh, Kevin L wrote: > Happy New Year everyone, > > I?m sure that everyone is aware of Meltdown and Spectre by now ? we, > like many other institutions, will be patching for it at the earliest > possible opportunity. > > Our understanding is that the most serious of the negative performance > impacts of these patches will be for things like I/O (disk / network) > ? given that, we are curious if IBM has any plans for a GPFS update > that could help mitigate those impacts? ?Or is there simply nothing > that can be done? > > If there is a GPFS update planned for this we?d be interested in > knowing so that we could coordinate the kernel and GPFS upgrades on > our cluster. > > Thanks? > > Kevin > > P.S. ?The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a > good year for everyone despite how it?s starting out. ?:-O > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu > ?- (615)875-9633 > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Thu Jan 4 22:55:18 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Thu, 4 Jan 2018 17:55:18 -0500 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=m7Pdt9KL82CJT_AT-PwkmO3PbHg88-IQ7Jq-dwhDOdY&s=5i66Rx3vse5LcaN4-WlyCwi_TDTOQGQR2-X_XyjbBpw&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From richardb+gpfsUG at ellexus.com Fri Jan 5 08:09:38 2018 From: richardb+gpfsUG at ellexus.com (Richard Booth) Date: Fri, 5 Jan 2018 08:09:38 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Message-ID: Dear Kevin The company I work for might be able to help get back the performance lost from the fixes for Meltdown and Spectre. I work for Ellexus, an I/O profiling company. Our tools could help identify any inefficient I/O patterns in your applications to reduce the impact of this. We can set up demos and a free trial if you're interested in seeing what we do and how we can help. Kind regards Richard Message: 1 Date: Thu, 4 Jan 2018 17:57:14 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Message-ID: <5D655862-7F60-47F6-8BD2-A5298F73F70F at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -- Richard Booth Account Manager, Ellexus Ltd richard.booth at ellexus.com +44 (0)1223 42 16 46 Ellexus is the I/O profiling company. www.ellexus.com *Ellexus Ltd is a limited company registered in England & Wales* *Company registration no. 07166034* *Registered address: 198 High Street, Tonbridge, Kent TN9 1BE, UK* *Operating address: St John's Innovation Centre, Cowley Road, Cambridge CB4 0WS, UK* -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Fri Jan 5 12:39:11 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 5 Jan 2018 18:09:11 +0530 Subject: [gpfsug-discuss] Password to GUI forgotten In-Reply-To: <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu> References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu><2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com><26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu><348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu><1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov> <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu> Message-ID: Hi Kevin, If you are stuck then please open a PMR and work with the IBM support folks to get this resolved. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Buterbaugh, Kevin L" To: "Hanley, Jesse A." Cc: gpfsug main discussion list Date: 12/19/2017 01:42 AM Subject: Re: [gpfsug-discuss] Password to GUI forgotten Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Jesse, Thanks for the suggestion ? I find the following error very interesting: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/rmuser admin EFSSP0010C CLI parser: The object "admin" specified for "userID" does not exist. /root root at testnsd1# That says to me that I don?t have an admin user, which - if true - would explain why not a single password I can think of works. ;-) But as I mentioned in my original post I had this up and working earlier this fall. While I can?t prove anything, I can?t imagine a scenario where I would deliberately choose a non-default username. So if ?admin? has been the default login for the GPFS GUI all along then I am really mystified. Thanks! Kevin On Dec 18, 2017, at 1:58 PM, Hanley, Jesse A. wrote: Kevin, I ran into this a couple times using 4.2.3. This is what we used to get around it: /usr/lpp/mmfs/gui/cli/rmuser admin /usr/lpp/mmfs/gui/cli/mkuser admin -p -g Administrator,SecurityAdmin You may need to run the initgui command if those objects are present. That typically gets run on first login to the GUI. Thanks, -- Jesse From: on behalf of "Buterbaugh, Kevin L" Reply-To: gpfsug main discussion list Date: Monday, December 18, 2017 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten Hi All, Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up. Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this. Richards? suggestion below errors for me on the ?-g Administrator? part. Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked. No matter what I do, I cannot log in to the GUI. Thanks for the suggestions, though? Kevin On Dec 7, 2017, at 6:10 AM, Sobey, Richard A wrote: Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error. That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself. [root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin EFSSG0019I The user admin2 has been successfully created. EFSSG1000I The command completed successfully. Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A Sent: 07 December 2017 11:57 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out. Get Outlook for Android From: gpfsug-discuss-bounces at spectrumscale.org < gpfsug-discuss-bounces at spectrumscale.org> on behalf of Buterbaugh, Kevin L Sent: Wednesday, December 6, 2017 10:41:12 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist. /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist. /root root at testnsd1# I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case. Thanks! Kevin On Dec 6, 2017, at 4:32 PM, Joshua Kwedar wrote: Hmm.. odd. Here?s what the lsuser output should look like. # /usr/lpp/mmfs/gui/cli/lsuser Name Long name Password status Group names Failed login attempts admin active Administrator,SecurityAdmin 0 EFSSG1000I The command completed successfully. Can you try something like? # /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin From: on behalf of "Buterbaugh, Kevin L" Reply-To: gpfsug main discussion list Date: Wednesday, December 6, 2017 at 5:15 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, Sorry - should?ve mentioned that: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 EFSSG0001C Cannot validate option: login /root root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts: /root root at testnsd1# Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November. Thanks... Kevin On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) wrote: The GUI password can be changed via command line using chuser. /usr/lpp/mmfs/gui/cli/chuser Usage is as follows (where userID = admin) chuser userID {-p | -l | -a | -d | -g | --expirePassword} [-o ] Josh K On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L < Kevin.Buterbaugh at Vanderbilt.Edu> wrote: Hi All, So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall. However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI. :-( Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch? If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route. Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-) Thanks? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=B07CFG_8Wsf_UKWDoD32OlUNVRy3yFTESgTgZPki4Wg&s=A5cFcjLgIwulsbnh_cPIBawmU32lkOd4dqyl9_yzELA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jan 5 14:44:04 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 5 Jan 2018 14:44:04 +0000 Subject: [gpfsug-discuss] Password to GUI forgotten In-Reply-To: References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu> <2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com> <26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu> <348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu> <1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov> <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu> Message-ID: Hi GPFS team, I did open a PMR and they (mainly Matthais) did help me get that issue resolved. Thanks for following up! Kevin On Jan 5, 2018, at 6:39 AM, IBM Spectrum Scale > wrote: Hi Kevin, If you are stuck then please open a PMR and work with the IBM support folks to get this resolved. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Buterbaugh, Kevin L" > To: "Hanley, Jesse A." > Cc: gpfsug main discussion list > Date: 12/19/2017 01:42 AM Subject: Re: [gpfsug-discuss] Password to GUI forgotten Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi Jesse, Thanks for the suggestion ? I find the following error very interesting: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/rmuser admin EFSSP0010C CLI parser: The object "admin" specified for "userID" does not exist. /root root at testnsd1# That says to me that I don?t have an admin user, which - if true - would explain why not a single password I can think of works. ;-) But as I mentioned in my original post I had this up and working earlier this fall. While I can?t prove anything, I can?t imagine a scenario where I would deliberately choose a non-default username. So if ?admin? has been the default login for the GPFS GUI all along then I am really mystified. Thanks! Kevin On Dec 18, 2017, at 1:58 PM, Hanley, Jesse A. > wrote: Kevin, I ran into this a couple times using 4.2.3. This is what we used to get around it: /usr/lpp/mmfs/gui/cli/rmuser admin /usr/lpp/mmfs/gui/cli/mkuser admin -p -g Administrator,SecurityAdmin You may need to run the initgui command if those objects are present. That typically gets run on first login to the GUI. Thanks, -- Jesse From: > on behalf of "Buterbaugh, Kevin L" > Reply-To: gpfsug main discussion list > Date: Monday, December 18, 2017 at 2:52 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Password to GUI forgotten Hi All, Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up. Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this. Richards? suggestion below errors for me on the ?-g Administrator? part. Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked. No matter what I do, I cannot log in to the GUI. Thanks for the suggestions, though? Kevin On Dec 7, 2017, at 6:10 AM, Sobey, Richard A > wrote: Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error. That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself. [root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin EFSSG0019I The user admin2 has been successfully created. EFSSG1000I The command completed successfully. Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org[mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A Sent: 07 December 2017 11:57 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Password to GUI forgotten This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out. Get Outlook for Android ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org> on behalf of Buterbaugh, Kevin L > Sent: Wednesday, December 6, 2017 10:41:12 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist. /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist. /root root at testnsd1# I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case. Thanks! Kevin On Dec 6, 2017, at 4:32 PM, Joshua Kwedar > wrote: Hmm.. odd. Here?s what the lsuser output should look like. # /usr/lpp/mmfs/gui/cli/lsuser Name Long name Password status Group names Failed login attempts admin active Administrator,SecurityAdmin 0 EFSSG1000I The command completed successfully. Can you try something like? # /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin From: > on behalf of "Buterbaugh, Kevin L" > Reply-To: gpfsug main discussion list > Date: Wednesday, December 6, 2017 at 5:15 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, Sorry - should?ve mentioned that: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 EFSSG0001C Cannot validate option: login /root root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts: /root root at testnsd1# Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November. Thanks... Kevin On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) > wrote: The GUI password can be changed via command line using chuser. /usr/lpp/mmfs/gui/cli/chuser Usage is as follows (where userID = admin) chuser userID {-p | -l | -a | -d | -g | --expirePassword} [-o ] Josh K On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L > wrote: Hi All, So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall. However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI. :-( Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch? If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route. Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-) Thanks? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=B07CFG_8Wsf_UKWDoD32OlUNVRy3yFTESgTgZPki4Wg&s=A5cFcjLgIwulsbnh_cPIBawmU32lkOd4dqyl9_yzELA&e= ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Jan 6 10:46:16 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Sat, 6 Jan 2018 16:16:16 +0530 Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ In-Reply-To: References: Message-ID: Hi Lyle, Can you please forward this to the right folks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Keith Ball To: gpfsug-discuss at spectrumscale.org Date: 12/20/2017 04:39 AM Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was recently trying to determine the latest RHEL release that will work with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code uses Scale 4.1.0.8. A specific X.Y GSS code build, from my experience, is intended to use a specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), which no longer has EUS support from RedHat (only 6.7 still does). GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be used, which is a surprising statement given GSS/ESS code's sensitivity to OS levels (any ESS I have ever seen has never been supported on more than one version of RHEL). According to the Scale FAQ ( https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux ), A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27! Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option here). Many Thanks, Keith_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=u7hLzIRwApuSZgw6uZ_j3uewYEyRTPxRmWJhpM_UAwY&s=bBgN8s6sKOU36_ivUi5NHGarxlF3TNLPyXS-7PA34F0&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Jan 6 11:17:34 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Sat, 6 Jan 2018 16:47:34 +0530 Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group without takeover In-Reply-To: References: Message-ID: Hi Veera, Can you please help in answering Damir's query. --------------------------- My question is, is there a way of brining back the IO server into the mix without the recoverygroup takeover happening? Could I just start a gpfs and have it back in the mix as a backup server for the recoverygroup and if so, how do you do that. Right now that server is designated as primary server for the recovery group. I would like to have both IO servers in the mix for redundancy purposes. --------------------------- Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Damir Krstic To: gpfsug main discussion list Date: 12/22/2017 11:15 PM Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group without takeover Sent by: gpfsug-discuss-bounces at spectrumscale.org It's been a very frustrating couple of months with our 2 ESS systems. IBM tells us we had blueflame bug and they came on site and updated our ESS to the latest version back in middle of November. Wednesday night one of the NSD servers in one of our ESS building blocks kernel panicked. No idea why and none of the logs are insightful. We have a PMR open with IBM. I am not very confident we will get to the bottom of what's causing kernel panics on our IO servers. The system has gone down over 4 times now in 2 months. When we tried brining it back up, it rejoined the recovery group and the IO on the entire cluster locked up until we were able to find couple of compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on those nodes resolved the issue of the filesystem locking up. So far we have never been successful in brining back an IO server and not having a filesystem lock up until we find a node with pending state with tscomm. Anyway, the system was stable for few minutes until the same IO server that went down on Wednesday night went into an arbitrating mode. It never recovered. We stopped gpfs on that server and IO recovered again. We left gpfs down and cluster seems to be OK. My question is, is there a way of brining back the IO server into the mix without the recoverygroup takeover happening? Could I just start a gpfs and have it back in the mix as a backup server for the recoverygroup and if so, how do you do that. Right now that server is designated as primary server for the recovery group. I would like to have both IO servers in the mix for redundancy purposes. This ESS situation is beyond frustrating and I don't see end in sight. Any help is appreciated._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=vGabwIUw-ziMAtM7VfTppRp3S16NsgGOk5qMe50gtIQ&s=eplQuGhWVZMQ3tBLeqhCKpZ0w0rIiU-2R-UuqHYSsVA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Jan 6 10:21:14 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Sat, 6 Jan 2018 15:51:14 +0530 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35 In-Reply-To: <44E99F55-25CC-48DB-9AD6-E7D6794694DC@nasa.gov> References: , <4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz> <44E99F55-25CC-48DB-9AD6-E7D6794694DC@nasa.gov> Message-ID: Hi Lyle, Can you forward this to the right person. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]" To: gpfsug main discussion list Date: 12/19/2017 02:01 PM Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35 Sent by: gpfsug-discuss-bounces at spectrumscale.org It?s not supported on SLES11 either. IBM didn?t (that I saw) talk much about this publicly or give customers a chance to provide feedback about the decision. I know it was raised at the UG in NY and I recall a number of people saying it would be a significant issue for them (myself included) as is the fact they no longer support Debian with scale 5.0. I?d raised the issue on the mailing list after the UG trying to start the discussion but IBM said they weren?t ready to talk about it publicly and I can only guess they had already set their sights and didn?t actually want feedback. This is actually pretty frustrating. I?m tempted to open an RFE but most of my RFEs either have been rejected or just sit idle so I?m not clear there?s a benefit. On December 19, 2017 at 03:08:27 EST, atmane khiredine wrote: IBM Spectrum Scale V5.0 not support RHEL 6.x Only RHEL 7.1 or later https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz ________________________________________ De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org] Envoy? : lundi 18 d?cembre 2017 22:46 ? : gpfsug-discuss at spectrumscale.org Objet : gpfsug-discuss Digest, Vol 71, Issue 35 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: FW: Spectrum Scale 5.0 now available on Fix Central (Michael L Taylor) 2. Re: gpfs 4.2.3.5 and RHEL 7.4... (Frederick Stock) 3. Re: gpfs 4.2.3.5 and RHEL 7.4... (Eric Horst) ---------------------------------------------------------------------- Message: 1 Date: Mon, 18 Dec 2017 13:27:42 -0700 From: "Michael L Taylor" To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central Message-ID: Content-Type: text/plain; charset="us-ascii" Hi Bob, Thanks for the note on 5.0.0 One correction however.... clusters can do a rolling upgrade to 5.0.0 from any 4.2.x level (not just 4.2.3). Today's Topics: 1. FW: Spectrum Scale 5.0 now available on Fix Central (Oesterlin, Robert) ---------------------------------------------------------------------- Message: 1 Date: Mon, 18 Dec 2017 19:43:35 +0000 From: "Oesterlin, Robert" To: gpfsug main discussion list Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central Message-ID: Content-Type: text/plain; charset="utf-8" The Scale 5.0 fix level is now up on Fix Central. You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: "dW-notify at us.ibm.com" Reply-To: "dW-notify at us.ibm.com" Date: Monday, December 18, 2017 at 1:27 PM Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum. [mid:4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75 at SDEB-EXC02.meteo.dz/forums.png] gpfs at us.ibm.com< https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e= > replied to the IBM Spectrum Scale V5.0 announcements< https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e= > topic thread in the General Parallel File System - Announce (GPFS - Announce)< https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0 000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=> forum. IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central: http://www-933.ibm.com/support/fixcentral< https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e= > This topic summarizes changes to the IBM Spectrum Scale licensed program and the IBM Spectrum Scale library. Summary of changes for IBM Spectrum Scale version 5 release 0.0 -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/157250d2/attachment-0001.html > ------------------------------ Message: 2 Date: Mon, 18 Dec 2017 16:10:55 -0500 From: "Frederick Stock" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... Message-ID: Content-Type: text/plain; charset="us-ascii" Yes the integrated protocols are the Samba and Ganesha that are bundled with Spectrum Scale. These require the use of the CES component for monitoring the protocols. If you do use them then you need to wait for a release of Spectrum Scale in which the integrated protocols are also supported on RHEL 7.4. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug-discuss at spectrumscale.org Date: 12/18/2017 03:09 PM Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... Sent by: gpfsug-discuss-bounces at spectrumscale.org Currently, the IBM support matrix says: https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux that 4.2.3.5 is supported on RHEL 7.4, but with a footnote: "AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4." We don't use AFM or the install toolkit. But we *do* make fairly heavy use of mmces and nfs-ganesha - is that what they mean by "Integrated Protocols"? (We're looking at doing upgrades next month while our HPC clusters are doing their upgrades - and going to 7.4 would be nice. If there's a mine field there, I need to make sure we stay at 7.3 - plus applicable non-7.4 updates) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/7c272339/attachment-0001.html > ------------------------------ Message: 3 Date: Mon, 18 Dec 2017 21:46:02 +0000 From: Eric Horst To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... Message-ID: Content-Type: text/plain; charset="utf-8" Grr, this might explain why I experienced unhappiness when I tried to start my long-delayed AFM based migration over the weekend. I had previously tested AFM and found everything working, but 7.4 may have slipped in last month. The AFM relationship seems to work but `mmafmctl premigrate` commands fail. I would revert packages if I could figure out where the issue lies. -Eric On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock wrote: > Yes the integrated protocols are the Samba and Ganesha that are bundled > with Spectrum Scale. These require the use of the CES component for > monitoring the protocols. If you do use them then you need to wait for a > release of Spectrum Scale in which the integrated protocols are also > supported on RHEL 7.4. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821> > stockf at us.ibm.com > > > > From: valdis.kletnieks at vt.edu > To: gpfsug-discuss at spectrumscale.org > Date: 12/18/2017 03:09 PM > Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Currently, the IBM support matrix says: > > https://www.ibm.com/support/knowledgecenter/STXKQY/ > gpfsclustersfaq.html#linux > > that 4.2.3.5 is supported on RHEL 7.4, but with a footnote: > > "AFM, Integrated Protocols, and Installation Toolkit are not supported on > RHEL 7.4." > > We don't use AFM or the install toolkit. But we *do* make fairly heavy use > of mmces and nfs-ganesha - is that what they mean by "Integrated > Protocols"? > > (We're looking at doing upgrades next month while our HPC clusters are > doing > their upgrades - and going to 7.4 would be nice. If there's a mine field > there, I need to > make sure we stay at 7.3 - plus applicable non-7.4 updates) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m= > 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_ > I4NKvn8aFGXCqcJARCbu4AOE&e= > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/c01cc906/attachment.html > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 71, Issue 35 ********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=8ayFP1mF84gza8fbX73eZnWv6MoAQ-ZNctxuN0EsFjQ&s=kEhQxw29VlbFlbUt5Ml-bieqcsRiCv9aTx1cgdDS7Vk&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Mon Jan 8 10:38:14 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 8 Jan 2018 18:38:14 +0800 Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ In-Reply-To: References: Message-ID: This has been asked to Yong Ze to address through another email thread, so ok to ignore this thread request. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "IBM Spectrum Scale" To: "Lyle Gayne" Cc: gpfsug-discuss-bounces at spectrumscale.org, gpfsug main discussion list Date: 01/06/2018 06:46 PM Subject: Re: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Lyle, Can you please forward this to the right folks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Keith Ball To: gpfsug-discuss at spectrumscale.org Date: 12/20/2017 04:39 AM Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was recently trying to determine the latest RHEL release that will work with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code uses Scale 4.1.0.8. A specific X.Y GSS code build, from my experience, is intended to use a specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), which no longer has EUS support from RedHat (only 6.7 still does). GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be used, which is a surprising statement given GSS/ESS code's sensitivity to OS levels (any ESS I have ever seen has never been supported on more than one version of RHEL). According to the Scale FAQ ( https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux ), A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27! Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option here). Many Thanks, Keith_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=u7hLzIRwApuSZgw6uZ_j3uewYEyRTPxRmWJhpM_UAwY&s=bBgN8s6sKOU36_ivUi5NHGarxlF3TNLPyXS-7PA34F0&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=4uHjPqPoMOnTJmSUAO5lPaI-I9TxUZG7tosuUx85sIg&s=tPqgg4TTZ9J88YaLOrAT74IcV_33GWv11K2oywQOGDw&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From lgayne at us.ibm.com Mon Jan 8 14:53:59 2018 From: lgayne at us.ibm.com (Lyle Gayne) Date: Mon, 8 Jan 2018 09:53:59 -0500 Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ In-Reply-To: References: Message-ID: Yong Ze Chen has already been responding to these. Thanks, Lyle From: IBM Spectrum Scale/Poughkeepsie/IBM To: Lyle Gayne/Poughkeepsie/IBM at IBMUS Cc: gpfsug-discuss-bounces at spectrumscale.org, gpfsug main discussion list Date: 01/06/2018 05:46 AM Subject: Re: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: Huzefa H Pancha Hi Lyle, Can you please forward this to the right folks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Keith Ball To: gpfsug-discuss at spectrumscale.org Date: 12/20/2017 04:39 AM Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was recently trying to determine the latest RHEL release that will work with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code uses Scale 4.1.0.8. A specific X.Y GSS code build, from my experience, is intended to use a specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), which no longer has EUS support from RedHat (only 6.7 still does). GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be used, which is a surprising statement given GSS/ESS code's sensitivity to OS levels (any ESS I have ever seen has never been supported on more than one version of RHEL). According to the Scale FAQ ( https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux ), A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27! Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option here). Many Thanks, ? Keith_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=u7hLzIRwApuSZgw6uZ_j3uewYEyRTPxRmWJhpM_UAwY&s=bBgN8s6sKOU36_ivUi5NHGarxlF3TNLPyXS-7PA34F0&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Jan 8 16:52:05 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 8 Jan 2018 16:52:05 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we?d love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again? Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Stephan.Peinkofer at lrz.de Mon Jan 8 21:10:30 2018 From: Stephan.Peinkofer at lrz.de (Peinkofer, Stephan) Date: Mon, 8 Jan 2018 21:10:30 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> , Message-ID: Dear List, my very personal experience today, using the patched kernel for SLES 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS (4.2.3-4) did not even start (the kernel modules seemed to compile fine using mmbuildgpl). Interestingly, even when I disabled PTI explicitely, using the nopti kernel option, GPFS refused to start with the same error!? mmfs.log always showed something like this: ... /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: InsModWrapper[95]: eval: line 1: 3915: Memory fault ... 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading the mmfs kernel extension ... Since I had no time to investigate the issue further and raise a ticket right now, I just downgraded to the previous kernel and everything worked again. As we have to patch at least the login nodes of our HPC clusters asap, I would also appreciate if we could get a statement from IBM how the KPTI patches are expected to interact with GPFS and if there are any (general) problems, when we can expect updated GPFS packages. Many thanks in advance. Best Regards, Stephan Peinkofer ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Buterbaugh, Kevin L Sent: Monday, January 8, 2018 5:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we?d love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again? Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Mon Jan 8 22:57:20 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 8 Jan 2018 17:57:20 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) Message-ID: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> I was thinking some more about the >32 subblock feature in scale 5.0. As mentioned by IBM the biggest advantage of that feature is on filesystems with large blocks (e.g. multiple MB). The majority of our filesystems have a block size of 1MB which got me thinking... wouldn't it be nice if they had a larger block size (there seem to be compelling performance reasons for large file I/O to do this). I'm wondering, what's the feasibility is of supporting filesystem pools of varying block sizes within a single filesystem? I thought allocation maps exist on a per-pool basis which gives me some hope it's not too hard. If one could do this then, yes, you'd still need new hardware to migrate to a larger block size (and >32 subblocks), but it could be done as part of a refresh cycle *and* (and this is the part most important to me) it could be driven entirely by the policy engine which means storage admins are largely hands off and the migration is by and large transparent to the end user. This doesn't really address the need for a tool to address a filesystem migration to 4k metadata blocks (although I wonder if something could be done to create a system_4k pool that contains 4k-aligned metadata NSDs where key data structures get re-written during a restripe in a 4k-aligned manner, but that's really grasping at straws for me). -Aaorn -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From bbanister at jumptrading.com Mon Jan 8 23:48:57 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 8 Jan 2018 23:48:57 +0000 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. Cheers, -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Monday, January 08, 2018 4:57 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) Note: External Email ------------------------------------------------- I was thinking some more about the >32 subblock feature in scale 5.0. As mentioned by IBM the biggest advantage of that feature is on filesystems with large blocks (e.g. multiple MB). The majority of our filesystems have a block size of 1MB which got me thinking... wouldn't it be nice if they had a larger block size (there seem to be compelling performance reasons for large file I/O to do this). I'm wondering, what's the feasibility is of supporting filesystem pools of varying block sizes within a single filesystem? I thought allocation maps exist on a per-pool basis which gives me some hope it's not too hard. If one could do this then, yes, you'd still need new hardware to migrate to a larger block size (and >32 subblocks), but it could be done as part of a refresh cycle *and* (and this is the part most important to me) it could be driven entirely by the policy engine which means storage admins are largely hands off and the migration is by and large transparent to the end user. This doesn't really address the need for a tool to address a filesystem migration to 4k metadata blocks (although I wonder if something could be done to create a system_4k pool that contains 4k-aligned metadata NSDs where key data structures get re-written during a restripe in a 4k-aligned manner, but that's really grasping at straws for me). -Aaorn -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From aaron.s.knister at nasa.gov Tue Jan 9 00:13:40 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 8 Jan 2018 19:13:40 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: Thanks, Bryan! That's a great use case I hadn't thought of. GPFS can already support a different block size for the system pool so in my very simplistic view of the world it's already possible (unless there's some implementation detail about the system pool that lends itself to a different block size from all other pools that doesn't apply to other non-system pools differing from each other). -Aaron On 1/8/18 6:48 PM, Bryan Banister wrote: > Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. > > I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). > > NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. > > Cheers, > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister > Sent: Monday, January 08, 2018 4:57 PM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) > > Note: External Email > ------------------------------------------------- > > I was thinking some more about the >32 subblock feature in scale 5.0. As > mentioned by IBM the biggest advantage of that feature is on filesystems > with large blocks (e.g. multiple MB). The majority of our filesystems > have a block size of 1MB which got me thinking... wouldn't it be nice if > they had a larger block size (there seem to be compelling performance > reasons for large file I/O to do this). > > I'm wondering, what's the feasibility is of supporting filesystem pools > of varying block sizes within a single filesystem? I thought allocation > maps exist on a per-pool basis which gives me some hope it's not too hard. > > If one could do this then, yes, you'd still need new hardware to migrate > to a larger block size (and >32 subblocks), but it could be done as part > of a refresh cycle *and* (and this is the part most important to me) it > could be driven entirely by the policy engine which means storage admins > are largely hands off and the migration is by and large transparent to > the end user. > > This doesn't really address the need for a tool to address a filesystem > migration to 4k metadata blocks (although I wonder if something could be > done to create a system_4k pool that contains 4k-aligned metadata NSDs > where key data structures get re-written during a restripe in a > 4k-aligned manner, but that's really grasping at straws for me). > > -Aaorn > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From Greg.Lehmann at csiro.au Tue Jan 9 03:46:48 2018 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Tue, 9 Jan 2018 03:46:48 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> , Message-ID: This had me wondering, so I tried SLES 12 SP3 and thankfully GPFS v5 still runs after the kernel patch and an mmbuildgpl. It was just a test box I had at the time, so I don't have any comments on performance. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peinkofer, Stephan Sent: Tuesday, 9 January 2018 7:11 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Dear List, my very personal experience today, using the patched kernel for SLES 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS (4.2.3-4) did not even start (the kernel modules seemed to compile fine using mmbuildgpl). Interestingly, even when I disabled PTI explicitely, using the nopti kernel option, GPFS refused to start with the same error!? mmfs.log always showed something like this: ... /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: InsModWrapper[95]: eval: line 1: 3915: Memory fault ... 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading the mmfs kernel extension ... Since I had no time to investigate the issue further and raise a ticket right now, I just downgraded to the previous kernel and everything worked again. As we have to patch at least the login nodes of our HPC clusters asap, I would also appreciate if we could get a statement from IBM how the KPTI patches are expected to interact with GPFS and if there are any (general) problems, when we can expect updated GPFS packages. Many thanks in advance. Best Regards, Stephan Peinkofer ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of Buterbaugh, Kevin L > Sent: Monday, January 8, 2018 5:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we'd love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again... Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS - until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I'm sure that everyone is aware of Meltdown and Spectre by now ... we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I'm sure that everyone is aware of Meltdown and Spectre by now ... we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ... given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we'd be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks... Kevin P.S. The "Happy New Year" wasn't intended as sarcasm ... I hope it is a good year for everyone despite how it's starting out. :-O - Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From heiner.billich at psi.ch Tue Jan 9 08:24:22 2018 From: heiner.billich at psi.ch (Billich Heinrich Rainer (PSI)) Date: Tue, 9 Jan 2018 08:24:22 +0000 Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Message-ID: Hello, I just upgraded to 4.2.3-5 and now see many failures ?ib_rdma_nic_unrecognized? in mmhealth, like Component Status Status Change Reasons ------------------------------------------------------------------------------------------ NETWORK DEGRADED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized(mlx4_0/1) mlx4_0/1 FAILED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized I didn?t see this messages with 4.2.3-4. The relevant lines in /usr/lpp/mmfs/lib/mmsysmon/NetworkService.py changed between -4 and -5. What seems to happen: I have Mellanox VPI cards with one port Infiniband and one port Ethernet. mmhealth complains about the Ethernet port. Hmm ? I did specify the active Infiniband ports only in verbsPorts, I don?t see why mmhealth cares about any other ports when it checks RDMA. So probably a bug, I?ll open a PMR unless somebody points me to a different solution. I tried but I can?t hide this event in mmhealth. Cheers, Heiner -- Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://www.psi.ch From orichards at pixitmedia.com Tue Jan 9 09:09:47 2018 From: orichards at pixitmedia.com (Orlando Richards) Date: Tue, 9 Jan 2018 09:09:47 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: <22812c5d-5013-18b9-7178-7a2c620770dc@pixitmedia.com> Hi all, If it helps - 4.2.3-6 compiles and starts fine on the RHEL 7.4 patched kernel: 3.10.0-693.11.6.el7.x86_64. ______ Orlando On 09/01/2018 03:46, Greg.Lehmann at csiro.au wrote: > > This had me wondering, so I tried SLES 12 SP3 and thankfully GPFS v5 > still runs after the kernel patch and an mmbuildgpl. It was just a > test box I had at the time, so I don?t have any comments on performance. > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Peinkofer, Stephan > *Sent:* Tuesday, 9 January 2018 7:11 AM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS > > Dear List, > > my very?personal?experience today, using the patched kernel for SLES > 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS > (4.2.3-4)?did not even start (the kernel modules seemed to?compile > fine using mmbuildgpl). Interestingly, even when I disabled PTI > explicitely, using the nopti kernel option, GPFS?refused to start with > the same error!? > > mmfs.log always showed something like this: > > ... > > /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: > InsModWrapper[95]: eval: line 1: 3915: Memory fault > > ... > > 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading > the mmfs kernel extension > > ... > > Since I had no time to investigate the issue further and raise a > ticket right now, I just downgraded?to the previous kernel and > everything worked again. > > As we have to patch at least?the login nodes of our HPC clusters asap, > I would also appreciate if we could get a statement from IBM how the > KPTI patches are expected to interact with GPFS and if there are any > (general)?problems, when we can expect updated GPFS packages. > > Many thanks in advance. > Best Regards, > > Stephan Peinkofer > > ------------------------------------------------------------------------ > > *From:*gpfsug-discuss-bounces at spectrumscale.org > > > on behalf of > Buterbaugh, Kevin L > > *Sent:* Monday, January 8, 2018 5:52 PM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS > > Hi GPFS Team, > > Thanks for this response. ?If it is at all possible I know that we > (and I would suspect many others are in this same boat) would greatly > appreciate a update from IBM on how a patched kernel impacts GPFS > functionality. ?Yes, we?d love to know the performance impact of the > patches on GPFS, but that pales in significance to knowing whether > GPFS version 4.x.x.x will even *start* with the patched kernel(s). > > Thanks again? > > Kevin > > > > On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: > > Kevin, > > The team is aware of Meltdown and Spectre. Due to the late > availability of production-ready test patches (they became > available today) we started today working on evaluating the impact > of applying these patches. The focus would be both on any > potential functional impacts (especially to the kernel modules > shipped with GPFS) and on the performance degradation which > affects user/kernel mode transitions. Performance characterization > will be complex, as some system calls which may get invoked often > by the mmfsd daemon will suddenly become significantly more > expensive because of the kernel changes. Depending on the main > areas affected, code changes might be possible to alleviate the > impact, by reducing frequency of certain calls, etc. Any such > changes will be deployed over time. > > At this point, we can't say what impact this will have on > stability or Performance on systems running GPFS ? until IBM > issues an official statement on this topic. We hope to have some > basic answers soon. > > > > Regards, The Spectrum Scale (GPFS) team > > ------------------------------------------------------------------------------------------------------------------ > If you feel that your question can benefit other users of Spectrum > Scale (GPFS), then please post it to the public IBM developerWroks > Forum at > https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 > . > > > If your query concerns a potential software error in Spectrum > Scale (GPFS) and you have an IBM software maintenance contract > please contact 1-800-237-5511 in the United States or your local > IBM Service Center in other countries. > > The forum is informally monitored as time permits and should not > be used for priority messages to the Spectrum Scale (GPFS) team. > > "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 > PM---Happy New Year everyone, I?m sure that everyone is aware of > Meltdown and Spectre by now ? we, like m > > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > Date: 01/04/2018 01:11 PM > Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > ------------------------------------------------------------------------ > > > > > Happy New Year everyone, > > I?m sure that everyone is aware of Meltdown and Spectre by now ? > we, like many other institutions, will be patching for it at the > earliest possible opportunity. > > Our understanding is that the most serious of the negative > performance impacts of these patches will be for things like I/O > (disk / network) ? given that, we are curious if IBM has any plans > for a GPFS update that could help mitigate those impacts? Or is > there simply nothing that can be done? > > If there is a GPFS update planned for this we?d be interested in > knowing so that we could coordinate the kernel and GPFS upgrades > on our cluster. > > Thanks? > > Kevin > > P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it > is a good year for everyone despite how it?s starting out. :-O > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu > - (615)875-9633 > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- *Orlando Richards* VP Product Development, Pixit Media 07930742808|orichards at pixitmedia.com www.pixitmedia.com |Tw:@pixitmedia -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From MDIETZ at de.ibm.com Tue Jan 9 09:43:58 2018 From: MDIETZ at de.ibm.com (Mathias Dietz) Date: Tue, 9 Jan 2018 10:43:58 +0100 Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized In-Reply-To: References: Message-ID: Hello Heiner, with 4.2.3-5 mmhealth is always monitoring all ports of a configured IB adapter even if the port is not specified in verbsPorts. Development has implemented a fix which is planned to be part of 4.2.3-7 (February). To get rid of the false alarm in the meantime you could disable the Infiniband monitoring altogether. To disable Infiniband monitoring on a node: 1. Open the file /var/mmfs/mmsysmon/mmsysmonitor.conf 2. Locate the [network]section 3. Add below: ib_rdma_enable_monitoring=False 4. Save file and run "mmsysmoncontrol restart" If you have questions feel free to contact me directly by email. Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale RAS Architect & Release Lead Architect (4.2.3/5.0) --------------------------------------------------------------------------- IBM Deutschland Am Weiher 24 65451 Kelsterbach Phone: +49 70342744105 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ----------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk WittkoppSitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Billich Heinrich Rainer (PSI)" To: gpfsug main discussion list Date: 01/09/2018 09:31 AM Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I just upgraded to 4.2.3-5 and now see many failures ?ib_rdma_nic_unrecognized? in mmhealth, like Component Status Status Change Reasons ------------------------------------------------------------------------------------------ NETWORK DEGRADED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized(mlx4_0/1) mlx4_0/1 FAILED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized I didn?t see this messages with 4.2.3-4. The relevant lines in /usr/lpp/mmfs/lib/mmsysmon/NetworkService.py changed between -4 and -5. What seems to happen: I have Mellanox VPI cards with one port Infiniband and one port Ethernet. mmhealth complains about the Ethernet port. Hmm ? I did specify the active Infiniband ports only in verbsPorts, I don?t see why mmhealth cares about any other ports when it checks RDMA. So probably a bug, I?ll open a PMR unless somebody points me to a different solution. I tried but I can?t hide this event in mmhealth. Cheers, Heiner -- Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://www.psi.ch _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Tue Jan 9 10:19:49 2018 From: p.childs at qmul.ac.uk (Peter Childs) Date: Tue, 9 Jan 2018 10:19:49 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: <22812c5d-5013-18b9-7178-7a2c620770dc@pixitmedia.com> References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> <22812c5d-5013-18b9-7178-7a2c620770dc@pixitmedia.com> Message-ID: <1515493189.28898.65.camel@qmul.ac.uk> On Tue, 2018-01-09 at 09:09 +0000, Orlando Richards wrote: Hi all, If it helps - 4.2.3-6 compiles and starts fine on the RHEL 7.4 patched kernel: 3.10.0-693.11.6.el7.x86_64. We have 4.2.3-4 working fine on CentOS 7.4 with patched kernel 3.10-0-693.11.6.el7.x86_64. So far we have found CentOs 7.4 (patched and unpatched) to be faster than CentOS 7.3, but CentOs 7.4 patched to be slower than unpatched CentOs 7.4. But I've not seen the details as to this testing so can't really give any further details. Peter ______ Orlando On 09/01/2018 03:46, Greg.Lehmann at csiro.au wrote: This had me wondering, so I tried SLES 12 SP3 and thankfully GPFS v5 still runs after the kernel patch and an mmbuildgpl. It was just a test box I had at the time, so I don?t have any comments on performance. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peinkofer, Stephan Sent: Tuesday, 9 January 2018 7:11 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Dear List, my very personal experience today, using the patched kernel for SLES 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS (4.2.3-4) did not even start (the kernel modules seemed to compile fine using mmbuildgpl). Interestingly, even when I disabled PTI explicitely, using the nopti kernel option, GPFS refused to start with the same error!? mmfs.log always showed something like this: ... /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: InsModWrapper[95]: eval: line 1: 3915: Memory fault ... 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading the mmfs kernel extension ... Since I had no time to investigate the issue further and raise a ticket right now, I just downgraded to the previous kernel and everything worked again. As we have to patch at least the login nodes of our HPC clusters asap, I would also appreciate if we could get a statement from IBM how the KPTI patches are expected to interact with GPFS and if there are any (general) problems, when we can expect updated GPFS packages. Many thanks in advance. Best Regards, Stephan Peinkofer ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of Buterbaugh, Kevin L > Sent: Monday, January 8, 2018 5:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we?d love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again? Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Orlando Richards VP Product Development, Pixit Media 07930742808 | orichards at pixitmedia.com www.pixitmedia.com | Tw:@pixitmedia [http://pixitmedia.com/sig/pixitmedia-9-2018.png] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Peter Childs ITS Research Storage Queen Mary, University of London -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcram at ddn.com Tue Jan 9 15:30:36 2018 From: jcram at ddn.com (Jeno Cram) Date: Tue, 9 Jan 2018 15:30:36 +0000 Subject: [gpfsug-discuss] SMB with LDAP Message-ID: Has anyone had any luck implementing CES with SMB using LDAP? This link doesn?t seem to have all of the information required to getting it working properly. https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_updateldapsmb.htm 1. I?m assuming smbpasswd -a is still required for all users? 2. What keeps the LDAP/SMB passwords in sync? 3. What access does the bind user need in LDAP? 4. What special requirements are required for Windows 10 clients to connect? Jeno Cram | Lead Application Support Engineer jcram at ddn.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Jan 9 15:51:03 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 9 Jan 2018 15:51:03 +0000 Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized In-Reply-To: References: Message-ID: <7828e86d6c494dfdb95a6dbbd6439f09@jumptrading.com> I can't help but comment that it's amazing that GPFS is using a txt config file instead of requiring a command run that stores config data into a non-editable (but still editable) flat file database... Wow 2018!! Hahahahaha! -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mathias Dietz Sent: Tuesday, January 09, 2018 3:44 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Note: External Email ________________________________ Hello Heiner, with 4.2.3-5 mmhealth is always monitoring all ports of a configured IB adapter even if the port is not specified in verbsPorts. Development has implemented a fix which is planned to be part of 4.2.3-7 (February). To get rid of the false alarm in the meantime you could disable the Infiniband monitoring altogether. To disable Infiniband monitoring on a node: 1. Open the file /var/mmfs/mmsysmon/mmsysmonitor.conf 2. Locate the [network]section 3. Add below: ib_rdma_enable_monitoring=False 4. Save file and run "mmsysmoncontrol restart" If you have questions feel free to contact me directly by email. Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale RAS Architect & Release Lead Architect (4.2.3/5.0) --------------------------------------------------------------------------- IBM Deutschland Am Weiher 24 65451 Kelsterbach Phone: +49 70342744105 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ----------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk WittkoppSitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Billich Heinrich Rainer (PSI)" > To: gpfsug main discussion list > Date: 01/09/2018 09:31 AM Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hello, I just upgraded to 4.2.3-5 and now see many failures 'ib_rdma_nic_unrecognized' in mmhealth, like Component Status Status Change Reasons ------------------------------------------------------------------------------------------ NETWORK DEGRADED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized(mlx4_0/1) mlx4_0/1 FAILED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized I didn't see this messages with 4.2.3-4. The relevant lines in /usr/lpp/mmfs/lib/mmsysmon/NetworkService.py changed between -4 and -5. What seems to happen: I have Mellanox VPI cards with one port Infiniband and one port Ethernet. mmhealth complains about the Ethernet port. Hmm - I did specify the active Infiniband ports only in verbsPorts, I don't see why mmhealth cares about any other ports when it checks RDMA. So probably a bug, I'll open a PMR unless somebody points me to a different solution. I tried but I can't hide this event in mmhealth. Cheers, Heiner -- Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://www.psi.ch _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jan 9 22:44:04 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 17:44:04 -0500 Subject: [gpfsug-discuss] interrupted inode expansion Message-ID: I just ran into this in our test environment with 4.2.3.6: http://www-01.ibm.com/support/docview.wss?crawler=1&uid=isg1IV94666 Has anyone else run into this? I'm about to request an efix for it. -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Tue Jan 9 22:47:59 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 17:47:59 -0500 Subject: [gpfsug-discuss] interrupted inode expansion In-Reply-To: References: Message-ID: <0b0daec8-11aa-8d6d-961e-75120520a125@nasa.gov> Question for the Scale team-- is this bug present in the 4.1 stream (or the 5.0 stream for that matter)? On 1/9/18 5:44 PM, Aaron Knister wrote: > I just ran into this in our test environment with 4.2.3.6: > > http://www-01.ibm.com/support/docview.wss?crawler=1&uid=isg1IV94666 > > Has anyone else run into this? I'm about to request an efix for it. > > -Aaron > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From christof.schmitt at us.ibm.com Tue Jan 9 23:34:06 2018 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 9 Jan 2018 23:34:06 +0000 Subject: [gpfsug-discuss] SMB with LDAP In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Wed Jan 10 00:56:08 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 19:56:08 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: <3a1a5b22-76cc-2aa1-7d89-c03619f8d679@nasa.gov> Brian, I stole your wording and created an RFE for this: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115012 -Aaron On 1/8/18 6:48 PM, Bryan Banister wrote: > Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. > > I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). > > NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. > > Cheers, > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister > Sent: Monday, January 08, 2018 4:57 PM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) > > Note: External Email > ------------------------------------------------- > > I was thinking some more about the >32 subblock feature in scale 5.0. As > mentioned by IBM the biggest advantage of that feature is on filesystems > with large blocks (e.g. multiple MB). The majority of our filesystems > have a block size of 1MB which got me thinking... wouldn't it be nice if > they had a larger block size (there seem to be compelling performance > reasons for large file I/O to do this). > > I'm wondering, what's the feasibility is of supporting filesystem pools > of varying block sizes within a single filesystem? I thought allocation > maps exist on a per-pool basis which gives me some hope it's not too hard. > > If one could do this then, yes, you'd still need new hardware to migrate > to a larger block size (and >32 subblocks), but it could be done as part > of a refresh cycle *and* (and this is the part most important to me) it > could be driven entirely by the policy engine which means storage admins > are largely hands off and the migration is by and large transparent to > the end user. > > This doesn't really address the need for a tool to address a filesystem > migration to 4k metadata blocks (although I wonder if something could be > done to create a system_4k pool that contains 4k-aligned metadata NSDs > where key data structures get re-written during a restripe in a > 4k-aligned manner, but that's really grasping at straws for me). > > -Aaorn > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Wed Jan 10 00:57:48 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 19:57:48 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: <3a1a5b22-76cc-2aa1-7d89-c03619f8d679@nasa.gov> References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> <3a1a5b22-76cc-2aa1-7d89-c03619f8d679@nasa.gov> Message-ID: <6a1912f4-d519-f20e-ceb1-af55e2cb7f31@nasa.gov> *Bryan (sorry for the typo) On 1/9/18 7:56 PM, Aaron Knister wrote: > Brian, > > I stole your wording and created an RFE for this: > > http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115012 > > -Aaron > > On 1/8/18 6:48 PM, Bryan Banister wrote: >> Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. >> >> I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). >> >> NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. >> >> Cheers, >> -Bryan >> >> -----Original Message----- >> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister >> Sent: Monday, January 08, 2018 4:57 PM >> To: gpfsug main discussion list >> Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) >> >> Note: External Email >> ------------------------------------------------- >> >> I was thinking some more about the >32 subblock feature in scale 5.0. As >> mentioned by IBM the biggest advantage of that feature is on filesystems >> with large blocks (e.g. multiple MB). The majority of our filesystems >> have a block size of 1MB which got me thinking... wouldn't it be nice if >> they had a larger block size (there seem to be compelling performance >> reasons for large file I/O to do this). >> >> I'm wondering, what's the feasibility is of supporting filesystem pools >> of varying block sizes within a single filesystem? I thought allocation >> maps exist on a per-pool basis which gives me some hope it's not too hard. >> >> If one could do this then, yes, you'd still need new hardware to migrate >> to a larger block size (and >32 subblocks), but it could be done as part >> of a refresh cycle *and* (and this is the part most important to me) it >> could be driven entirely by the policy engine which means storage admins >> are largely hands off and the migration is by and large transparent to >> the end user. >> >> This doesn't really address the need for a tool to address a filesystem >> migration to 4k metadata blocks (although I wonder if something could be >> done to create a system_4k pool that contains 4k-aligned metadata NSDs >> where key data structures get re-written during a restripe in a >> 4k-aligned manner, but that's really grasping at straws for me). >> >> -Aaorn >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ________________________________ >> >> Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Wed Jan 10 02:01:13 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 21:01:13 -0500 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central In-Reply-To: References: Message-ID: <7fe4019d-8034-489a-631b-0c7c1134cce2@nasa.gov> Hi Steve, I can definitely understand the want for a newer compiler, but here's a question for you and the Scale team-- what requires the newer compiler? The userspace tools or the kernel module? I ask because as we all know it can be quite easy to install a modern build chain even on an older operating system or even build binaries on a new operating system that will run on an older operating system. Surely the userspace components could be built with a newer compiler and the gplbin layer would just be built with whatever compiler is present on the system. I suspect there's more complexity and subtlety to the issue but I thought I'd at least ask. I know SLES12 has been out for a while now but that doesn't mean we can just pick up and move to a new OS without some difficulty, especially if the older OS is working just fine :) -Aaron On 12/19/17 8:18 AM, Steve Duersch wrote: > As Mike Taylor pointed out in a previous post this was an incorrect > statement. > You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a > rolling upgrade. > The minReleaseLevel is not pertinent to a rolling upgrade. The running > daemon is the important part. So you can't have any 4.1.x nodes in your > cluster and do a rolling upgrade to 5.0. > > Also, Aaron, as to the OS support. This decision was not made without > some angst. As I mentioned at the user group meeting in NYC...the key > point is that we would like to get to a more current compiler. This will > allow us to take advantage of newer features and functions and hopefully > make the code better for customers. SLES 12 has been around for over 2 > years. > > I hope this helps give some thinking behind the decision. > > > Steve Duersch > Spectrum Scale > 845-433-7902 > IBM Poughkeepsie, New York > > >> Today's Topics: >> >> ? ?1. Re: Spectrum Scale 5.0 now available on Fix Central >> ? ? ? (Sobey, Richard A) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 19 Dec 2017 09:06:08 +0000 >> From: "Sobey, Richard A" >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix >> ? ?Central >> Message-ID: >> ? ? >> > >> ? ? >> Content-Type: text/plain; charset="utf-8" >> >> Hi Robert >> >> Do you mean the minReleaseLevel from mmlsconfig or just making sure >> all the nodes are running 4.2.3? >> >> Cheers! >> Richard >> >> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug- >> discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert >> Sent: 18 December 2017 19:44 >> To: gpfsug main discussion list >> Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix > Central >> >> The Scale 5.0 fix level is now up on Fix Central. >> >> You need to be at Scale 4.2.3 (cluster level) to do a rolling >> upgrade to this level. >> >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From Robert.Oesterlin at nuance.com Wed Jan 10 02:12:16 2018 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 10 Jan 2018 02:12:16 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central Message-ID: I'm in the same boat here with deprecated support for RH6. We have a number of older applications supporting product that won't run on RH7. I'm still pondering what I'm going to do here. Or stay perpetually stuck on 4.2.3 on some clusters. Bob Oesterlin Sr Principal Storage Engineer, Nuance ?On 1/9/18, 8:02 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: I know SLES12 has been out for a while now but that doesn't mean we can just pick up and move to a new OS without some difficulty, especially if the older OS is working just fine :) From aaron.s.knister at nasa.gov Wed Jan 10 02:32:24 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 21:32:24 -0500 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central In-Reply-To: References: Message-ID: I just opened an RFE for RHEL6, SLES11 and Debian support in Scale 5.0: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115019 -Aaron On 1/9/18 9:12 PM, Oesterlin, Robert wrote: > I'm in the same boat here with deprecated support for RH6. We have a number of older applications supporting product that won't run on RH7. I'm still pondering what I'm going to do here. Or stay perpetually stuck on 4.2.3 on some clusters. > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > ?On 1/9/18, 8:02 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: > > I know SLES12 has been out for a while now but that doesn't mean we can > just pick up and move to a new OS without some difficulty, especially if > the older OS is working just fine :) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From sandeep.patil at in.ibm.com Wed Jan 10 06:43:11 2018 From: sandeep.patil at in.ibm.com (Sandeep Ramesh) Date: Wed, 10 Jan 2018 12:13:11 +0530 Subject: [gpfsug-discuss] Latest Technical Blogs on Spectrum Scale In-Reply-To: References: Message-ID: Dear User Group Members, Here are list of development blogs in the last quarter. Passing it to this email group as Doris had got a feedback in the UG meetings to notify the members with the latest updates periodically. Genomic Workloads ? How To Get it Right From Infrastructure Point Of View. https://developer.ibm.com/storage/2018/01/06/genomic-workloads-get-right-infrastructure-point-view/ IBM Spectrum Scale Versus Apache Hadoop HDFS https://developer.ibm.com/storage/2018/01/10/spectrumscale_vs_hdfs/ ESS Fault Tolerance https://developer.ibm.com/storage/2018/01/09/ess-fault-tolerance/ IBM Spectrum Scale MMFSCK ? Savvy Enhancements https://developer.ibm.com/storage/2018/01/05/ibm-spectrum-scale-mmfsck-savvy-enhancements/ ESS Disk Management https://developer.ibm.com/storage/2018/01/02/ess-disk-management/ IBM Spectrum Scale Object Protocol On Ubuntu https://developer.ibm.com/storage/2018/01/01/ibm-spectrum-scale-object-protocol-ubuntu/ IBM Spectrum Scale 5.0 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2017/12/20/ibm-spectrum-scale-5-0-whats-new-object/ A Complete Guide to ? Protocol Problem Determination Guide for IBM Spectrum Scale? ? Part 1 https://developer.ibm.com/storage/2017/12/19/complete-guide-protocol-problem-determination-guide-ibm-spectrum-scale-1/ IBM Spectrum Scale installation toolkit ? enhancements over releases https://developer.ibm.com/storage/2017/12/15/ibm-spectrum-scale-installation-toolkit-enhancements-releases/ Network requirements in an Elastic Storage Server Setup https://developer.ibm.com/storage/2017/12/13/network-requirements-in-an-elastic-storage-server-setup/ Co-resident migration with Transparent cloud tierin https://developer.ibm.com/storage/2017/12/05/co-resident-migration-transparent-cloud-tierin/ IBM Spectrum Scale on Hortonworks HDP Hadoop clusters : A Complete Big Data Solution https://developer.ibm.com/storage/2017/12/05/ibm-spectrum-scale-hortonworks-hdp-hadoop-clusters-complete-big-data-solution/ Big data analytics with Spectrum Scale using remote cluster mount & multi-filesystem support https://developer.ibm.com/storage/2017/11/28/big-data-analytics-spectrum-scale-using-remote-cluster-mount-multi-filesystem-support/ IBM Spectrum Scale HDFS Transparency Short Circuit Write Support https://developer.ibm.com/storage/2017/11/28/ibm-spectrum-scale-hdfs-transparency-short-circuit-write-support/ IBM Spectrum Scale HDFS Transparency Federation Support https://developer.ibm.com/storage/2017/11/27/ibm-spectrum-scale-hdfs-transparency-federation-support/ How to configure and performance tuning different system workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-different-system-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Spark workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-spark-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning database workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-database-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Hadoop workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/24/configure-performance-tuning-hadoop-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ IBM Spectrum Scale Sharing Nothing Cluster Performance Tuning https://developer.ibm.com/storage/2017/11/24/ibm-spectrum-scale-sharing-nothing-cluster-performance-tuning/ How to Configure IBM Spectrum Scale? with NIS based Authentication. https://developer.ibm.com/storage/2017/11/21/configure-ibm-spectrum-scale-nis-based-authentication/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Cc: Doris Conti/Poughkeepsie/IBM at IBMUS Date: 11/16/2017 08:15 PM Subject: Latest Technical Blogs on Spectrum Scale Dear User Group members, Here are the Development Blogs in last 3 months on Spectrum Scale Technical Topics. Spectrum Scale Monitoring ? Know More ? https://developer.ibm.com/storage/2017/11/16/spectrum-scale-monitoring-know/ IBM Spectrum Scale 5.0 Release ? What?s coming ! https://developer.ibm.com/storage/2017/11/14/ibm-spectrum-scale-5-0-release-whats-coming/ Four Essentials things to know for managing data ACLs on IBM Spectrum Scale? from Windows https://developer.ibm.com/storage/2017/11/13/four-essentials-things-know-managing-data-acls-ibm-spectrum-scale-windows/ GSSUTILS: A new way of running SSR, Deploying or Upgrading ESS Server https://developer.ibm.com/storage/2017/11/13/gssutils/ IBM Spectrum Scale Object Authentication https://developer.ibm.com/storage/2017/11/02/spectrum-scale-object-authentication/ Video Surveillance ? Choosing the right storage https://developer.ibm.com/storage/2017/11/02/video-surveillance-choosing-right-storage/ IBM Spectrum scale object deep dive training with problem determination https://www.slideshare.net/SmitaRaut/ibm-spectrum-scale-object-deep-dive-training Spectrum Scale as preferred software defined storage for Ubuntu OpenStack https://developer.ibm.com/storage/2017/09/29/spectrum-scale-preferred-software-defined-storage-ubuntu-openstack/ IBM Elastic Storage Server 2U24 Storage ? an All-Flash offering, a performance workhorse https://developer.ibm.com/storage/2017/10/06/ess-5-2-flash-storage/ A Complete Guide to Configure LDAP-based authentication with IBM Spectrum Scale? for File Access https://developer.ibm.com/storage/2017/09/21/complete-guide-configure-ldap-based-authentication-ibm-spectrum-scale-file-access/ Deploying IBM Spectrum Scale on AWS Quick Start https://developer.ibm.com/storage/2017/09/18/deploy-ibm-spectrum-scale-on-aws-quick-start/ Monitoring Spectrum Scale Object metrics https://developer.ibm.com/storage/2017/09/14/monitoring-spectrum-scale-object-metrics/ Tier your data with ease to Spectrum Scale Private Cloud(s) using Moonwalk Universal https://developer.ibm.com/storage/2017/09/14/tier-data-ease-spectrum-scale-private-clouds-using-moonwalk-universal/ Why do I see owner as ?Nobody? for my export mounted using NFSV4 Protocol on IBM Spectrum Scale?? https://developer.ibm.com/storage/2017/09/08/see-owner-nobody-export-mounted-using-nfsv4-protocol-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory and LDAP https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-ldap/ IBM Spectrum Scale? Authentication using Active Directory and RFC2307 https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-rfc2307/ High Availability Implementation with IBM Spectrum Virtualize and IBM Spectrum Scale https://developer.ibm.com/storage/2017/08/30/high-availability-implementation-ibm-spectrum-virtualize-ibm-spectrum-scale/ 10 Frequently asked Questions on configuring Authentication using AD + AUTO ID mapping on IBM Spectrum Scale?. https://developer.ibm.com/storage/2017/08/04/10-frequently-asked-questions-configuring-authentication-using-ad-auto-id-mapping-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory https://developer.ibm.com/storage/2017/07/30/ibm-spectrum-scale-auth-using-active-directory/ Five cool things that you didn?t know Transparent Cloud Tiering on Spectrum Scale can do https://developer.ibm.com/storage/2017/07/29/five-cool-things-didnt-know-transparent-cloud-tiering-spectrum-scale-can/ IBM Spectrum Scale GUI videos https://developer.ibm.com/storage/2017/07/25/ibm-spectrum-scale-gui-videos/ IBM Spectrum Scale? Authentication ? Planning for NFS Access https://developer.ibm.com/storage/2017/07/24/ibm-spectrum-scale-planning-nfs-access/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Wed Jan 10 14:32:09 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 10 Jan 2018 14:32:09 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central In-Reply-To: References: Message-ID: <089336cb58b6443e98d28bd4d6b16724@jumptrading.com> I felt some sort of obligation to stand by my own wording... +1 vote for me!!! -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Tuesday, January 09, 2018 8:32 PM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central Note: External Email ------------------------------------------------- I just opened an RFE for RHEL6, SLES11 and Debian support in Scale 5.0: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115019 -Aaron On 1/9/18 9:12 PM, Oesterlin, Robert wrote: > I'm in the same boat here with deprecated support for RH6. We have a number of older applications supporting product that won't run on RH7. I'm still pondering what I'm going to do here. Or stay perpetually stuck on 4.2.3 on some clusters. > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > ?On 1/9/18, 8:02 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: > > I know SLES12 has been out for a while now but that doesn't mean we can > just pick up and move to a new OS without some difficulty, especially if > the older OS is working just fine :) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From Achim.Rehor at de.ibm.com Wed Jan 10 15:57:48 2018 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Wed, 10 Jan 2018 16:57:48 +0100 Subject: [gpfsug-discuss] interrupted inode expansion In-Reply-To: <0b0daec8-11aa-8d6d-961e-75120520a125@nasa.gov> References: <0b0daec8-11aa-8d6d-961e-75120520a125@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: From vpuvvada at in.ibm.com Fri Jan 12 06:50:49 2018 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Fri, 12 Jan 2018 12:20:49 +0530 Subject: [gpfsug-discuss] Use AFM for migration of many small files In-Reply-To: <1515064616.28898.32.camel@qmul.ac.uk> References: <467FB293-D33B-46F4-BA1B-A5CB4D28DDE6@psi.ch> <1515064616.28898.32.camel@qmul.ac.uk> Message-ID: >The plan is to load the new cache from the old GPFS then dump once the cache is full. >We've already increase numThreashThreads from 4 to 8 and seen only marginal improvements, we could attempt to increase this further. AFM have replication performance issues with small files on high latency networks. There is a plan to fix these issues. >I'm also wondering if its worth increasing the Refresh Intervals to speed up read of already cache files. At this stage we want to fill the cache and don't care about write back until we switch the target to the >new NFS/GPFS from our old GPFS storage to a new box back at our off-site location, (otherwise known as the office) Increasing the refresh intervals will improve the application performance at cache site. It is better to set large refresh intervals if the cache is the only writer. ~Venkat (vpuvvada at in.ibm.com) From: Peter Childs To: "gpfsug-discuss at spectrumscale.org" Date: 01/04/2018 04:47 PM Subject: Re: [gpfsug-discuss] Use AFM for migration of many small files Sent by: gpfsug-discuss-bounces at spectrumscale.org We are doing something very similar using 4.2.3-4 and GPFS 4.2.1-1 on the nfs backend. Did you have any success? The plan is to load the new cache from the old GPFS then dump once the cache is full. We've already increase numThreashThreads from 4 to 8 and seen only marginal improvements, we could attempt to increase this further. I'm also wondering if its worth increasing the Refresh Intervals to speed up read of already cache files. At this stage we want to fill the cache and don't care about write back until we switch the target to the new NFS/GPFS from our old GPFS storage to a new box back at our off-site location, (otherwise known as the office) [root at afmgateway1 scratch]# mmlsfileset home home -L --afm Filesets in file system 'home': Attributes for fileset home: ============================= Status Linked Path /data2/home Id 42 Root inode 1343225859 Parent Id 0 Created Wed Jan 3 12:32:33 2018 Comment Inode space 41 Maximum number of inodes 100000000 Allocated inodes 15468544 Permission change flag chmodAndSetacl afm-associated Yes Target nfs://afm1/gpfs/data1/afm/home Mode single-writer File Lookup Refresh Interval 30 (default) File Open Refresh Interval 30 (default) Dir Lookup Refresh Interval 60 (default) Dir Open Refresh Interval 60 (default) Async Delay 15 (default) Last pSnapId 0 Display Home Snapshots no Number of Gateway Flush Threads 8 Prefetch Threshold 0 (default) Eviction Enabled no Thanks in advance. Peter Childs On Tue, 2017-09-05 at 19:57 +0530, Venkateswara R Puvvada wrote: Which version of Spectrum Scale ? What is the fileset mode ? >We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here >I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. How was the performance measured ? If parallel IO is enabled, AFM uses multiple gateway nodes to prefetch the large files (if file size if more than 1GB). Performance difference between small and lager file is huge (1000MB - 150MB = 850MB) here, and generally it is not the case. How many files were present in list file for prefetch ? Could you also share full internaldump from the gateway node ? >I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few >read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. AFM prefetches the files on multiple threads. Default flush threads for prefetch are 36 (fileset.afmNumFlushThreads (default 4) + afmNumIOFlushThreads (default 32)). >Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and >I need to look elsewhere to get better performance for prefetch of many smaller files? See above, AFM reads files on multiple threads parallelly. Try increasing the afmNumFlushThreads on fileset and verify if it improves the performance. ~Venkat (vpuvvada at in.ibm.com) From: "Billich Heinrich Rainer (PSI)" To: "gpfsug-discuss at spectrumscale.org" Date: 09/04/2017 10:18 PM Subject: [gpfsug-discuss] Use AFM for migration of many small files Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and I need to look elsewhere to get better performance for prefetch of many smaller files? We will migrate several filesets in parallel, but still with individual filesets up to 350TB in size 150MB/s isn?t fun. Also just about 150 files/s seconds looks poor. The setup is quite new, hence there may be other places to look at. It?s all RHEL7 an spectrum scale 4.2.2-3 on the afm cache. Thank you, Heiner --, Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://urldefense.proofpoint.com/v2/url?u=https-3A__www.psi.ch&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=eHcVdovN10-m-Qk0Ln2qvol3pkKNFwrzz2wgf1zXVXE&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=LbRyuSM_djs0FDXr27hPottQHAn3OGcivpyRcIDBN3U&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=eMSpfXQgE3wMSVM0G0vCYr6LEgERhP8iGfw3uNk8lLs&s=mcQ13uhvwS4yPbA2uCmwccc7l4mjTmL2fAdPLimS0Hc&e= -- Peter Childs ITS Research Storage Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=07QQkI0Rg8NyUEgPIuJwfg3elEXqTpOjIFpy2WbaEg0&s=kGEDPbMo64yU7Tcwu61ggT89tfq_3QdX-r6NoANXh78&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From coetzee.ray at gmail.com Fri Jan 12 10:29:43 2018 From: coetzee.ray at gmail.com (Ray Coetzee) Date: Fri, 12 Jan 2018 10:29:43 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale Message-ID: I'd like to ask the group of their experiences in improving the performance of applications that use mmap calls against files on Spectrum Scale. Besides using an NFS export from CES instead of a native GPFS mount, or precaching the dataset into the pagepool, what other approaches are there to offset the performance hit of the 4K IO size? Kind regards Ray Coetzee -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Jan 12 14:45:29 2018 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 12 Jan 2018 14:45:29 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: Message-ID: what version of Scale are you using right now ? On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee wrote: > I'd like to ask the group of their experiences in improving the > performance of applications that use mmap calls against files on Spectrum > Scale. > > Besides using an NFS export from CES instead of a native GPFS mount, or > precaching the dataset into the pagepool, what other approaches are there > to offset the performance hit of the 4K IO size? > > Kind regards > > Ray Coetzee > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Thu Jan 11 14:16:26 2018 From: john.hearns at asml.com (John Hearns) Date: Thu, 11 Jan 2018 14:16:26 +0000 Subject: [gpfsug-discuss] System running out of memory - SUnreclaim is huge Message-ID: I am having problems with GPFS servers running out of memory. We have an open PMR for this, however if anyone has seen this or has any ideas I would be grateful for a heads up. Servers have 128 Gbytes f RAM, kernel 2.6.32-573.18.1.el6.x86_64, GPFS version 4.2.3.4 In the latest incident the free memory went to below 1Gbyte, and we started to have processes killed, including our monitoring setup. I shut down GPFS on that server and /proc/meminfo still shows: Slab: 111192296 kB SReclaimable: 29020 kB SUnreclaim: 111163276 kB Am I barking up a wrong tree here and pointing the finger at GPFS? Something is causing the scsi_data_buffer slab memory usage (see below). One thing I did yesterday was to change the disk scheduler for each disk from cfq to dealine (as recommended in the tuning guide) However the server was already in short memory at that point. Slabtop shows Active / Total Objects (% used) : -306803185 / -306722574 (100.0%) Active / Total Slabs (% used) : 27749714 / 27749719 (100.0%) Active / Total Caches (% used) : 115 / 198 (58.1%) Active / Total Size (% used) : 93857848.58K / 93872319.47K (100.0%) Minimum / Average / Maximum Object : 0.02K / 0.02K / 4096.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 3987822096 3987821817 0% 0.02K 27693209 144 110772836K scsi_data_buffer 91155 64448 70% 0.06K 1545 59 6180K size-64 36064 32035 88% 0.03K 322 112 1288K size-32 35505 34334 96% 0.25K 2367 15 9468K skbuff_head_cache 33876 33874 99% 8.00K 33876 1 271008K size-8192 33804 33615 99% 0.14K 1252 27 5008K sysfs_dir_cache -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Fri Jan 12 16:12:13 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 12 Jan 2018 16:12:13 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: Message-ID: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> You could put all of your data onto SSDs in a RAID1 configuration so that you don?t have insane read-modify-write penalties on writes (RAID1) and avoid horrible seek thrashing that spinning rust requires (SSD random access medium) for your 4K I/O operations. One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not to touch it?. He gave many reasons why mmap operations on a distributed file system is incredibly hard and not recommended. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sven Oehme Sent: Friday, January 12, 2018 8:45 AM To: coetzee.ray at gmail.com; gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmap performance against Spectrum Scale Note: External Email ________________________________ what version of Scale are you using right now ? On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee > wrote: I'd like to ask the group of their experiences in improving the performance of applications that use mmap calls against files on Spectrum Scale. Besides using an NFS export from CES instead of a native GPFS mount, or precaching the dataset into the pagepool, what other approaches are there to offset the performance hit of the 4K IO size? Kind regards Ray Coetzee _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From coetzee.ray at gmail.com Fri Jan 12 20:51:13 2018 From: coetzee.ray at gmail.com (Ray Coetzee) Date: Fri, 12 Jan 2018 20:51:13 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. (Without the meltdown patch) Hey Bryan, I remember that quote from Yuri, that's why I hoped some "magic" may have been done since then. Other attempts to improve performance I've tried include: - Using LROC to have a larger chance of a cache hit (Unfortunately the entire dataset is multiple TB) - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for this purpose (Job runs halved in duration but nowhere near what NFS can give) - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and some others with little improvement. For those interested, as a performance comparison. The same job when run on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe scratch filesystem and over 60min on spindle based filesystem. Kind regards Ray Coetzee Email: coetzee.ray at gmail.com On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister wrote: > You could put all of your data onto SSDs in a RAID1 configuration so that > you don?t have insane read-modify-write penalties on writes (RAID1) and > avoid horrible seek thrashing that spinning rust requires (SSD random > access medium) for your 4K I/O operations. > > > > One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not > to touch it?. He gave many reasons why mmap operations on a distributed > file system is incredibly hard and not recommended. > > -Bryan > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org] *On Behalf Of *Sven Oehme > *Sent:* Friday, January 12, 2018 8:45 AM > *To:* coetzee.ray at gmail.com; gpfsug main discussion list < > gpfsug-discuss at spectrumscale.org> > *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale > > > > *Note: External Email* > ------------------------------ > > what version of Scale are you using right now ? > > > > On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee wrote: > > I'd like to ask the group of their experiences in improving the > performance of applications that use mmap calls against files on Spectrum > Scale. > > > > Besides using an NFS export from CES instead of a native GPFS mount, or > precaching the dataset into the pagepool, what other approaches are there > to offset the performance hit of the 4K IO size? > > > > Kind regards > > Ray Coetzee > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at calicolabs.com Fri Jan 12 20:56:00 2018 From: alex at calicolabs.com (Alex Chekholko) Date: Fri, 12 Jan 2018 12:56:00 -0800 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: Hi Ray, Sounds like you are better off hosting that workflow on the other storage system. Another thing I have done in the past is work with the software maintainer to replace all mmap() with lseek(). YMMV. I have previously told users that mmap "doesn't work" on GPFS. As I understand it, programmers use mmap to try to improve performance but here it has the opposite effect. Regards, Alex On Fri, Jan 12, 2018 at 12:51 PM, Ray Coetzee wrote: > Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. > (Without the meltdown patch) > > Hey Bryan, I remember that quote from Yuri, that's why I hoped some > "magic" may have been done since then. > > Other attempts to improve performance I've tried include: > > - Using LROC to have a larger chance of a cache hit (Unfortunately the > entire dataset is multiple TB) > - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for > this purpose (Job runs halved in duration but nowhere near what NFS can > give) > - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and > some others with little improvement. > > For those interested, as a performance comparison. The same job when run > on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe > scratch filesystem and over 60min on spindle based filesystem. > > Kind regards > > Ray Coetzee > Email: coetzee.ray at gmail.com > > > On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister > wrote: > >> You could put all of your data onto SSDs in a RAID1 configuration so that >> you don?t have insane read-modify-write penalties on writes (RAID1) and >> avoid horrible seek thrashing that spinning rust requires (SSD random >> access medium) for your 4K I/O operations. >> >> >> >> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not >> to touch it?. He gave many reasons why mmap operations on a distributed >> file system is incredibly hard and not recommended. >> >> -Bryan >> >> >> >> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: >> gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >> *Sent:* Friday, January 12, 2018 8:45 AM >> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >> gpfsug-discuss at spectrumscale.org> >> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >> >> >> >> *Note: External Email* >> ------------------------------ >> >> what version of Scale are you using right now ? >> >> >> >> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >> wrote: >> >> I'd like to ask the group of their experiences in improving the >> performance of applications that use mmap calls against files on Spectrum >> Scale. >> >> >> >> Besides using an NFS export from CES instead of a native GPFS mount, or >> precaching the dataset into the pagepool, what other approaches are there >> to offset the performance hit of the 4K IO size? >> >> >> >> Kind regards >> >> Ray Coetzee >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ------------------------------ >> >> Note: This email is for the confidential use of the named addressee(s) >> only and may contain proprietary, confidential or privileged information. >> If you are not the intended recipient, you are hereby notified that any >> review, dissemination or copying of this email is strictly prohibited, and >> to please notify the sender immediately and destroy this email and any >> attachments. Email transmission cannot be guaranteed to be secure or >> error-free. The Company, therefore, does not make any guarantees as to the >> completeness or accuracy of this email or any attachments. This email is >> for informational purposes only and does not constitute a recommendation, >> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >> or perform any type of transaction of a financial product. >> > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Jan 12 20:57:24 2018 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 12 Jan 2018 20:57:24 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: is this primary read or write ? On Fri, Jan 12, 2018, 12:51 PM Ray Coetzee wrote: > Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. > (Without the meltdown patch) > > Hey Bryan, I remember that quote from Yuri, that's why I hoped some > "magic" may have been done since then. > > Other attempts to improve performance I've tried include: > > - Using LROC to have a larger chance of a cache hit (Unfortunately the > entire dataset is multiple TB) > - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for > this purpose (Job runs halved in duration but nowhere near what NFS can > give) > - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and > some others with little improvement. > > For those interested, as a performance comparison. The same job when run > on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe > scratch filesystem and over 60min on spindle based filesystem. > > Kind regards > > Ray Coetzee > Email: coetzee.ray at gmail.com > > > On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister > wrote: > >> You could put all of your data onto SSDs in a RAID1 configuration so that >> you don?t have insane read-modify-write penalties on writes (RAID1) and >> avoid horrible seek thrashing that spinning rust requires (SSD random >> access medium) for your 4K I/O operations. >> >> >> >> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not >> to touch it?. He gave many reasons why mmap operations on a distributed >> file system is incredibly hard and not recommended. >> >> -Bryan >> >> >> >> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: >> gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >> *Sent:* Friday, January 12, 2018 8:45 AM >> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >> gpfsug-discuss at spectrumscale.org> >> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >> >> >> >> *Note: External Email* >> ------------------------------ >> >> what version of Scale are you using right now ? >> >> >> >> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >> wrote: >> >> I'd like to ask the group of their experiences in improving the >> performance of applications that use mmap calls against files on Spectrum >> Scale. >> >> >> >> Besides using an NFS export from CES instead of a native GPFS mount, or >> precaching the dataset into the pagepool, what other approaches are there >> to offset the performance hit of the 4K IO size? >> >> >> >> Kind regards >> >> Ray Coetzee >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ------------------------------ >> >> Note: This email is for the confidential use of the named addressee(s) >> only and may contain proprietary, confidential or privileged information. >> If you are not the intended recipient, you are hereby notified that any >> review, dissemination or copying of this email is strictly prohibited, and >> to please notify the sender immediately and destroy this email and any >> attachments. Email transmission cannot be guaranteed to be secure or >> error-free. The Company, therefore, does not make any guarantees as to the >> completeness or accuracy of this email or any attachments. This email is >> for informational purposes only and does not constitute a recommendation, >> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >> or perform any type of transaction of a financial product. >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jan 12 22:58:39 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 12 Jan 2018 19:58:39 -0300 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: Having multiple blocksizes in the same file system would unnecessarily complicate things. Consider migrating a file from one pool to another with different blocksizes... How to represent the indirect blocks (lists of blocks allocated to the file)? Especially consider that today, migration can proceed one block at a time, during migration a file is "mis-placed" -- has blocks spread over more than one pool.... The new feature that supports more than 32 sub-blocks per block - is a step in another direction but maybe addresses some of your concerns.... We do support different blocksizes for meta-data -- but meta-data is distinct from data and never migrates out of system pool. --marc K. From: Aaron Knister To: Date: 01/08/2018 09:25 PM Subject: Re: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) Sent by: gpfsug-discuss-bounces at spectrumscale.org Thanks, Bryan! That's a great use case I hadn't thought of. GPFS can already support a different block size for the system pool so in my very simplistic view of the world it's already possible (unless there's some implementation detail about the system pool that lends itself to a different block size from all other pools that doesn't apply to other non-system pools differing from each other). -Aaron On 1/8/18 6:48 PM, Bryan Banister wrote: > Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. > > I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). > > NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. > > Cheers, > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister > Sent: Monday, January 08, 2018 4:57 PM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) > > Note: External Email > ------------------------------------------------- > > I was thinking some more about the >32 subblock feature in scale 5.0. As > mentioned by IBM the biggest advantage of that feature is on filesystems > with large blocks (e.g. multiple MB). The majority of our filesystems > have a block size of 1MB which got me thinking... wouldn't it be nice if > they had a larger block size (there seem to be compelling performance > reasons for large file I/O to do this). > > I'm wondering, what's the feasibility is of supporting filesystem pools > of varying block sizes within a single filesystem? I thought allocation > maps exist on a per-pool basis which gives me some hope it's not too hard. > > If one could do this then, yes, you'd still need new hardware to migrate > to a larger block size (and >32 subblocks), but it could be done as part > of a refresh cycle *and* (and this is the part most important to me) it > could be driven entirely by the policy engine which means storage admins > are largely hands off and the migration is by and large transparent to > the end user. > > This doesn't really address the need for a tool to address a filesystem > migration to 4k metadata blocks (although I wonder if something could be > done to create a system_4k pool that contains 4k-aligned metadata NSDs > where key data structures get re-written during a restripe in a > 4k-aligned manner, but that's really grasping at straws for me). > > -Aaorn > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=zfQZ_ymVgGc2EseA0szLiRxH-FgYnw7qMdx2qKo3Zes&s=2KsLTkZ-3MRyIMQhp8WwTn524NpfiKv8gwTy4P36xX4&e= > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=zfQZ_ymVgGc2EseA0szLiRxH-FgYnw7qMdx2qKo3Zes&s=2KsLTkZ-3MRyIMQhp8WwTn524NpfiKv8gwTy4P36xX4&e= > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=zfQZ_ymVgGc2EseA0szLiRxH-FgYnw7qMdx2qKo3Zes&s=2KsLTkZ-3MRyIMQhp8WwTn524NpfiKv8gwTy4P36xX4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Jan 12 23:14:52 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 12 Jan 2018 18:14:52 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm Message-ID: Apologies if this has been covered elsewhere (I couldn't find it if it has). I'm curious how GPFS decides where to allocate new blocks. We've got a filesystem that we added some NSDs to a while back and it hurt there for a little while because it appeared as though GPFS was choosing to allocate new blocks much more frequently on the ~100% free LUNs than the existing LUNs (I can't recall how free they were at the time). Looking at it now, though, it seems GPFS is doing the opposite. There's now a ~10% difference between the LUNs added and the existing LUNs (20% free vs 30% free) and GPFS is choosing to allocate new writes at a ratio of about 3:1 on the disks with *fewer* free blocks than on the disks with more free blocks. That's completely inconsistent with what we saw when we initially added the disks which makes me wonder how GPFS is choosing to allocate new blocks (other than the obvious bits about failure group, and replication factor). Could someone explain (or point me at a whitepaper) what factors GPFS uses when allocating blocks, particularly as it pertains to choosing one NSD over another within the same failure group. Thanks! -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From janfrode at tanso.net Sat Jan 13 09:24:54 2018 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sat, 13 Jan 2018 09:24:54 +0000 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: Message-ID: Don?t have documentation/whitepaper, but as I recall, it will first allocate round-robin over failureGroup, then round-robin over nsdServers, and then round-robin over volumes. So if these new NSDs are defined as different failureGroup from the old disks, that might explain it.. -jf l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister : > Apologies if this has been covered elsewhere (I couldn't find it if it > has). I'm curious how GPFS decides where to allocate new blocks. > > We've got a filesystem that we added some NSDs to a while back and it > hurt there for a little while because it appeared as though GPFS was > choosing to allocate new blocks much more frequently on the ~100% free > LUNs than the existing LUNs (I can't recall how free they were at the > time). Looking at it now, though, it seems GPFS is doing the opposite. > There's now a ~10% difference between the LUNs added and the existing > LUNs (20% free vs 30% free) and GPFS is choosing to allocate new writes > at a ratio of about 3:1 on the disks with *fewer* free blocks than on > the disks with more free blocks. That's completely inconsistent with > what we saw when we initially added the disks which makes me wonder how > GPFS is choosing to allocate new blocks (other than the obvious bits > about failure group, and replication factor). Could someone explain (or > point me at a whitepaper) what factors GPFS uses when allocating blocks, > particularly as it pertains to choosing one NSD over another within the > same failure group. > > Thanks! > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Sat Jan 13 13:18:36 2018 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Sat, 13 Jan 2018 13:18:36 +0000 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Jan 13 16:18:25 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 13 Jan 2018 11:18:25 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: Message-ID: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> Thanks Everyone! I whipped up a script to dump the block layout of a file and then join that with mmdf information. As part of my exploration I wrote one 2GB file to each of this particular filesystem's 4 data pools last night (using "touch $file; mmchattr $file -P $pool; dd of=$file") and have attached a dump of the layout/nsd information for each file/pool. The fields for the output are: diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, freePct, freeKbFragments, freeKbFragmentsPct Here's the highlight from pool1: 36 264 d13_06_006 23437934592 1213 4548935680 (19%) 83304320 (0%) 59 74 d10_41_025 23437934592 1011 6993759232 (30%) 58642816 (0%) For this file (And anecdotally what I've seen looking at NSD I/O data for other files written to this pool) the pattern of more blocks being allocated to the NSDs that are ~20% free vs the NSDs that are 30% free seems to be fairly consistent. Looking at a snippet of pool2: 101 238 d15_15_011 23437934592 1415 2008394752 (9%) 181699328 (1%) 102 235 d15_16_012 23437934592 1415 2009153536 (9%) 182165312 (1%) 116 248 d11_42_026 23437934592 1011 4146111488 (18%) 134941504 (1%) 117 249 d13_42_026 23437934592 1213 4147710976 (18%) 135203776 (1%) there are slightly more blocks allocated in general on the NSDs with twice the amount of free space, but it doesn't seem to be a significant amount relative to the delta in free space. The pattern from pool1 certainly doesn't hold true here. Pool4 isn't very interesting because all of the NSDs are well balanced in terms of free space (all 16% free). Pool3, however, *is* particularly interesting. Here's a snippet: 106 222 d15_24_016 23437934592 1415 2957561856 (13%) 37436768 (0%) 107 222 d15_25_017 23437934592 1415 2957537280 (13%) 37353984 (0%) 108 222 d15_26_018 23437934592 1415 2957539328 (13%) 37335872 (0%) 125 222 d11_44_028 23437934592 1011 13297235968 (57%) 20505568 (0%) 126 222 d12_44_028 23437934592 1213 13296712704 (57%) 20632768 (0%) 127 222 d12_45_029 23437934592 1213 13297404928 (57%) 20557408 (0%) GPFS consistently allocated the same number of blocks to disks with ~4x the free space than it did the other disks in the pool. Suffice it to say-- I'm *very* confused :) -Aaron On 1/13/18 8:18 AM, Daniel Kidger wrote: > Aaron, > ? > Also are your new NSDs the same size as your existing ones? > i.e. the NSDs that are at a?higher %age full might have more free blocks > than the other NSDs? > Daniel > > ? > IBM Storage Professional Badge > > ? > ? > *Dr Daniel Kidger* > IBM?Technical Sales Specialist > Software Defined Solution Sales > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > ? > ? > ? > > ----- Original message ----- > From: Jan-Frode Myklebust > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: Re: [gpfsug-discuss] pool block allocation algorithm > Date: Sat, Jan 13, 2018 9:25 AM > ? > Don?t have documentation/whitepaper, but as I recall, it will first > allocate round-robin over failureGroup, then round-robin over > nsdServers, and then round-robin over volumes. So if these new NSDs > are defined as different failureGroup from the old disks, that might > explain it.. > > > -jf > l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister > >: > > Apologies if this has been covered elsewhere (I couldn't find it > if it > has). I'm curious how GPFS decides where to allocate new blocks. > > We've got a filesystem that we added some NSDs to a while back > and it > hurt there for a little while because it appeared as though GPFS was > choosing to allocate new blocks much more frequently on the > ~100% free > LUNs than the existing LUNs (I can't recall how free they were > at the > time). Looking at it now, though, it seems GPFS is doing the > opposite. > There's now a ~10% difference between the LUNs added and the > existing > LUNs (20% free vs 30% free) and GPFS is choosing to allocate new > writes > at a ratio of about 3:1 on the disks with *fewer* free blocks > than on > the disks with more free blocks. That's completely inconsistent with > what we saw when we initially added the disks which makes me > wonder how > GPFS is choosing to allocate new blocks (other than the obvious bits > about failure group, and replication factor). Could someone > explain (or > point me at a whitepaper) what factors GPFS uses when allocating > blocks, > particularly as it pertains to choosing one NSD over another > within the > same failure group. > > Thanks! > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= > > ? > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- 1 221 d12_01_001 23437934592 1213 4550865920 (19%) 83734976 (0%) 2 266 d12_02_002 23437934592 1213 4550282240 (19%) 83712000 (0%) 3 265 d12_03_003 23437934592 1213 4550419456 (19%) 83986432 (0%) 4 265 d12_04_004 23437934592 1213 4550207488 (19%) 83382624 (0%) 5 266 d12_05_005 23437934592 1213 4551135232 (19%) 83820832 (0%) 6 264 d12_06_006 23437934592 1213 4549501952 (19%) 83537984 (0%) 31 256 d13_01_001 23437934592 1213 4550247424 (19%) 83557184 (0%) 32 266 d13_02_002 23437934592 1213 4548240384 (19%) 83289888 (0%) 33 264 d13_03_003 23437934592 1213 4548679680 (19%) 83851360 (0%) 34 266 d13_04_004 23437934592 1213 4551052288 (19%) 83524320 (0%) 35 265 d13_05_005 23437934592 1213 4550079488 (19%) 83578624 (0%) 36 264 d13_06_006 23437934592 1213 4548935680 (19%) 83304320 (0%) 59 74 d10_41_025 23437934592 1011 6993759232 (30%) 58642816 (0%) 61 216 d14_01_001 23437934592 1415 4575688704 (20%) 83600032 (0%) 62 266 d14_02_002 23437934592 1415 4574487552 (20%) 83376960 (0%) 63 265 d14_03_003 23437934592 1415 4574819328 (20%) 83378240 (0%) 64 266 d14_04_004 23437934592 1415 4575392768 (20%) 83162080 (0%) 65 264 d14_05_005 23437934592 1415 4576001024 (20%) 83447968 (0%) 66 265 d14_06_006 23437934592 1415 4575699968 (20%) 83043040 (0%) 85 72 d11_41_025 23437934592 1011 6991225856 (30%) 58576768 (0%) 86 66 d12_41_025 23437934592 1213 6992339968 (30%) 58402688 (0%) 87 71 d12_42_026 23437934592 1213 6992146432 (30%) 58284032 (0%) 88 67 d13_41_025 23437934592 1213 6993167360 (30%) 58134624 (0%) 89 75 d14_41_025 23437934592 1415 6992531456 (30%) 58169600 (0%) 90 71 d14_42_026 23437934592 1415 6993435648 (30%) 58234720 (0%) 91 265 d15_01_001 23437934592 1415 4575676416 (20%) 83394144 (0%) 92 264 d15_02_002 23437934592 1415 4576221184 (20%) 82999168 (0%) 93 264 d15_03_003 23437934592 1415 4576231424 (20%) 83179680 (0%) 94 266 d15_04_004 23437934592 1415 4577104896 (20%) 83445088 (0%) 95 265 d15_05_005 23437934592 1415 4576627712 (20%) 82928064 (0%) 96 259 d15_06_006 23437934592 1415 4576740352 (20%) 83355776 (0%) 115 73 d15_41_025 23437934592 1415 6990783488 (30%) 58595296 (0%) 145 266 d10_01_001 23437934592 1011 4588489728 (20%) 83100448 (0%) 146 266 d10_02_002 23437934592 1011 4587276288 (20%) 83239488 (0%) 147 265 d10_03_003 23437934592 1011 4586872832 (20%) 83024000 (0%) 148 264 d10_04_004 23437934592 1011 4585822208 (20%) 82897920 (0%) 149 264 d10_05_005 23437934592 1011 4586415104 (20%) 83277056 (0%) 150 266 d10_06_006 23437934592 1011 4587424768 (20%) 83076544 (0%) 151 265 d11_01_001 23437934592 1011 4586209280 (20%) 83178368 (0%) 152 265 d11_02_002 23437934592 1011 4587372544 (20%) 83091232 (0%) 153 264 d11_03_003 23437934592 1011 4587196416 (20%) 83147040 (0%) 154 264 d11_04_004 23437934592 1011 4586633216 (20%) 83049024 (0%) 155 265 d11_05_005 23437934592 1011 4586576896 (20%) 83188768 (0%) 156 264 d11_06_006 23437934592 1011 4587261952 (20%) 83393952 (0%) -------------- next part -------------- 7 238 d12_11_007 23437934592 1213 2005351424 (9%) 184092352 (1%) 8 237 d12_12_008 23437934592 1213 2005532672 (9%) 183893472 (1%) 9 243 d12_13_009 23437934592 1213 2004583424 (9%) 183000704 (1%) 10 239 d12_14_010 23437934592 1213 2004461568 (9%) 182958048 (1%) 11 228 d12_15_011 23437934592 1213 2004750336 (9%) 183160064 (1%) 12 240 d12_16_012 23437934592 1213 2008772608 (9%) 183263936 (1%) 37 228 d13_11_007 23437934592 1213 2006091776 (9%) 183041024 (1%) 38 230 d13_12_008 23437934592 1213 2003761152 (9%) 182625024 (1%) 39 229 d13_13_009 23437934592 1213 2004123648 (9%) 182907552 (1%) 40 208 d13_14_010 23437934592 1213 2006372352 (9%) 182797376 (1%) 41 233 d13_15_011 23437934592 1213 2003358720 (9%) 182322880 (1%) 42 232 d13_16_012 23437934592 1213 2004774912 (9%) 182729792 (1%) 67 234 d14_11_007 23437934592 1415 2006553600 (9%) 182736800 (1%) 68 249 d14_12_008 23437934592 1415 2010278912 (9%) 182304832 (1%) 69 226 d14_13_009 23437934592 1415 2008749056 (9%) 182854464 (1%) 70 237 d14_14_010 23437934592 1415 2009348096 (9%) 182257024 (1%) 71 249 d14_15_011 23437934592 1415 2008017920 (9%) 182217152 (1%) 72 233 d14_16_012 23437934592 1415 2008158208 (9%) 182239680 (1%) 97 237 d15_11_007 23437934592 1415 2007257088 (9%) 181898304 (1%) 98 233 d15_12_008 23437934592 1415 2008309760 (9%) 182000928 (1%) 99 229 d15_13_009 23437934592 1415 2008755200 (9%) 182380544 (1%) 100 224 d15_14_010 23437934592 1415 2008460288 (9%) 181691616 (1%) 101 238 d15_15_011 23437934592 1415 2008394752 (9%) 181699328 (1%) 102 235 d15_16_012 23437934592 1415 2009153536 (9%) 182165312 (1%) 116 248 d11_42_026 23437934592 1011 4146111488 (18%) 134941504 (1%) 117 249 d13_42_026 23437934592 1213 4147710976 (18%) 135203776 (1%) 118 248 d15_42_026 23437934592 1415 4147094528 (18%) 134748320 (1%) 119 249 d10_43_027 23437934592 1011 4148652032 (18%) 135124288 (1%) 120 247 d11_43_027 23437934592 1011 4147335168 (18%) 134808256 (1%) 121 250 d12_43_027 23437934592 1213 4147177472 (18%) 134812096 (1%) 122 248 d13_43_027 23437934592 1213 4147270656 (18%) 134516192 (1%) 123 248 d14_43_027 23437934592 1415 4148153344 (18%) 134928896 (1%) 124 248 d15_43_027 23437934592 1415 4149500928 (18%) 134818880 (1%) 157 247 d10_11_007 23437934592 1011 2008516608 (9%) 182482592 (1%) 159 248 d10_13_009 23437934592 1011 2009745408 (9%) 182393184 (1%) 161 233 d10_15_011 23437934592 1011 2007539712 (9%) 182169856 (1%) 163 234 d11_11_007 23437934592 1011 2009283584 (9%) 182173824 (1%) 164 244 d11_12_008 23437934592 1011 2009091072 (9%) 182099200 (1%) 165 248 d11_13_009 23437934592 1011 2008827904 (9%) 182029632 (1%) 166 231 d11_14_010 23437934592 1011 2010264576 (9%) 181675424 (1%) 167 235 d11_15_011 23437934592 1011 2010696704 (9%) 181854304 (1%) 168 236 d11_16_012 23437934592 1011 2008183808 (9%) 182236800 (1%) -------------- next part -------------- 13 222 d12_21_013 23437934592 1213 2957768704 (13%) 37317824 (0%) 14 223 d12_22_014 23437934592 1213 2957680640 (13%) 37608576 (0%) 15 223 d12_23_015 23437934592 1213 2958575616 (13%) 37380672 (0%) 16 223 d12_24_016 23437934592 1213 2957406208 (13%) 37403648 (0%) 17 223 d12_25_017 23437934592 1213 2957883392 (13%) 37558016 (0%) 18 222 d12_26_018 23437934592 1213 2957815808 (13%) 37556416 (0%) 43 222 d13_21_013 23437934592 1213 2957149184 (13%) 37745280 (0%) 44 222 d13_22_014 23437934592 1213 2957812736 (13%) 37520736 (0%) 45 222 d13_23_015 23437934592 1213 2957990912 (13%) 37341952 (0%) 46 222 d13_24_016 23437934592 1213 2958204928 (13%) 37477536 (0%) 47 222 d13_25_017 23437934592 1213 2958315520 (13%) 37507072 (0%) 48 222 d13_26_018 23437934592 1213 2958086144 (13%) 37407648 (0%) 73 222 d14_21_013 23437934592 1415 2957812736 (13%) 37631488 (0%) 74 223 d14_22_014 23437934592 1415 2959673344 (13%) 37427680 (0%) 75 223 d14_23_015 23437934592 1415 2957627392 (13%) 37400992 (0%) 76 223 d14_24_016 23437934592 1415 2957408256 (13%) 37235776 (0%) 77 222 d14_25_017 23437934592 1415 2957407232 (13%) 37499200 (0%) 78 222 d14_26_018 23437934592 1415 2956661760 (13%) 37633696 (0%) 103 222 d15_21_013 23437934592 1415 2957427712 (13%) 37629664 (0%) 104 222 d15_22_014 23437934592 1415 2957506560 (13%) 37571968 (0%) 105 222 d15_23_015 23437934592 1415 2958114816 (13%) 37422400 (0%) 106 222 d15_24_016 23437934592 1415 2957561856 (13%) 37436768 (0%) 107 222 d15_25_017 23437934592 1415 2957537280 (13%) 37353984 (0%) 108 222 d15_26_018 23437934592 1415 2957539328 (13%) 37335872 (0%) 125 222 d11_44_028 23437934592 1011 13297235968 (57%) 20505568 (0%) 126 222 d12_44_028 23437934592 1213 13296712704 (57%) 20632768 (0%) 127 222 d12_45_029 23437934592 1213 13297404928 (57%) 20557408 (0%) 128 222 d13_44_028 23437934592 1213 13297271808 (57%) 20502304 (0%) 129 222 d14_44_028 23437934592 1415 13294722048 (57%) 20569824 (0%) 130 222 d14_45_029 23437934592 1415 13296071680 (57%) 20667104 (0%) 131 222 d15_44_028 23437934592 1415 13297007616 (57%) 20429536 (0%) 132 223 d10_44_028 23437934592 1011 13309036544 (57%) 20299552 (0%) 133 223 d10_45_029 23437934592 1011 13309213696 (57%) 20464576 (0%) 169 223 d10_21_013 23437934592 1011 2797943808 (12%) 36881600 (0%) 170 222 d10_22_014 23437934592 1011 2798030848 (12%) 37152256 (0%) 171 222 d10_23_015 23437934592 1011 2798152704 (12%) 36959840 (0%) 172 222 d10_24_016 23437934592 1011 2798002176 (12%) 36954976 (0%) 173 222 d10_25_017 23437934592 1011 2798524416 (12%) 36737120 (0%) 174 222 d10_26_018 23437934592 1011 2797768704 (12%) 36888096 (0%) 175 222 d11_21_013 23437934592 1011 2958944256 (13%) 37504288 (0%) 176 222 d11_22_014 23437934592 1011 2957225984 (13%) 37575648 (0%) 177 222 d11_23_015 23437934592 1011 2958100480 (13%) 37546336 (0%) 178 222 d11_24_016 23437934592 1011 2958453760 (13%) 37578784 (0%) 179 222 d11_25_017 23437934592 1011 2958053376 (13%) 37331808 (0%) 180 222 d11_26_018 23437934592 1011 2958290944 (13%) 37445824 (0%) -------------- next part -------------- 19 223 d12_31_019 23437934592 1213 3803873280 (16%) 117281408 (1%) 20 222 d12_32_020 23437934592 1213 3779237888 (16%) 116652576 (0%) 21 222 d12_33_021 23437934592 1213 3779206144 (16%) 116820288 (0%) 22 222 d12_34_022 23437934592 1213 3776828416 (16%) 116108032 (0%) 23 222 d12_35_023 23437934592 1213 3776950272 (16%) 116238752 (0%) 24 222 d12_36_024 23437934592 1213 3776524288 (16%) 116599744 (0%) 49 222 d13_31_019 23437934592 1213 3778293760 (16%) 116365600 (0%) 50 222 d13_32_020 23437934592 1213 3778381824 (16%) 115960448 (0%) 51 222 d13_33_021 23437934592 1213 3777104896 (16%) 116225984 (0%) 52 222 d13_34_022 23437934592 1213 3775632384 (16%) 116496576 (0%) 53 222 d13_35_023 23437934592 1213 3776899072 (16%) 116397120 (0%) 54 222 d13_36_024 23437934592 1213 3776140288 (16%) 116400000 (0%) 79 222 d14_31_019 23437934592 1415 3844638720 (16%) 113531904 (0%) 80 222 d14_32_020 23437934592 1415 3817139200 (16%) 113600928 (0%) 81 222 d14_33_021 23437934592 1415 3815447552 (16%) 113358272 (0%) 82 222 d14_34_022 23437934592 1415 3814561792 (16%) 113348160 (0%) 83 222 d14_35_023 23437934592 1415 3815387136 (16%) 113582432 (0%) 84 222 d14_36_024 23437934592 1415 3815303168 (16%) 112914816 (0%) 109 222 d15_31_019 23437934592 1415 3814614016 (16%) 113298240 (0%) 110 222 d15_32_020 23437934592 1415 3814936576 (16%) 113053664 (0%) 111 222 d15_33_021 23437934592 1415 3815122944 (16%) 113438656 (0%) 112 222 d15_34_022 23437934592 1415 3813474304 (16%) 113481216 (0%) 113 222 d15_35_023 23437934592 1415 3812912128 (16%) 113618624 (0%) 114 222 d15_36_024 23437934592 1415 3813067776 (16%) 113282848 (0%) 134 223 d11_45_029 23437934592 1011 3683404800 (16%) 100860000 (0%) 135 222 d13_45_029 23437934592 1213 3682211840 (16%) 101101824 (0%) 136 223 d15_45_029 23437934592 1415 3682046976 (16%) 101375328 (0%) 137 222 d10_46_030 23437934592 1011 3684398080 (16%) 100946912 (0%) 138 222 d11_46_030 23437934592 1011 3683755008 (16%) 101295200 (0%) 139 223 d12_46_030 23437934592 1213 3682649088 (16%) 100540832 (0%) 140 223 d13_46_030 23437934592 1213 3684602880 (16%) 100489376 (0%) 141 223 d14_46_030 23437934592 1415 3681666048 (16%) 100854112 (0%) 142 223 d15_46_030 23437934592 1415 3685487616 (16%) 100715744 (0%) 181 222 d10_31_019 23437934592 1011 3816681472 (16%) 113389344 (0%) 182 222 d10_32_020 23437934592 1011 3816535040 (16%) 113652672 (0%) 183 222 d10_33_021 23437934592 1011 3817772032 (16%) 113124224 (0%) 184 222 d10_34_022 23437934592 1011 3817069568 (16%) 113225504 (0%) 185 222 d10_35_023 23437934592 1011 3816517632 (16%) 113163488 (0%) 186 222 d10_36_024 23437934592 1011 3815605248 (16%) 113475648 (0%) 187 222 d11_31_019 23437934592 1011 3815664640 (16%) 113100992 (0%) 188 222 d11_32_020 23437934592 1011 3815985152 (16%) 113325504 (0%) 189 222 d11_33_021 23437934592 1011 3815148544 (16%) 113328480 (0%) 190 223 d11_34_022 23437934592 1011 3814780928 (16%) 113457440 (0%) 191 223 d11_35_023 23437934592 1011 3815219200 (16%) 113065504 (0%) 192 223 d11_36_024 23437934592 1011 3815383040 (16%) 113170592 (0%) From aaron.s.knister at nasa.gov Sat Jan 13 16:56:52 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 13 Jan 2018 11:56:52 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> Message-ID: Sorry, I didn't explicitly say it, but the output I sent should answer both Daniel's and Jan-Frode's questions. In short, though, the new NSDs were added to existing failure groups and they should all be (except in one or two cases where we re-formatted the LUN and the size changed slightly) the same size. -Aaron On 1/13/18 11:18 AM, Aaron Knister wrote: > Thanks Everyone! I whipped up a script to dump the block layout of a > file and then join that with mmdf information. As part of my exploration > I wrote one 2GB file to each of this particular filesystem's 4 data > pools last night (using "touch $file; mmchattr $file -P $pool; dd > of=$file") and have attached a dump of the layout/nsd information for > each file/pool. The fields for the output are: > > diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, > freePct, freeKbFragments, freeKbFragmentsPct > > > Here's the highlight from pool1: > > 36 264 d13_06_006 23437934592 1213 4548935680 (19%) > 83304320 (0%) > 59 74 d10_41_025 23437934592 1011 6993759232 (30%) > 58642816 (0%) > > For this file (And anecdotally what I've seen looking at NSD I/O data > for other files written to this pool) the pattern of more blocks being > allocated to the NSDs that are ~20% free vs the NSDs that are 30% free > seems to be fairly consistent. > > > Looking at a snippet of pool2: > 101 238 d15_15_011 23437934592 1415 2008394752 (9%) > 181699328 (1%) > 102 235 d15_16_012 23437934592 1415 2009153536 (9%) > 182165312 (1%) > 116 248 d11_42_026 23437934592 1011 4146111488 (18%) > 134941504 (1%) > 117 249 d13_42_026 23437934592 1213 4147710976 (18%) > 135203776 (1%) > > there are slightly more blocks allocated in general on the NSDs with > twice the amount of free space, but it doesn't seem to be a significant > amount relative to the delta in free space. The pattern from pool1 > certainly doesn't hold true here. > > Pool4 isn't very interesting because all of the NSDs are well balanced > in terms of free space (all 16% free). > > Pool3, however, *is* particularly interesting. Here's a snippet: > > 106 222 d15_24_016 23437934592 1415 2957561856 (13%) > 37436768 (0%) > 107 222 d15_25_017 23437934592 1415 2957537280 (13%) > 37353984 (0%) > 108 222 d15_26_018 23437934592 1415 2957539328 (13%) > 37335872 (0%) > 125 222 d11_44_028 23437934592 1011 13297235968 (57%) > 20505568 (0%) > 126 222 d12_44_028 23437934592 1213 13296712704 (57%) > 20632768 (0%) > 127 222 d12_45_029 23437934592 1213 13297404928 (57%) > 20557408 (0%) > > GPFS consistently allocated the same number of blocks to disks with ~4x > the free space than it did the other disks in the pool. > > Suffice it to say-- I'm *very* confused :) > > -Aaron > > On 1/13/18 8:18 AM, Daniel Kidger wrote: >> Aaron, >> ? >> Also are your new NSDs the same size as your existing ones? >> i.e. the NSDs that are at a?higher %age full might have more free blocks >> than the other NSDs? >> Daniel >> >> ? >> IBM Storage Professional Badge >> >> ? >> ? >> *Dr Daniel Kidger* >> IBM?Technical Sales Specialist >> Software Defined Solution Sales >> >> +44-(0)7818 522 266 >> daniel.kidger at uk.ibm.com >> >> ? >> ? >> ? >> >> ----- Original message ----- >> From: Jan-Frode Myklebust >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug main discussion list >> Cc: >> Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> Date: Sat, Jan 13, 2018 9:25 AM >> ? >> Don?t have documentation/whitepaper, but as I recall, it will first >> allocate round-robin over failureGroup, then round-robin over >> nsdServers, and then round-robin over volumes. So if these new NSDs >> are defined as different failureGroup from the old disks, that might >> explain it.. >> >> >> -jf >> l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >> >: >> >> Apologies if this has been covered elsewhere (I couldn't find it >> if it >> has). I'm curious how GPFS decides where to allocate new blocks. >> >> We've got a filesystem that we added some NSDs to a while back >> and it >> hurt there for a little while because it appeared as though GPFS was >> choosing to allocate new blocks much more frequently on the >> ~100% free >> LUNs than the existing LUNs (I can't recall how free they were >> at the >> time). Looking at it now, though, it seems GPFS is doing the >> opposite. >> There's now a ~10% difference between the LUNs added and the >> existing >> LUNs (20% free vs 30% free) and GPFS is choosing to allocate new >> writes >> at a ratio of about 3:1 on the disks with *fewer* free blocks >> than on >> the disks with more free blocks. That's completely inconsistent with >> what we saw when we initially added the disks which makes me >> wonder how >> GPFS is choosing to allocate new blocks (other than the obvious bits >> about failure group, and replication factor). Could someone >> explain (or >> point me at a whitepaper) what factors GPFS uses when allocating >> blocks, >> particularly as it pertains to choosing one NSD over another >> within the >> same failure group. >> >> Thanks! >> >> -Aaron >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= >> >> ? >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From peserocka at gmail.com Sat Jan 13 17:00:49 2018 From: peserocka at gmail.com (Peter Serocka) Date: Sat, 13 Jan 2018 18:00:49 +0100 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> Message-ID: <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Within reasonable capacity limits it would also expect to direct incoming data to disks that are best ?available? from a current performance perspective ? like doing least IOPS, having lowest latency and shortest filled queue length. You new NSDs, filled only with recent data, might quickly have become the most busy units before reaching capacity balance, simply because recent data tends to be more active than older stuff. Makes sense? ? Peter > On 2018 Jan 13 Sat, at 17:18, Aaron Knister wrote: > > Thanks Everyone! I whipped up a script to dump the block layout of a > file and then join that with mmdf information. As part of my exploration > I wrote one 2GB file to each of this particular filesystem's 4 data > pools last night (using "touch $file; mmchattr $file -P $pool; dd > of=$file") and have attached a dump of the layout/nsd information for > each file/pool. The fields for the output are: > > diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, > freePct, freeKbFragments, freeKbFragmentsPct > > > Here's the highlight from pool1: > > 36 264 d13_06_006 23437934592 1213 4548935680 (19%) > 83304320 (0%) > 59 74 d10_41_025 23437934592 1011 6993759232 (30%) > 58642816 (0%) > > For this file (And anecdotally what I've seen looking at NSD I/O data > for other files written to this pool) the pattern of more blocks being > allocated to the NSDs that are ~20% free vs the NSDs that are 30% free > seems to be fairly consistent. > > > Looking at a snippet of pool2: > 101 238 d15_15_011 23437934592 1415 2008394752 (9%) > 181699328 (1%) > 102 235 d15_16_012 23437934592 1415 2009153536 (9%) > 182165312 (1%) > 116 248 d11_42_026 23437934592 1011 4146111488 (18%) > 134941504 (1%) > 117 249 d13_42_026 23437934592 1213 4147710976 (18%) > 135203776 (1%) > > there are slightly more blocks allocated in general on the NSDs with > twice the amount of free space, but it doesn't seem to be a significant > amount relative to the delta in free space. The pattern from pool1 > certainly doesn't hold true here. > > Pool4 isn't very interesting because all of the NSDs are well balanced > in terms of free space (all 16% free). > > Pool3, however, *is* particularly interesting. Here's a snippet: > > 106 222 d15_24_016 23437934592 1415 2957561856 (13%) > 37436768 (0%) > 107 222 d15_25_017 23437934592 1415 2957537280 (13%) > 37353984 (0%) > 108 222 d15_26_018 23437934592 1415 2957539328 (13%) > 37335872 (0%) > 125 222 d11_44_028 23437934592 1011 13297235968 (57%) > 20505568 (0%) > 126 222 d12_44_028 23437934592 1213 13296712704 (57%) > 20632768 (0%) > 127 222 d12_45_029 23437934592 1213 13297404928 (57%) > 20557408 (0%) > > GPFS consistently allocated the same number of blocks to disks with ~4x > the free space than it did the other disks in the pool. > > Suffice it to say-- I'm *very* confused :) > > -Aaron > > On 1/13/18 8:18 AM, Daniel Kidger wrote: >> Aaron, >> >> Also are your new NSDs the same size as your existing ones? >> i.e. the NSDs that are at a higher %age full might have more free blocks >> than the other NSDs? >> Daniel >> >> >> IBM Storage Professional Badge >> >> >> >> *Dr Daniel Kidger* >> IBM Technical Sales Specialist >> Software Defined Solution Sales >> >> +44-(0)7818 522 266 >> daniel.kidger at uk.ibm.com >> >> >> >> >> >> ----- Original message ----- >> From: Jan-Frode Myklebust >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug main discussion list >> Cc: >> Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> Date: Sat, Jan 13, 2018 9:25 AM >> >> Don?t have documentation/whitepaper, but as I recall, it will first >> allocate round-robin over failureGroup, then round-robin over >> nsdServers, and then round-robin over volumes. So if these new NSDs >> are defined as different failureGroup from the old disks, that might >> explain it.. >> >> >> -jf >> l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >> >: >> >> Apologies if this has been covered elsewhere (I couldn't find it >> if it >> has). I'm curious how GPFS decides where to allocate new blocks. >> >> We've got a filesystem that we added some NSDs to a while back >> and it >> hurt there for a little while because it appeared as though GPFS was >> choosing to allocate new blocks much more frequently on the >> ~100% free >> LUNs than the existing LUNs (I can't recall how free they were >> at the >> time). Looking at it now, though, it seems GPFS is doing the >> opposite. >> There's now a ~10% difference between the LUNs added and the >> existing >> LUNs (20% free vs 30% free) and GPFS is choosing to allocate new >> writes >> at a ratio of about 3:1 on the disks with *fewer* free blocks >> than on >> the disks with more free blocks. That's completely inconsistent with >> what we saw when we initially added the disks which makes me >> wonder how >> GPFS is choosing to allocate new blocks (other than the obvious bits >> about failure group, and replication factor). Could someone >> explain (or >> point me at a whitepaper) what factors GPFS uses when allocating >> blocks, >> particularly as it pertains to choosing one NSD over another >> within the >> same failure group. >> >> Thanks! >> >> -Aaron >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= >> >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 -------------- next part -------------- 1 221 d12_01_001 23437934592 1213 4550865920 (19%) 83734976 (0%) 2 266 d12_02_002 23437934592 1213 4550282240 (19%) 83712000 (0%) 3 265 d12_03_003 23437934592 1213 4550419456 (19%) 83986432 (0%) 4 265 d12_04_004 23437934592 1213 4550207488 (19%) 83382624 (0%) 5 266 d12_05_005 23437934592 1213 4551135232 (19%) 83820832 (0%) 6 264 d12_06_006 23437934592 1213 4549501952 (19%) 83537984 (0%) 31 256 d13_01_001 23437934592 1213 4550247424 (19%) 83557184 (0%) 32 266 d13_02_002 23437934592 1213 4548240384 (19%) 83289888 (0%) 33 264 d13_03_003 23437934592 1213 4548679680 (19%) 83851360 (0%) 34 266 d13_04_004 23437934592 1213 4551052288 (19%) 83524320 (0%) 35 265 d13_05_005 23437934592 1213 4550079488 (19%) 83578624 (0%) 36 264 d13_06_006 23437934592 1213 4548935680 (19%) 83304320 (0%) 59 74 d10_41_025 23437934592 1011 6993759232 (30%) 58642816 (0%) 61 216 d14_01_001 23437934592 1415 4575688704 (20%) 83600032 (0%) 62 266 d14_02_002 23437934592 1415 4574487552 (20%) 83376960 (0%) 63 265 d14_03_003 23437934592 1415 4574819328 (20%) 83378240 (0%) 64 266 d14_04_004 23437934592 1415 4575392768 (20%) 83162080 (0%) 65 264 d14_05_005 23437934592 1415 4576001024 (20%) 83447968 (0%) 66 265 d14_06_006 23437934592 1415 4575699968 (20%) 83043040 (0%) 85 72 d11_41_025 23437934592 1011 6991225856 (30%) 58576768 (0%) 86 66 d12_41_025 23437934592 1213 6992339968 (30%) 58402688 (0%) 87 71 d12_42_026 23437934592 1213 6992146432 (30%) 58284032 (0%) 88 67 d13_41_025 23437934592 1213 6993167360 (30%) 58134624 (0%) 89 75 d14_41_025 23437934592 1415 6992531456 (30%) 58169600 (0%) 90 71 d14_42_026 23437934592 1415 6993435648 (30%) 58234720 (0%) 91 265 d15_01_001 23437934592 1415 4575676416 (20%) 83394144 (0%) 92 264 d15_02_002 23437934592 1415 4576221184 (20%) 82999168 (0%) 93 264 d15_03_003 23437934592 1415 4576231424 (20%) 83179680 (0%) 94 266 d15_04_004 23437934592 1415 4577104896 (20%) 83445088 (0%) 95 265 d15_05_005 23437934592 1415 4576627712 (20%) 82928064 (0%) 96 259 d15_06_006 23437934592 1415 4576740352 (20%) 83355776 (0%) 115 73 d15_41_025 23437934592 1415 6990783488 (30%) 58595296 (0%) 145 266 d10_01_001 23437934592 1011 4588489728 (20%) 83100448 (0%) 146 266 d10_02_002 23437934592 1011 4587276288 (20%) 83239488 (0%) 147 265 d10_03_003 23437934592 1011 4586872832 (20%) 83024000 (0%) 148 264 d10_04_004 23437934592 1011 4585822208 (20%) 82897920 (0%) 149 264 d10_05_005 23437934592 1011 4586415104 (20%) 83277056 (0%) 150 266 d10_06_006 23437934592 1011 4587424768 (20%) 83076544 (0%) 151 265 d11_01_001 23437934592 1011 4586209280 (20%) 83178368 (0%) 152 265 d11_02_002 23437934592 1011 4587372544 (20%) 83091232 (0%) 153 264 d11_03_003 23437934592 1011 4587196416 (20%) 83147040 (0%) 154 264 d11_04_004 23437934592 1011 4586633216 (20%) 83049024 (0%) 155 265 d11_05_005 23437934592 1011 4586576896 (20%) 83188768 (0%) 156 264 d11_06_006 23437934592 1011 4587261952 (20%) 83393952 (0%) -------------- next part -------------- 7 238 d12_11_007 23437934592 1213 2005351424 (9%) 184092352 (1%) 8 237 d12_12_008 23437934592 1213 2005532672 (9%) 183893472 (1%) 9 243 d12_13_009 23437934592 1213 2004583424 (9%) 183000704 (1%) 10 239 d12_14_010 23437934592 1213 2004461568 (9%) 182958048 (1%) 11 228 d12_15_011 23437934592 1213 2004750336 (9%) 183160064 (1%) 12 240 d12_16_012 23437934592 1213 2008772608 (9%) 183263936 (1%) 37 228 d13_11_007 23437934592 1213 2006091776 (9%) 183041024 (1%) 38 230 d13_12_008 23437934592 1213 2003761152 (9%) 182625024 (1%) 39 229 d13_13_009 23437934592 1213 2004123648 (9%) 182907552 (1%) 40 208 d13_14_010 23437934592 1213 2006372352 (9%) 182797376 (1%) 41 233 d13_15_011 23437934592 1213 2003358720 (9%) 182322880 (1%) 42 232 d13_16_012 23437934592 1213 2004774912 (9%) 182729792 (1%) 67 234 d14_11_007 23437934592 1415 2006553600 (9%) 182736800 (1%) 68 249 d14_12_008 23437934592 1415 2010278912 (9%) 182304832 (1%) 69 226 d14_13_009 23437934592 1415 2008749056 (9%) 182854464 (1%) 70 237 d14_14_010 23437934592 1415 2009348096 (9%) 182257024 (1%) 71 249 d14_15_011 23437934592 1415 2008017920 (9%) 182217152 (1%) 72 233 d14_16_012 23437934592 1415 2008158208 (9%) 182239680 (1%) 97 237 d15_11_007 23437934592 1415 2007257088 (9%) 181898304 (1%) 98 233 d15_12_008 23437934592 1415 2008309760 (9%) 182000928 (1%) 99 229 d15_13_009 23437934592 1415 2008755200 (9%) 182380544 (1%) 100 224 d15_14_010 23437934592 1415 2008460288 (9%) 181691616 (1%) 101 238 d15_15_011 23437934592 1415 2008394752 (9%) 181699328 (1%) 102 235 d15_16_012 23437934592 1415 2009153536 (9%) 182165312 (1%) 116 248 d11_42_026 23437934592 1011 4146111488 (18%) 134941504 (1%) 117 249 d13_42_026 23437934592 1213 4147710976 (18%) 135203776 (1%) 118 248 d15_42_026 23437934592 1415 4147094528 (18%) 134748320 (1%) 119 249 d10_43_027 23437934592 1011 4148652032 (18%) 135124288 (1%) 120 247 d11_43_027 23437934592 1011 4147335168 (18%) 134808256 (1%) 121 250 d12_43_027 23437934592 1213 4147177472 (18%) 134812096 (1%) 122 248 d13_43_027 23437934592 1213 4147270656 (18%) 134516192 (1%) 123 248 d14_43_027 23437934592 1415 4148153344 (18%) 134928896 (1%) 124 248 d15_43_027 23437934592 1415 4149500928 (18%) 134818880 (1%) 157 247 d10_11_007 23437934592 1011 2008516608 (9%) 182482592 (1%) 159 248 d10_13_009 23437934592 1011 2009745408 (9%) 182393184 (1%) 161 233 d10_15_011 23437934592 1011 2007539712 (9%) 182169856 (1%) 163 234 d11_11_007 23437934592 1011 2009283584 (9%) 182173824 (1%) 164 244 d11_12_008 23437934592 1011 2009091072 (9%) 182099200 (1%) 165 248 d11_13_009 23437934592 1011 2008827904 (9%) 182029632 (1%) 166 231 d11_14_010 23437934592 1011 2010264576 (9%) 181675424 (1%) 167 235 d11_15_011 23437934592 1011 2010696704 (9%) 181854304 (1%) 168 236 d11_16_012 23437934592 1011 2008183808 (9%) 182236800 (1%) -------------- next part -------------- 13 222 d12_21_013 23437934592 1213 2957768704 (13%) 37317824 (0%) 14 223 d12_22_014 23437934592 1213 2957680640 (13%) 37608576 (0%) 15 223 d12_23_015 23437934592 1213 2958575616 (13%) 37380672 (0%) 16 223 d12_24_016 23437934592 1213 2957406208 (13%) 37403648 (0%) 17 223 d12_25_017 23437934592 1213 2957883392 (13%) 37558016 (0%) 18 222 d12_26_018 23437934592 1213 2957815808 (13%) 37556416 (0%) 43 222 d13_21_013 23437934592 1213 2957149184 (13%) 37745280 (0%) 44 222 d13_22_014 23437934592 1213 2957812736 (13%) 37520736 (0%) 45 222 d13_23_015 23437934592 1213 2957990912 (13%) 37341952 (0%) 46 222 d13_24_016 23437934592 1213 2958204928 (13%) 37477536 (0%) 47 222 d13_25_017 23437934592 1213 2958315520 (13%) 37507072 (0%) 48 222 d13_26_018 23437934592 1213 2958086144 (13%) 37407648 (0%) 73 222 d14_21_013 23437934592 1415 2957812736 (13%) 37631488 (0%) 74 223 d14_22_014 23437934592 1415 2959673344 (13%) 37427680 (0%) 75 223 d14_23_015 23437934592 1415 2957627392 (13%) 37400992 (0%) 76 223 d14_24_016 23437934592 1415 2957408256 (13%) 37235776 (0%) 77 222 d14_25_017 23437934592 1415 2957407232 (13%) 37499200 (0%) 78 222 d14_26_018 23437934592 1415 2956661760 (13%) 37633696 (0%) 103 222 d15_21_013 23437934592 1415 2957427712 (13%) 37629664 (0%) 104 222 d15_22_014 23437934592 1415 2957506560 (13%) 37571968 (0%) 105 222 d15_23_015 23437934592 1415 2958114816 (13%) 37422400 (0%) 106 222 d15_24_016 23437934592 1415 2957561856 (13%) 37436768 (0%) 107 222 d15_25_017 23437934592 1415 2957537280 (13%) 37353984 (0%) 108 222 d15_26_018 23437934592 1415 2957539328 (13%) 37335872 (0%) 125 222 d11_44_028 23437934592 1011 13297235968 (57%) 20505568 (0%) 126 222 d12_44_028 23437934592 1213 13296712704 (57%) 20632768 (0%) 127 222 d12_45_029 23437934592 1213 13297404928 (57%) 20557408 (0%) 128 222 d13_44_028 23437934592 1213 13297271808 (57%) 20502304 (0%) 129 222 d14_44_028 23437934592 1415 13294722048 (57%) 20569824 (0%) 130 222 d14_45_029 23437934592 1415 13296071680 (57%) 20667104 (0%) 131 222 d15_44_028 23437934592 1415 13297007616 (57%) 20429536 (0%) 132 223 d10_44_028 23437934592 1011 13309036544 (57%) 20299552 (0%) 133 223 d10_45_029 23437934592 1011 13309213696 (57%) 20464576 (0%) 169 223 d10_21_013 23437934592 1011 2797943808 (12%) 36881600 (0%) 170 222 d10_22_014 23437934592 1011 2798030848 (12%) 37152256 (0%) 171 222 d10_23_015 23437934592 1011 2798152704 (12%) 36959840 (0%) 172 222 d10_24_016 23437934592 1011 2798002176 (12%) 36954976 (0%) 173 222 d10_25_017 23437934592 1011 2798524416 (12%) 36737120 (0%) 174 222 d10_26_018 23437934592 1011 2797768704 (12%) 36888096 (0%) 175 222 d11_21_013 23437934592 1011 2958944256 (13%) 37504288 (0%) 176 222 d11_22_014 23437934592 1011 2957225984 (13%) 37575648 (0%) 177 222 d11_23_015 23437934592 1011 2958100480 (13%) 37546336 (0%) 178 222 d11_24_016 23437934592 1011 2958453760 (13%) 37578784 (0%) 179 222 d11_25_017 23437934592 1011 2958053376 (13%) 37331808 (0%) 180 222 d11_26_018 23437934592 1011 2958290944 (13%) 37445824 (0%) -------------- next part -------------- 19 223 d12_31_019 23437934592 1213 3803873280 (16%) 117281408 (1%) 20 222 d12_32_020 23437934592 1213 3779237888 (16%) 116652576 (0%) 21 222 d12_33_021 23437934592 1213 3779206144 (16%) 116820288 (0%) 22 222 d12_34_022 23437934592 1213 3776828416 (16%) 116108032 (0%) 23 222 d12_35_023 23437934592 1213 3776950272 (16%) 116238752 (0%) 24 222 d12_36_024 23437934592 1213 3776524288 (16%) 116599744 (0%) 49 222 d13_31_019 23437934592 1213 3778293760 (16%) 116365600 (0%) 50 222 d13_32_020 23437934592 1213 3778381824 (16%) 115960448 (0%) 51 222 d13_33_021 23437934592 1213 3777104896 (16%) 116225984 (0%) 52 222 d13_34_022 23437934592 1213 3775632384 (16%) 116496576 (0%) 53 222 d13_35_023 23437934592 1213 3776899072 (16%) 116397120 (0%) 54 222 d13_36_024 23437934592 1213 3776140288 (16%) 116400000 (0%) 79 222 d14_31_019 23437934592 1415 3844638720 (16%) 113531904 (0%) 80 222 d14_32_020 23437934592 1415 3817139200 (16%) 113600928 (0%) 81 222 d14_33_021 23437934592 1415 3815447552 (16%) 113358272 (0%) 82 222 d14_34_022 23437934592 1415 3814561792 (16%) 113348160 (0%) 83 222 d14_35_023 23437934592 1415 3815387136 (16%) 113582432 (0%) 84 222 d14_36_024 23437934592 1415 3815303168 (16%) 112914816 (0%) 109 222 d15_31_019 23437934592 1415 3814614016 (16%) 113298240 (0%) 110 222 d15_32_020 23437934592 1415 3814936576 (16%) 113053664 (0%) 111 222 d15_33_021 23437934592 1415 3815122944 (16%) 113438656 (0%) 112 222 d15_34_022 23437934592 1415 3813474304 (16%) 113481216 (0%) 113 222 d15_35_023 23437934592 1415 3812912128 (16%) 113618624 (0%) 114 222 d15_36_024 23437934592 1415 3813067776 (16%) 113282848 (0%) 134 223 d11_45_029 23437934592 1011 3683404800 (16%) 100860000 (0%) 135 222 d13_45_029 23437934592 1213 3682211840 (16%) 101101824 (0%) 136 223 d15_45_029 23437934592 1415 3682046976 (16%) 101375328 (0%) 137 222 d10_46_030 23437934592 1011 3684398080 (16%) 100946912 (0%) 138 222 d11_46_030 23437934592 1011 3683755008 (16%) 101295200 (0%) 139 223 d12_46_030 23437934592 1213 3682649088 (16%) 100540832 (0%) 140 223 d13_46_030 23437934592 1213 3684602880 (16%) 100489376 (0%) 141 223 d14_46_030 23437934592 1415 3681666048 (16%) 100854112 (0%) 142 223 d15_46_030 23437934592 1415 3685487616 (16%) 100715744 (0%) 181 222 d10_31_019 23437934592 1011 3816681472 (16%) 113389344 (0%) 182 222 d10_32_020 23437934592 1011 3816535040 (16%) 113652672 (0%) 183 222 d10_33_021 23437934592 1011 3817772032 (16%) 113124224 (0%) 184 222 d10_34_022 23437934592 1011 3817069568 (16%) 113225504 (0%) 185 222 d10_35_023 23437934592 1011 3816517632 (16%) 113163488 (0%) 186 222 d10_36_024 23437934592 1011 3815605248 (16%) 113475648 (0%) 187 222 d11_31_019 23437934592 1011 3815664640 (16%) 113100992 (0%) 188 222 d11_32_020 23437934592 1011 3815985152 (16%) 113325504 (0%) 189 222 d11_33_021 23437934592 1011 3815148544 (16%) 113328480 (0%) 190 223 d11_34_022 23437934592 1011 3814780928 (16%) 113457440 (0%) 191 223 d11_35_023 23437934592 1011 3815219200 (16%) 113065504 (0%) 192 223 d11_36_024 23437934592 1011 3815383040 (16%) 113170592 (0%) -------------- next part -------------- > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From aaron.s.knister at nasa.gov Sat Jan 13 17:26:51 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 13 Jan 2018 12:26:51 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Message-ID: Thanks, Peter. That definitely makes sense and I was actually wondering if performance was a factor. Do you know where to look to see what GPFS' perception of "performance" is for a given NSD? -Aaron On 1/13/18 12:00 PM, Peter Serocka wrote: > Within reasonable capacity limits it would also expect > to direct incoming data to disks that are best ?available? > from a current performance perspective ? like doing least > IOPS, having lowest latency and shortest filled queue length. > > You new NSDs, filled only with recent data, might quickly have > become the most busy units before reaching capacity balance, > simply because recent data tends to be more active than older stuff. > > Makes sense? > > ? Peter > >> On 2018 Jan 13 Sat, at 17:18, Aaron Knister wrote: >> >> Thanks Everyone! I whipped up a script to dump the block layout of a >> file and then join that with mmdf information. As part of my exploration >> I wrote one 2GB file to each of this particular filesystem's 4 data >> pools last night (using "touch $file; mmchattr $file -P $pool; dd >> of=$file") and have attached a dump of the layout/nsd information for >> each file/pool. The fields for the output are: >> >> diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, >> freePct, freeKbFragments, freeKbFragmentsPct >> >> >> Here's the highlight from pool1: >> >> 36? 264? d13_06_006??? 23437934592? 1213??? 4548935680? (19%) >> 83304320?? (0%) >> 59?? 74? d10_41_025??? 23437934592? 1011??? 6993759232? (30%) >> 58642816?? (0%) >> >> For this file (And anecdotally what I've seen looking at NSD I/O data >> for other files written to this pool) the pattern of more blocks being >> allocated to the NSDs that are ~20% free vs the NSDs that are 30% free >> seems to be fairly consistent. >> >> >> Looking at a snippet of pool2: >> 101? 238? d15_15_011??? 23437934592? 1415??? 2008394752?? (9%) >> 181699328?? (1%) >> 102? 235? d15_16_012??? 23437934592? 1415??? 2009153536?? (9%) >> 182165312?? (1%) >> 116? 248? d11_42_026??? 23437934592? 1011??? 4146111488? (18%) >> 134941504?? (1%) >> 117? 249? d13_42_026??? 23437934592? 1213??? 4147710976? (18%) >> 135203776?? (1%) >> >> there are slightly more blocks allocated in general on the NSDs with >> twice the amount of free space, but it doesn't seem to be a significant >> amount relative to the delta in free space. The pattern from pool1 >> certainly doesn't hold true here. >> >> Pool4 isn't very interesting because all of the NSDs are well balanced >> in terms of free space (all 16% free). >> >> Pool3, however, *is* particularly interesting. Here's a snippet: >> >> 106? 222? d15_24_016??? 23437934592? 1415??? 2957561856? (13%) >> 37436768?? (0%) >> 107? 222? d15_25_017??? 23437934592? 1415??? 2957537280? (13%) >> 37353984?? (0%) >> 108? 222? d15_26_018??? 23437934592? 1415??? 2957539328? (13%) >> 37335872?? (0%) >> 125? 222? d11_44_028??? 23437934592? 1011?? 13297235968? (57%) >> 20505568?? (0%) >> 126? 222? d12_44_028??? 23437934592? 1213?? 13296712704? (57%) >> 20632768?? (0%) >> 127? 222? d12_45_029??? 23437934592? 1213?? 13297404928? (57%) >> 20557408?? (0%) >> >> GPFS consistently allocated the same number of blocks to disks with ~4x >> the free space than it did the other disks in the pool. >> >> Suffice it to say-- I'm *very* confused :) >> >> -Aaron >> >> On 1/13/18 8:18 AM, Daniel Kidger wrote: >>> Aaron, >>>? >>> Also are your new NSDs the same size as your existing ones? >>> i.e. the NSDs that are at a higher %age full might have more free blocks >>> than the other NSDs? >>> Daniel >>> >>>? >>> IBM Storage Professional Badge >>> >>>? >>>???????????????? >>> *Dr Daniel Kidger* >>> IBM Technical Sales Specialist >>> Software Defined Solution Sales >>> >>> +44-(0)7818 522 266 >>> daniel.kidger at uk.ibm.com >>> >>>? >>>? >>>? >>> >>>??? ----- Original message ----- >>>??? From: Jan-Frode Myklebust >>>??? Sent by: gpfsug-discuss-bounces at spectrumscale.org >>>??? To: gpfsug main discussion list >>>??? Cc: >>>??? Subject: Re: [gpfsug-discuss] pool block allocation algorithm >>>??? Date: Sat, Jan 13, 2018 9:25 AM >>>???? >>>??? Don?t have documentation/whitepaper, but as I recall, it will first >>>??? allocate round-robin over failureGroup, then round-robin over >>>??? nsdServers, and then round-robin over volumes. So if these new NSDs >>>??? are defined as different failureGroup from the old disks, that might >>>??? explain it.. >>> >>> >>>??? -jf >>>??? l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >>>??? >: >>> >>>??????? Apologies if this has been covered elsewhere (I couldn't find it >>>??????? if it >>>??????? has). I'm curious how GPFS decides where to allocate new blocks. >>> >>>??????? We've got a filesystem that we added some NSDs to a while back >>>??????? and it >>>??????? hurt there for a little while because it appeared as though GPFS was >>>??????? choosing to allocate new blocks much more frequently on the >>>??????? ~100% free >>>??????? LUNs than the existing LUNs (I can't recall how free they were >>>??????? at the >>>??????? time). Looking at it now, though, it seems GPFS is doing the >>>??????? opposite. >>>??????? There's now a ~10% difference between the LUNs added and the >>>??????? existing >>>??????? LUNs (20% free vs 30% free) and GPFS is choosing to allocate new >>>??????? writes >>>??????? at a ratio of about 3:1 on the disks with *fewer* free blocks >>>??????? than on >>>??????? the disks with more free blocks. That's completely inconsistent with >>>??????? what we saw when we initially added the disks which makes me >>>??????? wonder how >>>??????? GPFS is choosing to allocate new blocks (other than the obvious bits >>>??????? about failure group, and replication factor). Could someone >>>??????? explain (or >>>??????? point me at a whitepaper) what factors GPFS uses when allocating >>>??????? blocks, >>>??????? particularly as it pertains to choosing one NSD over another >>>??????? within the >>>??????? same failure group. >>> >>>??????? Thanks! >>> >>>??????? -Aaron >>> >>>??????? -- >>>??????? Aaron Knister >>>??????? NASA Center for Climate Simulation (Code 606.2) >>>??????? Goddard Space Flight Center >>>??????? (301) 286-2776 >>>??????? _______________________________________________ >>>??????? gpfsug-discuss mailing list >>>??????? gpfsug-discuss at spectrumscale.org >>>??????? >>>??????? http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>??????? >>> >>>??? _______________________________________________ >>>??? gpfsug-discuss mailing list >>>??? gpfsug-discuss at spectrumscale.org >>>??? https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= >>> >>>? >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From wsawdon at us.ibm.com Sat Jan 13 19:43:22 2018 From: wsawdon at us.ibm.com (Wayne Sawdon) Date: Sat, 13 Jan 2018 11:43:22 -0800 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov><48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Message-ID: Originally, GPFS used a strict round robin, first over failure groups, then over volumes within each failure group. That had performance issues when one or more volumes was low on space. Then for a while there were a variety of weighted stripe methods including by free space and by capacity. The file system had an option allowing the user to change the stripe method. That option was removed when we switched to a "best effort" round robin, which does a round robin over the failure groups then volumes based on the allocation regions that a node owns. When the stripe width at a node drops below half of the failure groups or half of the volumes that node will acquire new allocation regions. Basically we vary the stripe width to avoid searching for free space on specific volumes. It will eventually even itself out or you could restripe the file system to even it out immediately. -Wayne ps. And of course, allocation in FPO is completely different. gpfsug-discuss-bounces at spectrumscale.org wrote on 01/13/2018 09:26:51 AM: > From: Aaron Knister > To: gpfsug main discussion list > Date: 01/13/2018 09:27 AM > Subject: Re: [gpfsug-discuss] pool block allocation algorithm > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Thanks, Peter. That definitely makes sense and I was actually wondering > if performance was a factor. Do you know where to look to see what GPFS' > perception of "performance" is for a given NSD? > > -Aaron > > On 1/13/18 12:00 PM, Peter Serocka wrote: > > Within reasonable capacity limits it would also expect > > to direct incoming data to disks that are best ?available? > > from a current performance perspective ? like doing least > > IOPS, having lowest latency and shortest filled queue length. > > > > You new NSDs, filled only with recent data, might quickly have > > become the most busy units before reaching capacity balance, > > simply because recent data tends to be more active than older stuff. > > > > Makes sense? > > > > ? Peter > > > >> On 2018 Jan 13 Sat, at 17:18, Aaron Knister > wrote: > >> > >> Thanks Everyone! I whipped up a script to dump the block layout of a > >> file and then join that with mmdf information. As part of my exploration > >> I wrote one 2GB file to each of this particular filesystem's 4 data > >> pools last night (using "touch $file; mmchattr $file -P $pool; dd > >> of=$file") and have attached a dump of the layout/nsd information for > >> each file/pool. The fields for the output are: > >> > >> diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, > >> freePct, freeKbFragments, freeKbFragmentsPct > >> > >> > >> Here's the highlight from pool1: > >> > >> 36? 264? d13_06_006??? 23437934592? 1213??? 4548935680? (19%) > >> 83304320?? (0%) > >> 59?? 74? d10_41_025??? 23437934592? 1011??? 6993759232? (30%) > >> 58642816?? (0%) > >> > >> For this file (And anecdotally what I've seen looking at NSD I/O data > >> for other files written to this pool) the pattern of more blocks being > >> allocated to the NSDs that are ~20% free vs the NSDs that are 30% free > >> seems to be fairly consistent. > >> > >> > >> Looking at a snippet of pool2: > >> 101? 238? d15_15_011??? 23437934592? 1415??? 2008394752?? (9%) > >> 181699328?? (1%) > >> 102? 235? d15_16_012??? 23437934592? 1415??? 2009153536?? (9%) > >> 182165312?? (1%) > >> 116? 248? d11_42_026??? 23437934592? 1011??? 4146111488? (18%) > >> 134941504?? (1%) > >> 117? 249? d13_42_026??? 23437934592? 1213??? 4147710976? (18%) > >> 135203776?? (1%) > >> > >> there are slightly more blocks allocated in general on the NSDs with > >> twice the amount of free space, but it doesn't seem to be a significant > >> amount relative to the delta in free space. The pattern from pool1 > >> certainly doesn't hold true here. > >> > >> Pool4 isn't very interesting because all of the NSDs are well balanced > >> in terms of free space (all 16% free). > >> > >> Pool3, however, *is* particularly interesting. Here's a snippet: > >> > >> 106? 222? d15_24_016??? 23437934592? 1415??? 2957561856? (13%) > >> 37436768?? (0%) > >> 107? 222? d15_25_017??? 23437934592? 1415??? 2957537280? (13%) > >> 37353984?? (0%) > >> 108? 222? d15_26_018??? 23437934592? 1415??? 2957539328? (13%) > >> 37335872?? (0%) > >> 125? 222? d11_44_028??? 23437934592? 1011?? 13297235968? (57%) > >> 20505568?? (0%) > >> 126? 222? d12_44_028??? 23437934592? 1213?? 13296712704? (57%) > >> 20632768?? (0%) > >> 127? 222? d12_45_029??? 23437934592? 1213?? 13297404928? (57%) > >> 20557408?? (0%) > >> > >> GPFS consistently allocated the same number of blocks to disks with ~4x > >> the free space than it did the other disks in the pool. > >> > >> Suffice it to say-- I'm *very* confused :) > >> > >> -Aaron > >> > >> On 1/13/18 8:18 AM, Daniel Kidger wrote: > >>> Aaron, > >>> > >>> Also are your new NSDs the same size as your existing ones? > >>> i.e. the NSDs that are at a higher %age full might have more free blocks > >>> than the other NSDs? > >>> Daniel > >>> > >>> > >>> IBM Storage Professional Badge > >>> u=https-3A__www.youracclaim.com_user_danel-2Dkidger&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=hu8pcNGJmsITfq8y9fzxDf9WoXD1Kr5ptVLEEpbwcjU&e=> > >>> > >>> > >>> *Dr Daniel Kidger* > >>> IBM Technical Sales Specialist > >>> Software Defined Solution Sales > >>> > >>> +44-(0)7818 522 266 > >>> daniel.kidger at uk.ibm.com > >>> > >>> > >>> > >>> > >>> > >>>??? ----- Original message ----- > >>>??? From: Jan-Frode Myklebust > >>>??? Sent by: gpfsug-discuss-bounces at spectrumscale.org > >>>??? To: gpfsug main discussion list > >>>??? Cc: > >>>??? Subject: Re: [gpfsug-discuss] pool block allocation algorithm > >>>??? Date: Sat, Jan 13, 2018 9:25 AM > >>> > >>>??? Don?t have documentation/whitepaper, but as I recall, it will first > >>>??? allocate round-robin over failureGroup, then round-robin over > >>>??? nsdServers, and then round-robin over volumes. So if these new NSDs > >>>??? are defined as different failureGroup from the old disks, that might > >>>??? explain it.. > >>> > >>> > >>>??? -jf > >>>??? l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister > >>>??? >: > >>> > >>>??????? Apologies if this has been covered elsewhere (I couldn't find it > >>>??????? if it > >>>??????? has). I'm curious how GPFS decides where to allocate new blocks. > >>> > >>>??????? We've got a filesystem that we added some NSDs to a while back > >>>??????? and it > >>>??????? hurt there for a little while because it appeared as > though GPFS was > >>>??????? choosing to allocate new blocks much more frequently on the > >>>??????? ~100% free > >>>??????? LUNs than the existing LUNs (I can't recall how free they were > >>>??????? at the > >>>??????? time). Looking at it now, though, it seems GPFS is doing the > >>>??????? opposite. > >>>??????? There's now a ~10% difference between the LUNs added and the > >>>??????? existing > >>>??????? LUNs (20% free vs 30% free) and GPFS is choosing to allocate new > >>>??????? writes > >>>??????? at a ratio of about 3:1 on the disks with *fewer* free blocks > >>>??????? than on > >>>??????? the disks with more free blocks. That's completely > inconsistent with > >>>??????? what we saw when we initially added the disks which makes me > >>>??????? wonder how > >>>??????? GPFS is choosing to allocate new blocks (other than the > obvious bits > >>>??????? about failure group, and replication factor). Could someone > >>>??????? explain (or > >>>??????? point me at a whitepaper) what factors GPFS uses when allocating > >>>??????? blocks, > >>>??????? particularly as it pertains to choosing one NSD over another > >>>??????? within the > >>>??????? same failure group. > >>> > >>>??????? Thanks! > >>> > >>>??????? -Aaron > >>> > >>>??????? -- > >>>??????? Aaron Knister > >>>??????? NASA Center for Climate Simulation (Code 606.2) > >>>??????? Goddard Space Flight Center > >>>??????? (301) 286-2776 > >>>??????? _______________________________________________ > >>>??????? gpfsug-discuss mailing list > >>>??????? gpfsug-discuss at spectrumscale.org > >>>??????? u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=QYsXVDOdNRcII7FPtAbCXEyYJzNSd_UXq8bmreALKxs&e= > > > >>>??????? https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > >>>??????? u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- > GTf8EwJ6AkZQiTsRQZ73UH20&e=> > >>> > >>>??? _______________________________________________ > >>>??? gpfsug-discuss mailing list > >>>??? gpfsug-discuss at spectrumscale.org > >>>??? https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- > GTf8EwJ6AkZQiTsRQZ73UH20&e= > >>> > >>> > >>> Unless stated otherwise above: > >>> IBM United Kingdom Limited - Registered in England and Wales with number > >>> 741598. > >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >>> > >>> > >>> > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > >>> > >> > >> -- > >> Aaron Knister > >> NASA Center for Climate Simulation (Code 606.2) > >> Goddard Space Flight Center > >> (301) 286-2776 > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sun Jan 14 22:22:15 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sun, 14 Jan 2018 17:22:15 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Message-ID: <80dccad3-f531-3a8e-ab28-3ffa83dbd990@nasa.gov> Thanks, Wayne. What you said makes sense although I'm not sure I completely grok it. Can you comment on whether or not historic LUN performance factors into allocation decisions? -Aaron On 1/13/18 2:43 PM, Wayne Sawdon wrote: > Originally, GPFS used a strict round robin, first over failure groups, > then over volumes within each > failure group. That had performance issues when one or more volumes was > low on space. Then > for a while there were a variety of weighted stripe methods including by > free space and by capacity. > The file system had an option allowing the user to change the stripe > method. That option was > removed when we switched to a "best effort" round robin, which does a > round robin over the > failure groups then volumes based on the allocation regions that a node > owns. When the stripe width > at a node drops below half of the failure groups or half of the volumes > that node will acquire new > allocation regions. > > Basically we vary the stripe width to avoid searching for free space on > specific volumes. It will > eventually even itself out or you could restripe the file system to even > it out immediately. > > -Wayne > > ps. And of course, allocation in FPO is completely different. > > > gpfsug-discuss-bounces at spectrumscale.org wrote on 01/13/2018 09:26:51 AM: > >> From: Aaron Knister >> To: gpfsug main discussion list >> Date: 01/13/2018 09:27 AM >> Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> >> Thanks, Peter. That definitely makes sense and I was actually wondering >> if performance was a factor. Do you know where to look to see what GPFS' >> perception of "performance" is for a given NSD? >> >> -Aaron >> >> On 1/13/18 12:00 PM, Peter Serocka wrote: >> > Within reasonable capacity limits it would also expect >> > to direct incoming data to disks that are best ?available? >> > from a current performance perspective ? like doing least >> > IOPS, having lowest latency and shortest filled queue length. >> > >> > You new NSDs, filled only with recent data, might quickly have >> > become the most busy units before reaching capacity balance, >> > simply because recent data tends to be more active than older stuff. >> > >> > Makes sense? >> > >> > ? Peter >> > >> >> On 2018 Jan 13 Sat, at 17:18, Aaron Knister >> wrote: >> >> >> >> Thanks Everyone! I whipped up a script to dump the block layout of a >> >> file and then join that with mmdf information. As part of my > exploration >> >> I wrote one 2GB file to each of this particular filesystem's 4 data >> >> pools last night (using "touch $file; mmchattr $file -P $pool; dd >> >> of=$file") and have attached a dump of the layout/nsd information for >> >> each file/pool. The fields for the output are: >> >> >> >> diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, >> >> freePct, freeKbFragments, freeKbFragmentsPct >> >> >> >> >> >> Here's the highlight from pool1: >> >> >> >> 36? 264? d13_06_006??? 23437934592? 1213??? 4548935680? (19%) >> >> 83304320?? (0%) >> >> 59?? 74? d10_41_025??? 23437934592? 1011??? 6993759232? (30%) >> >> 58642816?? (0%) >> >> >> >> For this file (And anecdotally what I've seen looking at NSD I/O data >> >> for other files written to this pool) the pattern of more blocks being >> >> allocated to the NSDs that are ~20% free vs the NSDs that are 30% free >> >> seems to be fairly consistent. >> >> >> >> >> >> Looking at a snippet of pool2: >> >> 101? 238? d15_15_011??? 23437934592? 1415??? 2008394752?? (9%) >> >> 181699328?? (1%) >> >> 102? 235? d15_16_012??? 23437934592? 1415??? 2009153536?? (9%) >> >> 182165312?? (1%) >> >> 116? 248? d11_42_026??? 23437934592? 1011??? 4146111488? (18%) >> >> 134941504?? (1%) >> >> 117? 249? d13_42_026??? 23437934592? 1213??? 4147710976? (18%) >> >> 135203776?? (1%) >> >> >> >> there are slightly more blocks allocated in general on the NSDs with >> >> twice the amount of free space, but it doesn't seem to be a significant >> >> amount relative to the delta in free space. The pattern from pool1 >> >> certainly doesn't hold true here. >> >> >> >> Pool4 isn't very interesting because all of the NSDs are well balanced >> >> in terms of free space (all 16% free). >> >> >> >> Pool3, however, *is* particularly interesting. Here's a snippet: >> >> >> >> 106? 222? d15_24_016??? 23437934592? 1415??? 2957561856? (13%) >> >> 37436768?? (0%) >> >> 107? 222? d15_25_017??? 23437934592? 1415??? 2957537280? (13%) >> >> 37353984?? (0%) >> >> 108? 222? d15_26_018??? 23437934592? 1415??? 2957539328? (13%) >> >> 37335872?? (0%) >> >> 125? 222? d11_44_028??? 23437934592? 1011?? 13297235968? (57%) >> >> 20505568?? (0%) >> >> 126? 222? d12_44_028??? 23437934592? 1213?? 13296712704? (57%) >> >> 20632768?? (0%) >> >> 127? 222? d12_45_029??? 23437934592? 1213?? 13297404928? (57%) >> >> 20557408?? (0%) >> >> >> >> GPFS consistently allocated the same number of blocks to disks with ~4x >> >> the free space than it did the other disks in the pool. >> >> >> >> Suffice it to say-- I'm *very* confused :) >> >> >> >> -Aaron >> >> >> >> On 1/13/18 8:18 AM, Daniel Kidger wrote: >> >>> Aaron, >> >>>? >> >>> Also are your new NSDs the same size as your existing ones? >> >>> i.e. the NSDs that are at a higher %age full might have more free > blocks >> >>> than the other NSDs? >> >>> Daniel >> >>> >> >>>? >> >>> IBM Storage Professional Badge >> >>> > > u=https-3A__www.youracclaim.com_user_danel-2Dkidger&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=hu8pcNGJmsITfq8y9fzxDf9WoXD1Kr5ptVLEEpbwcjU&e=> >> >>>? >> >>>???????????????? >> >>> *Dr Daniel Kidger* >> >>> IBM Technical Sales Specialist >> >>> Software Defined Solution Sales >> >>> >> >>> +44-(0)7818 522 266 >> >>> daniel.kidger at uk.ibm.com >> >>> >> >>>? >> >>>? >> >>>? >> >>> >> >>>??? ----- Original message ----- >> >>>??? From: Jan-Frode Myklebust >> >>>??? Sent by: gpfsug-discuss-bounces at spectrumscale.org >> >>>??? To: gpfsug main discussion list >> >>>??? Cc: >> >>>??? Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> >>>??? Date: Sat, Jan 13, 2018 9:25 AM >> >>>???? >> >>>??? Don?t have documentation/whitepaper, but as I recall, it will first >> >>>??? allocate round-robin over failureGroup, then round-robin over >> >>>??? nsdServers, and then round-robin over volumes. So if these new NSDs >> >>>??? are defined as different failureGroup from the old disks, that > might >> >>>??? explain it.. >> >>> >> >>> >> >>>??? -jf >> >>>??? l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >> >>>??? >: >> >>> >> >>>??????? Apologies if this has been covered elsewhere (I couldn't > find it >> >>>??????? if it >> >>>??????? has). I'm curious how GPFS decides where to allocate new > blocks. >> >>> >> >>>??????? We've got a filesystem that we added some NSDs to a while back >> >>>??????? and it >> >>>??????? hurt there for a little while because it appeared as >> though GPFS was >> >>>??????? choosing to allocate new blocks much more frequently on the >> >>>??????? ~100% free >> >>>??????? LUNs than the existing LUNs (I can't recall how free they were >> >>>??????? at the >> >>>??????? time). Looking at it now, though, it seems GPFS is doing the >> >>>??????? opposite. >> >>>??????? There's now a ~10% difference between the LUNs added and the >> >>>??????? existing >> >>>??????? LUNs (20% free vs 30% free) and GPFS is choosing to > allocate new >> >>>??????? writes >> >>>??????? at a ratio of about 3:1 on the disks with *fewer* free blocks >> >>>??????? than on >> >>>??????? the disks with more free blocks. That's completely >> inconsistent with >> >>>??????? what we saw when we initially added the disks which makes me >> >>>??????? wonder how >> >>>??????? GPFS is choosing to allocate new blocks (other than the >> obvious bits >> >>>??????? about failure group, and replication factor). Could someone >> >>>??????? explain (or >> >>>??????? point me at a whitepaper) what factors GPFS uses when > allocating >> >>>??????? blocks, >> >>>??????? particularly as it pertains to choosing one NSD over another >> >>>??????? within the >> >>>??????? same failure group. >> >>> >> >>>??????? Thanks! >> >>> >> >>>??????? -Aaron >> >>> >> >>>??????? -- >> >>>??????? Aaron Knister >> >>>??????? NASA Center for Climate Simulation (Code 606.2) >> >>>??????? Goddard Space Flight Center >> >>>??????? (301) 286-2776 >> >>>??????? _______________________________________________ >> >>>??????? gpfsug-discuss mailing list >> >>>??????? gpfsug-discuss at spectrumscale.org >> >>>??????? > u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=QYsXVDOdNRcII7FPtAbCXEyYJzNSd_UXq8bmreALKxs&e= >> > >> >>>??????? https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> >>>??????? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- >> GTf8EwJ6AkZQiTsRQZ73UH20&e=> >> >>> >> >>>??? _______________________________________________ >> >>>??? gpfsug-discuss mailing list >> >>>??? gpfsug-discuss at spectrumscale.org >> >>>??? https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- >> GTf8EwJ6AkZQiTsRQZ73UH20&e= >> >>> >> >>>? >> >>> Unless stated otherwise above: >> >>> IBM United Kingdom Limited - Registered in England and Wales with > number >> >>> 741598. >> >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > PO6 3AU >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> gpfsug-discuss mailing list >> >>> gpfsug-discuss at spectrumscale.org >> >>> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> >>> >> >> >> >> -- >> >> Aaron Knister >> >> NASA Center for Climate Simulation (Code 606.2) >> >> Goddard Space Flight Center >> >> (301) 286-2776 >> >> _______________________________________________ >> >> gpfsug-discuss mailing list >> >> gpfsug-discuss at spectrumscale.org >> >> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From kaustubh.katruwar at in.ibm.com Mon Jan 15 09:50:37 2018 From: kaustubh.katruwar at in.ibm.com (Kaustubh I Katruwar) Date: Mon, 15 Jan 2018 15:20:37 +0530 Subject: [gpfsug-discuss] SMB with LDAP In-Reply-To: References: Message-ID: Hi, Can you also describe your problem? The link talks about updating the LDAP server entries to help integrate with Spectrum Scale to serve SMB connections. Thanks and Regards, Kaustubh I. Katruwar From: Jeno Cram To: "gpfsug-discuss at spectrumscale.org" Date: 09/01/2018 09:06 PM Subject: [gpfsug-discuss] SMB with LDAP Sent by: gpfsug-discuss-bounces at spectrumscale.org Has anyone had any luck implementing CES with SMB using LDAP? This link doesn?t seem to have all of the information required to getting it working properly. https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_updateldapsmb.htm I?m assuming smbpasswd -a is still required for all users? What keeps the LDAP/SMB passwords in sync? What access does the bind user need in LDAP? What special requirements are required for Windows 10 clients to connect? Jeno Cram | Lead Application Support Engineer jcram at ddn.com _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=gTocoNEzBZA5cNptWoAoyoucYhc_7VA6hXTPOXtubUc&m=NEFCJtsgntiVgznfKbVs0BqwQJOubKG8dhgNlMXvQzs&s=0U9-rvVW4HxBCh0tdCYaElQer5JBxJJ9KFFWkkutUQU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From a.khiredine at meteo.dz Mon Jan 15 10:41:12 2018 From: a.khiredine at meteo.dz (atmane khiredine) Date: Mon, 15 Jan 2018 10:41:12 +0000 Subject: [gpfsug-discuss] temperature disk gss Message-ID: <4B32CB5C696F2849BDEF7DF9EACE884B85A15F93@SDEB-EXC01.meteo.dz> Dear All, is there a way to display the temperature of the drives in GSS Thanks Everyone! Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz From coetzee.ray at gmail.com Mon Jan 15 12:26:43 2018 From: coetzee.ray at gmail.com (Ray Coetzee) Date: Mon, 15 Jan 2018 12:26:43 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: Hi Sven yes, it's primarily reads. Kind regards Ray Coetzee Mob: +44 759 704 7060 Skype: ray.coetzee Email: coetzee.ray at gmail.com On Fri, Jan 12, 2018 at 8:57 PM, Sven Oehme wrote: > is this primary read or write ? > > > On Fri, Jan 12, 2018, 12:51 PM Ray Coetzee wrote: > >> Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. >> (Without the meltdown patch) >> >> Hey Bryan, I remember that quote from Yuri, that's why I hoped some >> "magic" may have been done since then. >> >> Other attempts to improve performance I've tried include: >> >> - Using LROC to have a larger chance of a cache hit (Unfortunately >> the entire dataset is multiple TB) >> - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for >> this purpose (Job runs halved in duration but nowhere near what NFS can >> give) >> - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and >> some others with little improvement. >> >> For those interested, as a performance comparison. The same job when run >> on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe >> scratch filesystem and over 60min on spindle based filesystem. >> >> Kind regards >> >> Ray Coetzee >> Email: coetzee.ray at gmail.com >> >> >> On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister < >> bbanister at jumptrading.com> wrote: >> >>> You could put all of your data onto SSDs in a RAID1 configuration so >>> that you don?t have insane read-modify-write penalties on writes (RAID1) >>> and avoid horrible seek thrashing that spinning rust requires (SSD random >>> access medium) for your 4K I/O operations. >>> >>> >>> >>> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best >>> not to touch it?. He gave many reasons why mmap operations on a >>> distributed file system is incredibly hard and not recommended. >>> >>> -Bryan >>> >>> >>> >>> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- >>> bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >>> *Sent:* Friday, January 12, 2018 8:45 AM >>> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >>> gpfsug-discuss at spectrumscale.org> >>> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >>> >>> >>> >>> *Note: External Email* >>> ------------------------------ >>> >>> what version of Scale are you using right now ? >>> >>> >>> >>> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >>> wrote: >>> >>> I'd like to ask the group of their experiences in improving the >>> performance of applications that use mmap calls against files on Spectrum >>> Scale. >>> >>> >>> >>> Besides using an NFS export from CES instead of a native GPFS mount, or >>> precaching the dataset into the pagepool, what other approaches are there >>> to offset the performance hit of the 4K IO size? >>> >>> >>> >>> Kind regards >>> >>> Ray Coetzee >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> ------------------------------ >>> >>> Note: This email is for the confidential use of the named addressee(s) >>> only and may contain proprietary, confidential or privileged information. >>> If you are not the intended recipient, you are hereby notified that any >>> review, dissemination or copying of this email is strictly prohibited, and >>> to please notify the sender immediately and destroy this email and any >>> attachments. Email transmission cannot be guaranteed to be secure or >>> error-free. The Company, therefore, does not make any guarantees as to the >>> completeness or accuracy of this email or any attachments. This email is >>> for informational purposes only and does not constitute a recommendation, >>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >>> or perform any type of transaction of a financial product. >>> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckerner at illinois.edu Mon Jan 15 14:03:26 2018 From: ckerner at illinois.edu (Chad Kerner) Date: Mon, 15 Jan 2018 08:03:26 -0600 Subject: [gpfsug-discuss] temperature disk gss In-Reply-To: References: <4300efd8e3134cf1ba50454cac76bd09@CHIHT3.ad.uillinois.edu> Message-ID: You can get it from the Smart statistics with smartctl -A . smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.10.0-229.el7.x86_64] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === Current Drive Temperature: 30 C Drive Trip Temperature: 65 C Manufactured in week 39 of year 2016 Specified cycle count over device lifetime: 50000 Accumulated start-stop cycles: 6 Specified load-unload count over device lifetime: 600000 Accumulated load-unload cycles: 35 Elements in grown defect list: 0 Vendor (Seagate) cache information Blocks sent to initiator = 418499985932288 Chad -- Chad Kerner, Senior Storage Engineer National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Jan 15, 2018 4:48 AM, "atmane khiredine" wrote: Dear All, is there a way to display the temperature of the drives in GSS Thanks Everyone! Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From alvise.dorigo at psi.ch Mon Jan 15 13:58:12 2018 From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI)) Date: Mon, 15 Jan 2018 13:58:12 +0000 Subject: [gpfsug-discuss] temperature disk gss In-Reply-To: <4B32CB5C696F2849BDEF7DF9EACE884B85A15F93@SDEB-EXC01.meteo.dz> References: <4B32CB5C696F2849BDEF7DF9EACE884B85A15F93@SDEB-EXC01.meteo.dz> Message-ID: <83A6EEB0EC738F459A39439733AE80451BBC421D@MBX114.d.ethz.ch> Hi Atmane, in my Gl2 system I can do: # mmlsenclosure all -L|grep -B2 tempSensor component type serial number component id failed value unit properties -------------- ------------- ------------ ------ ----- ---- ---------- tempSensor XXXXXXXXX DCM_0A no 41 C tempSensor XXXXXXXXX DCM_0B no 40 C tempSensor XXXXXXXXX DCM_1A no 49 C tempSensor XXXXXXXXX DCM_1B no 42 C tempSensor XXXXXXXXX DCM_2A no 50 C tempSensor XXXXXXXXX DCM_2B no 41 C tempSensor XXXXXXXXX DCM_3A no 48 C tempSensor XXXXXXXXX DCM_3B no 42 C tempSensor XXXXXXXXX DCM_4A no 47 C tempSensor XXXXXXXXX DCM_4B no 42 C tempSensor XXXXXXXXX ESM_A no 43 C tempSensor XXXXXXXXX ESM_B no 44 C tempSensor XXXXXXXXX POWERSUPPLY_BOT no 41 C tempSensor XXXXXXXXX POWERSUPPLY_TOP no 39 C tempSensor XXXXXXXXX DCM_0A no 50 C tempSensor XXXXXXXXX DCM_0B no 41 C tempSensor XXXXXXXXX DCM_1A no 48 C tempSensor XXXXXXXXX DCM_1B no 41 C tempSensor XXXXXXXXX DCM_2A no 47 C tempSensor XXXXXXXXX DCM_2B no 42 C tempSensor XXXXXXXXX DCM_3A no 47 C tempSensor XXXXXXXXX DCM_3B no 41 C tempSensor XXXXXXXXX DCM_4A no 47 C tempSensor XXXXXXXXX DCM_4B no 43 C tempSensor XXXXXXXXX ESM_A no 43 C tempSensor XXXXXXXXX ESM_B no 44 C tempSensor XXXXXXXXX POWERSUPPLY_BOT no 43 C tempSensor SV55261173 POWERSUPPLY_TOP no 39 C ( serials, 2nd column, are intentionally hidden by me). Check your admin guide to map the sensor names (3rd columns) to the particular locations inside your enclosures (which include locations very close to the disk drives) Alvise ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of atmane khiredine [a.khiredine at meteo.dz] Sent: Monday, January 15, 2018 11:41 AM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] temperature disk gss Dear All, is there a way to display the temperature of the drives in GSS Thanks Everyone! Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From oehmes at gmail.com Mon Jan 15 15:00:17 2018 From: oehmes at gmail.com (Sven Oehme) Date: Mon, 15 Jan 2018 15:00:17 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: do you have a repeatable test that runs on just a small number of nodes ? we where working (still are) on some mmap enhancements lest year, but didn't get all of them ready for the DCUT of Scale 5.0, so we integrated only a subset that was ready into Scale 5.0 and left the enhancement turned off by default (you can toggle it on/off via mmchconfig). if you are interested in testing this send me a direct mail and i will provide instructions on how to turn this on. it does help mmap read workloads significant (think factors) in testing with synthetic benchmarks in my lab, but i would be very interested in real application results and also interested to get a trace with the new code to see where we need to make further improvements. sven On Mon, Jan 15, 2018 at 4:26 AM Ray Coetzee wrote: > Hi Sven > yes, it's primarily reads. > > Kind regards > > Ray Coetzee > Mob: +44 759 704 7060 <+44%207597%20047060> > > Skype: ray.coetzee > > Email: coetzee.ray at gmail.com > > > On Fri, Jan 12, 2018 at 8:57 PM, Sven Oehme wrote: > >> is this primary read or write ? >> >> >> On Fri, Jan 12, 2018, 12:51 PM Ray Coetzee wrote: >> >>> Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. >>> (Without the meltdown patch) >>> >>> Hey Bryan, I remember that quote from Yuri, that's why I hoped some >>> "magic" may have been done since then. >>> >>> Other attempts to improve performance I've tried include: >>> >>> - Using LROC to have a larger chance of a cache hit (Unfortunately >>> the entire dataset is multiple TB) >>> - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for >>> this purpose (Job runs halved in duration but nowhere near what NFS can >>> give) >>> - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, >>> and some others with little improvement. >>> >>> For those interested, as a performance comparison. The same job when run >>> on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe >>> scratch filesystem and over 60min on spindle based filesystem. >>> >>> Kind regards >>> >>> Ray Coetzee >>> Email: coetzee.ray at gmail.com >>> >>> >>> On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister < >>> bbanister at jumptrading.com> wrote: >>> >>>> You could put all of your data onto SSDs in a RAID1 configuration so >>>> that you don?t have insane read-modify-write penalties on writes (RAID1) >>>> and avoid horrible seek thrashing that spinning rust requires (SSD random >>>> access medium) for your 4K I/O operations. >>>> >>>> >>>> >>>> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best >>>> not to touch it?. He gave many reasons why mmap operations on a >>>> distributed file system is incredibly hard and not recommended. >>>> >>>> -Bryan >>>> >>>> >>>> >>>> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: >>>> gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >>>> *Sent:* Friday, January 12, 2018 8:45 AM >>>> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >>>> gpfsug-discuss at spectrumscale.org> >>>> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >>>> >>>> >>>> >>>> *Note: External Email* >>>> ------------------------------ >>>> >>>> what version of Scale are you using right now ? >>>> >>>> >>>> >>>> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >>>> wrote: >>>> >>>> I'd like to ask the group of their experiences in improving the >>>> performance of applications that use mmap calls against files on Spectrum >>>> Scale. >>>> >>>> >>>> >>>> Besides using an NFS export from CES instead of a native GPFS mount, or >>>> precaching the dataset into the pagepool, what other approaches are there >>>> to offset the performance hit of the 4K IO size? >>>> >>>> >>>> >>>> Kind regards >>>> >>>> Ray Coetzee >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>>> >>>> ------------------------------ >>>> >>>> Note: This email is for the confidential use of the named addressee(s) >>>> only and may contain proprietary, confidential or privileged information. >>>> If you are not the intended recipient, you are hereby notified that any >>>> review, dissemination or copying of this email is strictly prohibited, and >>>> to please notify the sender immediately and destroy this email and any >>>> attachments. Email transmission cannot be guaranteed to be secure or >>>> error-free. The Company, therefore, does not make any guarantees as to the >>>> completeness or accuracy of this email or any attachments. This email is >>>> for informational purposes only and does not constitute a recommendation, >>>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >>>> or perform any type of transaction of a financial product. >>>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Christian.Fey at sva.de Mon Jan 15 16:16:05 2018 From: Christian.Fey at sva.de (Fey, Christian) Date: Mon, 15 Jan 2018 16:16:05 +0000 Subject: [gpfsug-discuss] switching on perfileset-quota Message-ID: <05334d4771da49b4bb5f4cb43f1c3558@sva.de> Hi all, does anyone here have experience about a possible impact of switching on "--perfileset-quota" for an existing gpfs filesystem in order to achieve quotas on per-user basis? During my very small tests I did not discover anything, but I would like to be sure. The existing filesystem already has quotas in place that hopefully do not get messed up. Is there anything I need to consider? Mit freundlichen Gr??en / Best Regards Christian Fey SVA System Vertrieb Alexander GmbH Borsigstra?e 14 65205 Wiesbaden Tel.: +49 6122 536-0 Fax: +49 6122 536-399 E-Mail: christian.fey at sva.de http://www.sva.de Gesch?ftsf?hrung: Philipp Alexander, Sven Eichelbaum Sitz der Gesellschaft: Wiesbaden Registergericht: Amtsgericht Wiesbaden, HRB 10315 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5415 bytes Desc: not available URL: From pierre-marie.brunet at cnes.fr Tue Jan 16 10:40:46 2018 From: pierre-marie.brunet at cnes.fr (Brunet Pierre-Marie) Date: Tue, 16 Jan 2018 10:40:46 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint Message-ID: Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I'm talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Tue Jan 16 12:41:28 2018 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 16 Jan 2018 12:41:28 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint Message-ID: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com> Hi PMB This is one of the areas where even the most experienced admins struggle. There is no single answer here. Much of it depends on your particular use case and the file system layout, storage choices, block sizes all go together. IBM has started to provide some templates for this (the one out is for Genomics) but much is left to do. If you could share some details on the overall user environment and data myself and others could offer some ideas. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Brunet Pierre-Marie Reply-To: gpfsug main discussion list Date: Tuesday, January 16, 2018 at 4:51 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] GPFS best practises : end user standpoint Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I?m talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jan 16 13:03:06 2018 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Tue, 16 Jan 2018 13:03:06 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com> References: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com> Message-ID: Aaron Knister liked your message with Boxer. On January 16, 2018 at 07:41:40 EST, Oesterlin, Robert wrote: Hi PMB This is one of the areas where even the most experienced admins struggle. There is no single answer here. Much of it depends on your particular use case and the file system layout, storage choices, block sizes all go together. IBM has started to provide some templates for this (the one out is for Genomics) but much is left to do. If you could share some details on the overall user environment and data myself and others could offer some ideas. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Brunet Pierre-Marie Reply-To: gpfsug main discussion list Date: Tuesday, January 16, 2018 at 4:51 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] GPFS best practises : end user standpoint Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I?m talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jan 16 13:47:19 2018 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Tue, 16 Jan 2018 13:47:19 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com>, Message-ID: <86593829-3D51-4203-98E0-E22B0A727C87@nasa.gov> Apologies for that. My mobile exchange email client has a like button you can tap in the email action menu. I just discovered it this morning accidentally and thought ?wonder what this does. Better push it to find out.? Nothing happened or so I thought. Apparently all it does is make you look like a moron on public mailing lists. Doh! On January 16, 2018 at 08:03:06 EST, Aaron Knister wrote: Aaron Knister liked your message with Boxer. On January 16, 2018 at 07:41:40 EST, Oesterlin, Robert wrote: Hi PMB This is one of the areas where even the most experienced admins struggle. There is no single answer here. Much of it depends on your particular use case and the file system layout, storage choices, block sizes all go together. IBM has started to provide some templates for this (the one out is for Genomics) but much is left to do. If you could share some details on the overall user environment and data myself and others could offer some ideas. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Brunet Pierre-Marie Reply-To: gpfsug main discussion list Date: Tuesday, January 16, 2018 at 4:51 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] GPFS best practises : end user standpoint Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I?m talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Tue Jan 16 15:47:54 2018 From: carlz at us.ibm.com (Carl Zetie) Date: Tue, 16 Jan 2018 15:47:54 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Tue Jan 16 16:08:15 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 16 Jan 2018 16:08:15 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: Message-ID: <1516118895.3326.25.camel@strath.ac.uk> On Tue, 2018-01-16 at 15:47 +0000, Carl Zetie wrote: > Maybe this would make for a good session at a future user group > meeting -- perhaps as an interactive session? IBM could potentially > provide a facilitator from our Design practice. > ? Most of it in my view is standard best practice regardless of the file system in use. So in our mandatory training for the HPC, we tell our users don't use whacked out characters in your file names and directories. Specifically no backticks, no asterik's, no question marks, no newlines (yes really), no slashes (either forward or backward) and for Mac users don't start the name with a space (forces sorting to the top). We recommend sticking to plain ASCII so no accented characters either (harder if your native language is not English I guess but we are UK based so...). We don't enforce that but if it causes the user problems then they are on their own. We also strongly recommend using ISO 8601 date formats in file names to get date sorting from a directory listing too. Surprisingly not widely known about, but a great "life hack". Then it boils down to don't create zillions of files. I would love to be able to somehow do per directory file number quota's where one could say set a default of a few thousand. Users would then have to justify needing a larger quota. Sure you can set a file number quota but that does not stop them putting them all in one directory. If users really need to have zillions of files then charge them more so you can afford to beef up your metadata disks to SSD. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 16 16:35:05 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 16 Jan 2018 16:35:05 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516118895.3326.25.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> Message-ID: Hi Jonathan, Comments / questions inline. Thanks! Kevin > On Jan 16, 2018, at 10:08 AM, Jonathan Buzzard wrote: > > On Tue, 2018-01-16 at 15:47 +0000, Carl Zetie wrote: >> Maybe this would make for a good session at a future user group >> meeting -- perhaps as an interactive session? IBM could potentially >> provide a facilitator from our Design practice. >> > > Most of it in my view is standard best practice regardless of the file > system in use. > > So in our mandatory training for the HPC, we tell our users don't use > whacked out characters in your file names and directories. Specifically > no backticks, no asterik's, no question marks, no newlines (yes > really), no slashes (either forward or backward) and for Mac users > don't start the name with a space (forces sorting to the top). We > recommend sticking to plain ASCII so no accented characters either > (harder if your native language is not English I guess but we are UK > based so...). We don't enforce that but if it causes the user problems > then they are on their own. We?re in Tennessee, so not only do we not speak English, we barely speak American ? y?all will just have to understand, bless your hearts! ;-). But seriously, like most Universities, we have a ton of users for whom English is not their ?primary? language, so dealing with ?interesting? filenames is pretty hard to avoid. And users? problems are our problems whether or not they?re our problem. > > We also strongly recommend using ISO 8601 date formats in file names to > get date sorting from a directory listing too. Surprisingly not widely > known about, but a great "life hack". > > Then it boils down to don't create zillions of files. I would love to > be able to somehow do per directory file number quota's where one could > say set a default of a few thousand. Users would then have to justify > needing a larger quota. Sure you can set a file number quota but that > does not stop them putting them all in one directory. If you?ve got (bio)medical users using your cluster I don?t see how you avoid this ? they?re using commercial apps that do this kind of stupid stuff (10?s of thousands of files in a directory and the full path to each file is longer than the contents of the files themselves!). This reminds me of way back in 2005 when we moved from an NFS server to GPFS ? I was moving users over by tarring up their home directories on the NFS server, copying the tarball over to GPFS and untarring it there ? worked great for 699 out of 700 users. But there was one user for whom the untar would fail every time I tried ? turned out that back in early versions of GPFS 2.3 IBM hadn?t considered that someone would put 6 million files in one directory! :-O > > If users really need to have zillions of files then charge them more so > you can afford to beef up your metadata disks to SSD. OK, so here?s my main question ? you?re right that SSD?s are the answer ? but how do you charge them more? SSDs are move expensive than hard disks, and enterprise SSDs are stupid expensive ? and users barely want to pay hard drive prices for their storage. If you?ve got the magic answer to how to charge them enough to pay for SSDs I?m sure I?m not the only one who?d love to hear how you do it?!?! > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cdd3310fd309b4986f95c08d55cfb5d10%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636517157039256068&sdata=jZZV718gaMie92MW43qaxlDl6EQcULdk6FONrXpsP7c%3D&reserved=0 From heinz.ernst at id.ethz.ch Tue Jan 16 16:56:52 2018 From: heinz.ernst at id.ethz.ch (Ernst Heinz (ID SD)) Date: Tue, 16 Jan 2018 16:56:52 +0000 Subject: [gpfsug-discuss] GPFS GA 5.0.0.0: mmces commands with inconsistent output Message-ID: <0CE1E9F220AF564DB0114D8539010F9B5229A68C@MBX116.d.ethz.ch> Hello to all peers and gurus Since more or less two weeks we have gpfs GA 5.0.0.0 running on our testenvironment Today I've seen following behavior on our SpectrumScale-testcluster which slighdly surprised me Following: Checking status of the cluster on different ways [root at testnas13ces01 idsd_erh_t1]# mmces state cluster CLUSTER AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13.ethz.ch FAILED DISABLED HEALTHY DISABLED DEPEND DISABLED DEPEND FAILED [root at testnas13ces01 idsd_erh_t1]# mmces state show -a NODE AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13ces01-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY testnas13ces02-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY does anyone of you guys has an explanation therefore? Is there someone else who has seen a behavior like this? By the way we have a similar view on one of our clusters on gpfs 4.2.3.4 (open PMR: 30218.112.848) Any kind of response would be very grateful Kind regards Heinz =============================================================== Heinz Ernst ID-Systemdienste WEC C 16 Weinbergstrasse 11 CH-8092 Zurich heinz.ernst at id.ethz.ch Phone: +41 44 633 84 48 Mobile: +41 79 216 15 50 =============================================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Tue Jan 16 17:25:47 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 16 Jan 2018 17:25:47 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: <1516118895.3326.25.camel@strath.ac.uk> Message-ID: <1516123547.3326.29.camel@strath.ac.uk> On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: [SNIP] > > We?re in Tennessee, so not only do we not speak English, we barely > speak American ? y?all will just have to understand, bless your > hearts!??;-).? > > But seriously, like most Universities, we have a ton of users for > whom English is not their ?primary? language, so dealing with > ?interesting? filenames is pretty hard to avoid.??And users? problems > are our problems whether or not they?re our problem. > User comes with problem, you investigate find problem is due to "wacky" characters point them to the mandatory training documentation, tell them they need to rename their files to something sane and take no further action. Sure English is not their primary language but *they* have chosen to study in an English speaking country so best to actually embrace that. I do get it, many of our users are not native English speakers as well. Yes it's a tough policy but on the other hand pandering to them does them no favours either. [SNIP] > If you?ve got (bio)medical users using your cluster I don?t see how > you avoid this ? they?re using commercial apps that do this kind of > stupid stuff (10?s of thousands of files in a directory and the full > path to each file is longer than the contents of the files > themselves!). Well then they have justified the use; aka it's not their fault so you up the quota for them. Though they could use different less brain dead software. The idea is to force a bump in the road so the users are aware that what they are doing is considered bad practice. Most users have no idea that putting a million files in a directory is not sensible and worse that trying to access them using a GUI file manager is positively brain dead. [SNIP] > OK, so here?s my main question ? you?re right that SSD?s are the > answer ? but how do you charge them more???SSDs are move expensive > than hard disks, and enterprise SSDs are stupid expensive ? and users > barely want to pay hard drive prices for their storage.??If you?ve > got the magic answer to how to charge them enough to pay for SSDs I?m > sure I?m not the only one who?d love to hear how you do it?!?! > Give every user a one million file number quota. Need to store more than one million files, then you are going to have to pay $X per extra million files. Either they cough up the money to continue using their brain dead software or they switch to less stupid software. If they complain you just say that enterprise SSD's are stupidly expensive and you are using that space up at an above average rate and so have to pay the costs. I am quite sure someone storing 1PB has to pay more than someone storing 1TB, so why should someone storing 20 million files not have to pay more than someone storing 100k files? The only difference is people are used to paying more to store extra bytes and not used to paying more for more files, but that is because most sane people don't store millions and millions of files necessitating the purchase of large amounts of expensive enterprise SSD's. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From valdis.kletnieks at vt.edu Wed Jan 17 11:35:58 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 17 Jan 2018 06:35:58 -0500 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516123547.3326.29.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> Message-ID: <72644.1516188958@turing-police.cc.vt.edu> On Tue, 16 Jan 2018 17:25:47 +0000, Jonathan Buzzard said: > User comes with problem, you investigate find problem is due to "wacky" > characters point them to the mandatory training documentation, tell > them they need to rename their files to something sane and take no > further action. Sure English is not their primary language but *they* > have chosen to study in an English speaking country so best to actually > embrace that. > I do get it, many of our users are not native English speakers as well. > Yes it's a tough policy but on the other hand pandering to them does > them no favours either. Go ahead and try to make that policy stick in Tokyo or Cairo or Berlin or any other city or country where English isn't the primary language. Seriously - this is something that IBM has to make work well across all languages if they want to make this fly globally and not just in places that are English-predominant. Though to be honest, most of the problems I've had have been with embedded blanks in filenames, newline characters in filenames (fortunately only a half dozen or so out of almost 500M files), and esc-[-random from people banging on F8 instead of 8, etc. Have the occasional backspace character creep in too. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From jonathan.buzzard at strath.ac.uk Wed Jan 17 13:48:14 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 17 Jan 2018 13:48:14 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <72644.1516188958@turing-police.cc.vt.edu> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> <72644.1516188958@turing-police.cc.vt.edu> Message-ID: <1516196894.3326.37.camel@strath.ac.uk> On Wed, 2018-01-17 at 06:35 -0500, valdis.kletnieks at vt.edu wrote: > On Tue, 16 Jan 2018 17:25:47 +0000, Jonathan Buzzard said: > > User comes with problem, you investigate find problem is due to > > "wacky" > > characters point them to the mandatory training documentation, tell > > them they need to rename their files to something sane and take no > > further action. Sure English is not their primary language but > > *they* > > have chosen to study in an English speaking country so best to > > actually > > embrace that. > > I do get it, many of our users are not native English speakers as > > well. > > Yes it's a tough policy but on the other hand pandering to them > > does > > them no favours either. > > Go ahead and try to make that policy stick in Tokyo or Cairo or > Berlin or any other city or country where English isn't the primary > language. Sure hard, but not *my* problem :-) Though to be fair missing accent's off is not that bad. Obviously if your language does not use the Roman alphabet then things are somewhat more serious. Note I am in the UK and you just don't use a ? in your file name (if your sensible) so I really do get it. > > Seriously - this is something that IBM has to make work well across > all languages if they want to make this fly globally and not just in > places that are English-predominant. Oh GPFS works fine; well unless it is ancient versions of mmbackup in which case they just cause your backup to be aborted without doing anything! It's all the other end user programs that, especially end user scripts that cause problems. So either we advise users that special characters might cause them problems and best to not use them in the first place, or wait till they have a problem and then tell them they need to rename some thousands of files. In sort it's not IBM's problem other than they are one of the major culprits who helped create this mess back in the day :-) > Though to be honest, most of the problems I've had have been with > embedded blanks in filenames, newline characters in filenames > (fortunately only a half dozen or so out of almost 500M files), and > esc-[-random from people banging on F8 instead of 8, etc.? Have the > occasional backspace character creep in too. The mind boggles how you manage to get a backspace character into a file name. A new line is bad enough, but a backspace!!! To be honest its more this sort of nonsense than none ASCII characters that cause the problems. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From valdis.kletnieks at vt.edu Wed Jan 17 14:43:05 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 17 Jan 2018 09:43:05 -0500 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516196894.3326.37.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> <72644.1516188958@turing-police.cc.vt.edu> <1516196894.3326.37.camel@strath.ac.uk> Message-ID: <8491.1516200185@turing-police.cc.vt.edu> On Wed, 17 Jan 2018 13:48:14 +0000, Jonathan Buzzard said: > The mind boggles how you manage to get a backspace character into a > file name. You can get yourself into all sorts of trouble with stty - in particular, if you're ssh'ed from one system to another, and they disagree on whether the "make the last character go away" key is Delete or Backspace. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jan 17 16:56:42 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 17 Jan 2018 16:56:42 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516123547.3326.29.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> Message-ID: <38F290EB-D2DD-40E8-910E-C0C9C2812E1E@vanderbilt.edu> Inline? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 On Jan 16, 2018, at 11:25 AM, Jonathan Buzzard > wrote: On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: [SNIP] I am quite sure someone storing 1PB has to pay more than someone storing 1TB, so why should someone storing 20 million files not have to pay more than someone storing 100k files? Because they won?t ? they?ll do something more brain dead like put a WD MyBook they bought at Costco on their desk and expect their job to copy data back and forth from it to /tmp on the compute node. We have to offer a service that users are willing to pay for ? we can?t dictate to them the way things WILL be. There?s a big difference between the way things should be and the way things actually are ? trust me, those of us in the United States know that better than most people around the world after the past year! Bigly! Buh-leave ME! :-O JAB. -------------- next part -------------- An HTML attachment was scrubbed... URL: From MDIETZ at de.ibm.com Wed Jan 17 17:08:02 2018 From: MDIETZ at de.ibm.com (Mathias Dietz) Date: Wed, 17 Jan 2018 18:08:02 +0100 Subject: [gpfsug-discuss] GPFS GA 5.0.0.0: mmces commands with inconsistent output In-Reply-To: <0CE1E9F220AF564DB0114D8539010F9B5229A68C@MBX116.d.ethz.ch> References: <0CE1E9F220AF564DB0114D8539010F9B5229A68C@MBX116.d.ethz.ch> Message-ID: Hi, let me start with a recommendation first before I explain how the cluster state is build. Starting with 4.2.1 please use the mmhealth command instead of using the mmces state/events command. The mmces state/event command will be deprecated in future releases. mmhealth node show -> show the node state for all components (incl. CES) mmhealth node show CES -> shows the CES components only. mmhealth cluster show -> show the cluster state Now to your problem: The Spectrum Scale health monitoring is done by a daemon which runs on each cluster node. This daemon is monitoring the state of all Spectrum Scale components on the local system and based on the resulting monitoring events it compiles a local system state (shown by mmhealth node show). By having a decentralized monitoring we reduce the monitoring overhead and increase resiliency against network glitches. In order to show a cluster wide state view we have to consolidate the events from all cluster nodes on a single node. The health monitoring daemon running on the cluster manager is taking the role (CSM) to receive events from all nodes through RPC calls and to compile the cluster state (shown by mmhealth cluster show) There can be cases where the (async) event forwarding to the CSM is delayed or dropped because of network delays, high system load, cluster manager failover or split brain cases. Those cases should resolve automatically after some time when event is resend. Summary: the cluster state might be temporary out of sync (eventually consistent), for getting a current state you should refer to mmhealth node show. If the problem does not resolve automatically, restarting the monitoring daemon will force a re-sync. Please open a PMR for the 5.0 issue too if the problem persist. Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale Development - Release Lead Architect (4.2.x) Spectrum Scale RAS Architect --------------------------------------------------------------------------- IBM Deutschland Am Weiher 24 65451 Kelsterbach Phone: +49 70342744105 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ----------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk WittkoppSitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Ernst Heinz (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Date: 01/16/2018 06:09 PM Subject: [gpfsug-discuss] GPFS GA 5.0.0.0: mmces commands with inconsistent output Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello to all peers and gurus Since more or less two weeks we have gpfs GA 5.0.0.0 running on our testenvironment Today I?ve seen following behavior on our SpectrumScale-testcluster which slighdly surprised me Following: Checking status of the cluster on different ways [root at testnas13ces01 idsd_erh_t1]# mmces state cluster CLUSTER AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13.ethz.ch FAILED DISABLED HEALTHY DISABLED DEPEND DISABLED DEPEND FAILED [root at testnas13ces01 idsd_erh_t1]# mmces state show -a NODE AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13ces01-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY testnas13ces02-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY does anyone of you guys has an explanation therefore? Is there someone else who has seen a behavior like this? By the way we have a similar view on one of our clusters on gpfs 4.2.3.4 (open PMR: 30218.112.848) Any kind of response would be very grateful Kind regards Heinz =============================================================== Heinz Ernst ID-Systemdienste WEC C 16 Weinbergstrasse 11 CH-8092 Zurich heinz.ernst at id.ethz.ch Phone: +41 44 633 84 48 Mobile: +41 79 216 15 50 =============================================================== _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Wed Jan 17 17:31:39 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 17 Jan 2018 17:31:39 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <38F290EB-D2DD-40E8-910E-C0C9C2812E1E@vanderbilt.edu> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> <38F290EB-D2DD-40E8-910E-C0C9C2812E1E@vanderbilt.edu> Message-ID: <1516210299.3326.43.camel@strath.ac.uk> On Wed, 2018-01-17 at 16:56 +0000, Buterbaugh, Kevin L wrote: > Inline? > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu?- (615)875-9633 > > > On Jan 16, 2018, at 11:25 AM, Jonathan Buzzard > rath.ac.uk> wrote: > > > > On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: > > > > [SNIP] > > > > I am quite sure someone storing 1PB has to pay more than someone > > storing 1TB, so why should someone storing 20 million files not > > have to pay more than someone storing 100k files?? > > > ? > Because they won?t ? they?ll do something more brain dead like put a > WD MyBook they bought at Costco on their desk and expect their job to > copy data back and forth from it to /tmp on the compute node. ?We > have to offer a service that users are willing to pay for ? we can?t > dictate to them the way things WILL be. However users need to be willing to pay for something that works. As they say cheap, fast and resilient pick any two. Getting back to your scenario, their performance will suck assuming it would work in the first place and secondly it's not your problem any more :-) > There?s a big difference between the way things should be and the way > things actually are ? trust me, those of us in the United States know > that better than most people around the world after the past year! > ?Bigly! ?Buh-leave ME! ?:-O > In my experience part of the problem is system administrators *not* pushing back a bit to users when they want something unreasonable. If you don't all you do is raise a generation of spoilt children. Secondly in my experience when you do push back most users are reasonable and understanding when it's explained to them in a manner they understand that what they are trying to do is not sensible. Oh and for the record I have been right their and had to put a file quota on a user that effectively stopped them dead in their tracks because half the files on the file system where from that one user, and everyone else was complaining that performance was going down the tube. I relevant example is a research group in my university where too cheap to by a proper file server so brought a large NAS box instead from a well known vendor. Just before Christmas the NAS box popped, it's still not replaced yet. Said research group have put their hands in their pockets for a server from a tier one vendor with a 4hr response maintenance plan. Funny that :-) JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From makaplan at us.ibm.com Wed Jan 17 20:10:45 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 17 Jan 2018 17:10:45 -0300 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: <1516118895.3326.25.camel@strath.ac.uk> Message-ID: Yes, "special" characters in pathnames can lead to trouble... But just for the record... GPFS supports the same liberal file name policy as standard POSIX. Namely any bytestring is valid, except: / delimits directory names The \0 (zero or Null Character) byte value marks the end of the pathname. There are limits on the length of an individual file or directory name. AND there is an OS imposed limit on the total length of a pathname you can pass through the file system APIs. From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/16/2018 01:58 PM Subject: Re: [gpfsug-discuss] GPFS best practises : end user standpoint Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Jonathan, Comments / questions inline. Thanks! Kevin > On Jan 16, 2018, at 10:08 AM, Jonathan Buzzard wrote: > > On Tue, 2018-01-16 at 15:47 +0000, Carl Zetie wrote: >> Maybe this would make for a good session at a future user group >> meeting -- perhaps as an interactive session? IBM could potentially >> provide a facilitator from our Design practice. >> > > Most of it in my view is standard best practice regardless of the file > system in use. > > So in our mandatory training for the HPC, we tell our users don't use > whacked out characters in your file names and directories. Specifically > no backticks, no asterik's, no question marks, no newlines (yes > really), no slashes (either forward or backward) and for Mac users > don't start the name with a space (forces sorting to the top). We > recommend sticking to plain ASCII so no accented characters either > (harder if your native language is not English I guess but we are UK > based so...). We don't enforce that but if it causes the user problems > then they are on their own. We?re in Tennessee, so not only do we not speak English, we barely speak American ? y?all will just have to understand, bless your hearts! ;-). But seriously, like most Universities, we have a ton of users for whom English is not their ?primary? language, so dealing with ?interesting? filenames is pretty hard to avoid. And users? problems are our problems whether or not they?re our problem. > > We also strongly recommend using ISO 8601 date formats in file names to > get date sorting from a directory listing too. Surprisingly not widely > known about, but a great "life hack". > > Then it boils down to don't create zillions of files. I would love to > be able to somehow do per directory file number quota's where one could > say set a default of a few thousand. Users would then have to justify > needing a larger quota. Sure you can set a file number quota but that > does not stop them putting them all in one directory. If you?ve got (bio)medical users using your cluster I don?t see how you avoid this ? they?re using commercial apps that do this kind of stupid stuff (10?s of thousands of files in a directory and the full path to each file is longer than the contents of the files themselves!). This reminds me of way back in 2005 when we moved from an NFS server to GPFS ? I was moving users over by tarring up their home directories on the NFS server, copying the tarball over to GPFS and untarring it there ? worked great for 699 out of 700 users. But there was one user for whom the untar would fail every time I tried ? turned out that back in early versions of GPFS 2.3 IBM hadn?t considered that someone would put 6 million files in one directory! :-O > > If users really need to have zillions of files then charge them more so > you can afford to beef up your metadata disks to SSD. OK, so here?s my main question ? you?re right that SSD?s are the answer ? but how do you charge them more? SSDs are move expensive than hard disks, and enterprise SSDs are stupid expensive ? and users barely want to pay hard drive prices for their storage. If you?ve got the magic answer to how to charge them enough to pay for SSDs I?m sure I?m not the only one who?d love to hear how you do it?!?! > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fgpfsug.org-252Fmailman-252Flistinfo-252Fgpfsug-2Ddiscuss-26data-3D02-257C01-257CKevin.Buterbaugh-2540vanderbilt.edu-257Cdd3310fd309b4986f95c08d55cfb5d10-257Cba5a7f39e3be4ab3b45067fa80faecad-257C0-257C1-257C636517157039256068-26sdata-3DjZZV718gaMie92MW43qaxlDl6EQcULdk6FONrXpsP7c-253D-26reserved-3D0&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=7n72wm4bwHfrK-yGlxSHLkVEIq0FDXA7XrPI_pyQq1M&s=1cYF3dt9odnG5zCHjcxMGl9_LbVAVNFrFu1iuv5585U&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=7n72wm4bwHfrK-yGlxSHLkVEIq0FDXA7XrPI_pyQq1M&s=_puCWAGyopu4m3M7evNjbILg3LkDCmiI9vJN2IG1iBE&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Wed Jan 17 22:12:01 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 17 Jan 2018 22:12:01 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Message-ID: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn't say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to 'generic' not 'dmm'. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with 'generic' vs. 'dmm' vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Wed Jan 17 08:39:35 2018 From: john.hearns at asml.com (John Hearns) Date: Wed, 17 Jan 2018 08:39:35 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516123547.3326.29.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> Message-ID: My own thoughts on this one are that I believe when software is being developed, the developers use 'small' use cases. At the company which develops the software, the developers will probably have a desktop machine with a modest amount of RAM and disk space. Then the company might have a small to medium sized HPC cluster. I know I am stretching things a bit, but I would imagine a lot of effort goes into verifying the correct operation of software on given data sets. When the software is released to customers, either commercially or internally within a company, it is suddenly applied to larger datasets, and is applied repetitively. Hence the creation of directories filled with thousands of small files. My own example from this here is in a former job, wind tunnel data was captured as thousands of PNG files which were frames from a camera. The data was shipped back to me on a hard drive, and I was asked to store it on an HSM system with tape as the lowest tier. I knew that the PNG files had all bene processed anyway, and significant data had been extracted. The engineers wanted to keep the data 'just in case'. I knew that keeping thousands of files is bad for filesystem performance and also on a tape based system you can have the fiels stored on many tapes, so if you ever do trigger a recall you have a festival of robot tape loading. So what I did was zip up all the directories. -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jonathan Buzzard Sent: Tuesday, January 16, 2018 6:26 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPFS best practises : end user standpoint On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: [SNIP] > > We?re in Tennessee, so not only do we not speak English, we barely > speak American ? y?all will just have to understand, bless your > hearts! ;-). > > But seriously, like most Universities, we have a ton of users for whom > English is not their ?primary? language, so dealing with ?interesting? > filenames is pretty hard to avoid. And users? problems are our > problems whether or not they?re our problem. > User comes with problem, you investigate find problem is due to "wacky" characters point them to the mandatory training documentation, tell them they need to rename their files to something sane and take no further action. Sure English is not their primary language but *they* have chosen to study in an English speaking country so best to actually embrace that. I do get it, many of our users are not native English speakers as well. Yes it's a tough policy but on the other hand pandering to them does them no favours either. [SNIP] > If you?ve got (bio)medical users using your cluster I don?t see how > you avoid this ? they?re using commercial apps that do this kind of > stupid stuff (10?s of thousands of files in a directory and the full > path to each file is longer than the contents of the files > themselves!). Well then they have justified the use; aka it's not their fault so you up the quota for them. Though they could use different less brain dead software. The idea is to force a bump in the road so the users are aware that what they are doing is considered bad practice. Most users have no idea that putting a million files in a directory is not sensible and worse that trying to access them using a GUI file manager is positively brain dead. [SNIP] > OK, so here?s my main question ? you?re right that SSD?s are the > answer ? but how do you charge them more? SSDs are move expensive > than hard disks, and enterprise SSDs are stupid expensive ? and users > barely want to pay hard drive prices for their storage. If you?ve got > the magic answer to how to charge them enough to pay for SSDs I?m sure > I?m not the only one who?d love to hear how you do it?!?! > Give every user a one million file number quota. Need to store more than one million files, then you are going to have to pay $X per extra million files. Either they cough up the money to continue using their brain dead software or they switch to less stupid software. If they complain you just say that enterprise SSD's are stupidly expensive and you are using that space up at an above average rate and so have to pay the costs. I am quite sure someone storing 1PB has to pay more than someone storing 1TB, so why should someone storing 20 million files not have to pay more than someone storing 100k files? The only difference is people are used to paying more to store extra bytes and not used to paying more for more files, but that is because most sane people don't store millions and millions of files necessitating the purchase of large amounts of expensive enterprise SSD's. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7Cf9b43f106c124ced6a4108d55d063196%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=c5B3JAJZDp3YiCN2uOzTmf%2BlsLMVRw6BsIzacQuORN8%3D&reserved=0 -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. From jjdoherty at yahoo.com Thu Jan 18 00:28:06 2018 From: jjdoherty at yahoo.com (Jim Doherty) Date: Thu, 18 Jan 2018 00:28:06 +0000 (UTC) Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> Message-ID: <1743984571.54681.1516235286972@mail.yahoo.com> Run a mmlsnsd -X ? I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN,?? not the /dev/mapper/ path.?? In our case the device is setup as dmm? [root at service5 ~]# mmlsnsd -X ?Disk name??? NSD volume ID????? Device???????? Devtype? Node name??????????????? Remarks???????? ? --------------------------------------------------------------------------------------------------- ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service5.pok.stglabs.ibm.com server node ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service6.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-4????? dmm????? service5.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-3????? dmm????? service6.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-1????? dmm????? service5.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-2????? dmm????? service6.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-3????? dmm????? service5.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-4????? dmm????? service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s? it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic ? 0972B6CD587CD8E0 /dev/sdk generic ? 0972B6CD587CD8E0 /dev/sdu generic ? 0972B6CD587CD8E0 /dev/sdah generic ? 0972B6CD587CD8E0 /dev/dm-0 dmm ? [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister wrote: Hi all, ? We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. ? The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: ? # Known disk types currently are: # #?? powerdisk? - EMC power path disk #?? vpath????? - IBM virtual path disk #?? dmm??????? - Device-Mapper Multipath (DMM) #?? dlmfdrv??? - Hitachi dlm #?? hdisk????? - AIX hard disk #?? lv???????? - AIX logical volume.? Historical usage only. #??????????????? Not allowed as a new device to mmcrnsd. #?? gpt??????? - GPFS partition on Windows disk #?? generic??? - Device having no unique failover or multipathing #??????????????? characteristic (predominantly Linux devices). #?? dasd?????? - DASD device (for Linux on z Systems) ? We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?.? This is configured in the /var/mmfs/etc/nsddevices file: ? if [[ $osName = Linux ]] then ? : # Add function to discover disks in the Linux environment. ? ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ? ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 "generic"}' ? ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi ? Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? ? Thanks in advance! -Bryan Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Thu Jan 18 15:09:22 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 18 Jan 2018 15:09:22 +0000 Subject: [gpfsug-discuss] Kerberos NFS Message-ID: Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Jan 18 15:49:15 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 15:49:15 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <1743984571.54681.1516235286972@mail.yahoo.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: <38f69158067c4b7495d525b9aba161ff@jumptrading.com> Hi Jim, Our NSDs are reporting the /dev/mapper/ devices in the `mmlsnsd -X` output, but due to the nsddevices file configuration they are all configured as ?generic? Devtype. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjdoherty at yahoo.com Thu Jan 18 16:32:58 2018 From: jjdoherty at yahoo.com (Jim Doherty) Date: Thu, 18 Jan 2018 16:32:58 +0000 (UTC) Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <38f69158067c4b7495d525b9aba161ff@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <38f69158067c4b7495d525b9aba161ff@jumptrading.com> Message-ID: <244901478.616245.1516293178108@mail.yahoo.com> What does the /var/mmfs/gen/mmsdrfs file show? [root at service5 ~]# grep SG_DISK /var/mmfs/gen/mmsdrfs | cut -f 15 -d ':' dmm dmm dmm dmm On Thursday, January 18, 2018, 10:49:32 AM EST, Bryan Banister wrote: #yiv4648160129 #yiv4648160129 -- _filtered #yiv4648160129 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv4648160129 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv4648160129 {font-family:Verdana;panose-1:2 11 6 4 3 5 4 4 2 4;} _filtered #yiv4648160129 {}#yiv4648160129 #yiv4648160129 p.yiv4648160129MsoNormal, #yiv4648160129 li.yiv4648160129MsoNormal, #yiv4648160129 div.yiv4648160129MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv4648160129 a:link, #yiv4648160129 span.yiv4648160129MsoHyperlink {color:blue;text-decoration:underline;}#yiv4648160129 a:visited, #yiv4648160129 span.yiv4648160129MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv4648160129 p.yiv4648160129msonormal, #yiv4648160129 li.yiv4648160129msonormal, #yiv4648160129 div.yiv4648160129msonormal {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 p.yiv4648160129msochpdefault, #yiv4648160129 li.yiv4648160129msochpdefault, #yiv4648160129 div.yiv4648160129msochpdefault {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 span.yiv4648160129msohyperlink {}#yiv4648160129 span.yiv4648160129msohyperlinkfollowed {}#yiv4648160129 span.yiv4648160129emailstyle17 {}#yiv4648160129 p.yiv4648160129msonormal1, #yiv4648160129 li.yiv4648160129msonormal1, #yiv4648160129 div.yiv4648160129msonormal1 {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;}#yiv4648160129 span.yiv4648160129msohyperlink1 {color:#0563C1;text-decoration:underline;}#yiv4648160129 span.yiv4648160129msohyperlinkfollowed1 {color:#954F72;text-decoration:underline;}#yiv4648160129 span.yiv4648160129emailstyle171 {color:windowtext;}#yiv4648160129 p.yiv4648160129msochpdefault1, #yiv4648160129 li.yiv4648160129msochpdefault1, #yiv4648160129 div.yiv4648160129msochpdefault1 {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 span.yiv4648160129EmailStyle28 {color:#1F497D;}#yiv4648160129 .yiv4648160129MsoChpDefault {font-size:10.0pt;} _filtered #yiv4648160129 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv4648160129 div.yiv4648160129WordSection1 {}#yiv4648160129 Hi Jim, ? Our NSDs are reporting the /dev/mapper/ devices in the `mmlsnsd -X` output, but due to the nsddevices file configuration they are all configured as ?generic? Devtype. -Bryan ? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org]On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file ? Note: External Email ? Run a mmlsnsd -X ? I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN,?? not the /dev/mapper/ path.?? In our case the device is setup as dmm? ? [root at service5 ~]# mmlsnsd -X ?Disk name??? NSD volume ID????? Device???????? Devtype? Node name??????????????? Remarks???????? ? --------------------------------------------------------------------------------------------------- ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service5.pok.stglabs.ibm.com server node ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service6.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-4????? dmm????? service5.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-3????? dmm????? service6.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-1????? dmm????? service5.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-2????? dmm????? service6.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-3????? dmm????? service5.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-4????? dmm????? service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# ? If you run an tspreparedisk -s? it will show you all of the paths. ? [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic ? 0972B6CD587CD8E0 /dev/sdk generic ? 0972B6CD587CD8E0 /dev/sdu generic ? 0972B6CD587CD8E0 /dev/sdah generic ? 0972B6CD587CD8E0 /dev/dm-0 dmm ? [root at service5 ~]# ? Jim ? ? ? Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister wrote: ? ? Hi all, ? We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. ? The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: ? # Known disk types currently are: # #?? powerdisk? - EMC power path disk #?? vpath????? - IBM virtual path disk #?? dmm??????? - Device-Mapper Multipath (DMM) #?? dlmfdrv??? - Hitachi dlm #?? hdisk????? - AIX hard disk #?? lv???????? - AIX logical volume.? Historical usage only. #??????????????? Not allowed as a new device to mmcrnsd. #?? gpt??????? - GPFS partition on Windows disk #?? generic??? - Device having no unique failover or multipathing #??????????????? characteristic (predominantly Linux devices). #?? dasd?????? - DASD device (for Linux on z Systems) ? We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?.? This is configured in the /var/mmfs/etc/nsddevices file: ? if [[ $osName = Linux ]] then ? : # Add function to discover disks in the Linux environment. ? ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ? ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 "generic"}' ? ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi ? Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? ? Thanks in advance! -Bryan ? Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Jan 18 16:35:23 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 16:35:23 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <244901478.616245.1516293178108@mail.yahoo.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <38f69158067c4b7495d525b9aba161ff@jumptrading.com> <244901478.616245.1516293178108@mail.yahoo.com> Message-ID: <9715f09cbdc74465ac2ba7cd883c9a8e@jumptrading.com> Generic: # grep SG_DISK /var/mmfs/gen/mmsdrfs | cut -f 15 -d ':' generic generic generic generic generic Cheers, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Thursday, January 18, 2018 10:33 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ What does the /var/mmfs/gen/mmsdrfs file show? [root at service5 ~]# grep SG_DISK /var/mmfs/gen/mmsdrfs | cut -f 15 -d ':' dmm dmm dmm dmm On Thursday, January 18, 2018, 10:49:32 AM EST, Bryan Banister > wrote: Hi Jim, Our NSDs are reporting the /dev/mapper/ devices in the `mmlsnsd -X` output, but due to the nsddevices file configuration they are all configured as ?generic? Devtype. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From JRLang at uwyo.edu Thu Jan 18 15:20:18 2018 From: JRLang at uwyo.edu (Jeffrey R. Lang) Date: Thu, 18 Jan 2018 15:20:18 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <1743984571.54681.1516235286972@mail.yahoo.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: So is there a way to change it if it?s set incorrectly? jeff From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Jan 18 16:59:15 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 16:59:15 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the Devtype to the ?correct? setting, for example: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi My question is what is the correct setting? And what does GPFS do differently with each setting? Thanks, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. Lang Sent: Thursday, January 18, 2018 9:20 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ So is there a way to change it if it?s set incorrectly? jeff From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From er.a.ross at gmail.com Thu Jan 18 20:01:16 2018 From: er.a.ross at gmail.com (Eric Ross) Date: Thu, 18 Jan 2018 14:01:16 -0600 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: Bryan, While waiting on the "official" word as to what each setting does differently, I remembered there was a brief explanation of the differences (at least of dmm vs generic) in the /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit (https://github.com/finley/GPFS-Goodies) I use to use when I was at IBM. //snip dmm vs. generic is used by GPFS to prioritize internal order of searching through available disks, then later GPFS discards other disk device names that it finds that match as the same NSD device by a different path. For this reason, dmm vs. generic is an important distinction if you are not explicitly producing the entire and exclusive set of disks that GPFS should use, as output from this script (nsddevices) _and_ exiting this script with a "return 0". -Brian Finley //end snip If I read that correctly, it would indicate GPFS uses those additional labels (at least dmm vs generic) as a mechanism to determine which device files to prefer when scanning a system and finding the same NSD available via different devices (i.e. /dev/mapper/foo vs /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed device, I guess it would just ignore the /dev/sd${foo} devices when it scans them. Also, the difference only seems to matter if you're not explicitly creating a list a Brian F. indicates. If you simply generate the list and exit (via return 0), GPFS wouldn't continue scanning, and then find the associated /dev/sd${foo} devices to begin with, and therefore wouldn't need a label like dmm to prioritize it over say a generic device. - Eric On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister wrote: > Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > Devtype to the ?correct? setting, for example: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > My question is what is the correct setting? > > > > And what does GPFS do differently with each setting? > > > > Thanks, > > -Bryan > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. > Lang > Sent: Thursday, January 18, 2018 9:20 AM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ________________________________ > > So is there a way to change it if it?s set incorrectly? > > > > jeff > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty > Sent: Wednesday, January 17, 2018 6:28 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > > > Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our > case the device is setup as dmm > > > > [root at service5 ~]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > --------------------------------------------------------------------------------------------------- > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service5.pok.stglabs.ibm.com server node > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service6.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > service5.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > service6.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > service5.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > service6.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-3 dmm > service5.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-4 dmm > service6.pok.stglabs.ibm.com server node > > [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > [root at service5 ~]# > > > > If you run an tspreparedisk -s it will show you all of the paths. > > > > [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > 0972B6CD587CD8E0 /dev/sda generic > 0972B6CD587CD8E0 /dev/sdk generic > 0972B6CD587CD8E0 /dev/sdu generic > 0972B6CD587CD8E0 /dev/sdah generic > 0972B6CD587CD8E0 /dev/dm-0 dmm > [root at service5 ~]# > > > > Jim > > > > > > > > Jim > > On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: > > > > > > Hi all, > > > > We are reviewing some of our configurations and were not sure what to make > of the NSD Device Types that GPFS uses and what, if anything, do they change > about how GPFS accesses/recovers/manages/etc the underlying storage based on > this setting. > > > > The documentation doesn?t say much about it other than to consult the > /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > section: > > > > # Known disk types currently are: > > # > > # powerdisk - EMC power path disk > > # vpath - IBM virtual path disk > > # dmm - Device-Mapper Multipath (DMM) > > # dlmfdrv - Hitachi dlm > > # hdisk - AIX hard disk > > # lv - AIX logical volume. Historical usage only. > > # Not allowed as a new device to mmcrnsd. > > # gpt - GPFS partition on Windows disk > > # generic - Device having no unique failover or multipathing > > # characteristic (predominantly Linux devices). > > # dasd - DASD device (for Linux on z Systems) > > > > We have our storage under Linux Device-Mapper Multipath control (two device > paths to all storage, active/passive) and are accessible under /dev/mapper, > but the NSD types are current set to ?generic? not ?dmm?. This is > configured in the /var/mmfs/etc/nsddevices file: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > Can somebody from IBM explain what the correct setting should be and what > differences GPFS does with ?generic? vs. ?dmm? vs. others? > > > > Thanks in advance! > > -Bryan > > > > ________________________________ > > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From bbanister at jumptrading.com Thu Jan 18 20:09:09 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 20:09:09 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: <149af11d97db41b0a343932cd936eafc@jumptrading.com> Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross Sent: Thursday, January 18, 2018 2:01 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ------------------------------------------------- Bryan, While waiting on the "official" word as to what each setting does differently, I remembered there was a brief explanation of the differences (at least of dmm vs generic) in the /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit (https://github.com/finley/GPFS-Goodies) I use to use when I was at IBM. //snip dmm vs. generic is used by GPFS to prioritize internal order of searching through available disks, then later GPFS discards other disk device names that it finds that match as the same NSD device by a different path. For this reason, dmm vs. generic is an important distinction if you are not explicitly producing the entire and exclusive set of disks that GPFS should use, as output from this script (nsddevices) _and_ exiting this script with a "return 0". -Brian Finley //end snip If I read that correctly, it would indicate GPFS uses those additional labels (at least dmm vs generic) as a mechanism to determine which device files to prefer when scanning a system and finding the same NSD available via different devices (i.e. /dev/mapper/foo vs /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed device, I guess it would just ignore the /dev/sd${foo} devices when it scans them. Also, the difference only seems to matter if you're not explicitly creating a list a Brian F. indicates. If you simply generate the list and exit (via return 0), GPFS wouldn't continue scanning, and then find the associated /dev/sd${foo} devices to begin with, and therefore wouldn't need a label like dmm to prioritize it over say a generic device. - Eric On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister wrote: > Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > Devtype to the ?correct? setting, for example: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > My question is what is the correct setting? > > > > And what does GPFS do differently with each setting? > > > > Thanks, > > -Bryan > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. > Lang > Sent: Thursday, January 18, 2018 9:20 AM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ________________________________ > > So is there a way to change it if it?s set incorrectly? > > > > jeff > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty > Sent: Wednesday, January 17, 2018 6:28 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > > > Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our > case the device is setup as dmm > > > > [root at service5 ~]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > --------------------------------------------------------------------------------------------------- > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service5.pok.stglabs.ibm.com server node > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service6.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > service5.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > service6.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > service5.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > service6.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-3 dmm > service5.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-4 dmm > service6.pok.stglabs.ibm.com server node > > [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > [root at service5 ~]# > > > > If you run an tspreparedisk -s it will show you all of the paths. > > > > [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > 0972B6CD587CD8E0 /dev/sda generic > 0972B6CD587CD8E0 /dev/sdk generic > 0972B6CD587CD8E0 /dev/sdu generic > 0972B6CD587CD8E0 /dev/sdah generic > 0972B6CD587CD8E0 /dev/dm-0 dmm > [root at service5 ~]# > > > > Jim > > > > > > > > Jim > > On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: > > > > > > Hi all, > > > > We are reviewing some of our configurations and were not sure what to make > of the NSD Device Types that GPFS uses and what, if anything, do they change > about how GPFS accesses/recovers/manages/etc the underlying storage based on > this setting. > > > > The documentation doesn?t say much about it other than to consult the > /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > section: > > > > # Known disk types currently are: > > # > > # powerdisk - EMC power path disk > > # vpath - IBM virtual path disk > > # dmm - Device-Mapper Multipath (DMM) > > # dlmfdrv - Hitachi dlm > > # hdisk - AIX hard disk > > # lv - AIX logical volume. Historical usage only. > > # Not allowed as a new device to mmcrnsd. > > # gpt - GPFS partition on Windows disk > > # generic - Device having no unique failover or multipathing > > # characteristic (predominantly Linux devices). > > # dasd - DASD device (for Linux on z Systems) > > > > We have our storage under Linux Device-Mapper Multipath control (two device > paths to all storage, active/passive) and are accessible under /dev/mapper, > but the NSD types are current set to ?generic? not ?dmm?. This is > configured in the /var/mmfs/etc/nsddevices file: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > Can somebody from IBM explain what the correct setting should be and what > differences GPFS does with ?generic? vs. ?dmm? vs. others? > > > > Thanks in advance! > > -Bryan > > > > ________________________________ > > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From bbanister at jumptrading.com Thu Jan 18 20:12:15 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 20:12:15 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <149af11d97db41b0a343932cd936eafc@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: I should have also stated that the iostat results also report data going through the device-mapper-multipath devices as desired, -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Bryan Banister Sent: Thursday, January 18, 2018 2:09 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ------------------------------------------------- Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross Sent: Thursday, January 18, 2018 2:01 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ------------------------------------------------- Bryan, While waiting on the "official" word as to what each setting does differently, I remembered there was a brief explanation of the differences (at least of dmm vs generic) in the /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit (https://github.com/finley/GPFS-Goodies) I use to use when I was at IBM. //snip dmm vs. generic is used by GPFS to prioritize internal order of searching through available disks, then later GPFS discards other disk device names that it finds that match as the same NSD device by a different path. For this reason, dmm vs. generic is an important distinction if you are not explicitly producing the entire and exclusive set of disks that GPFS should use, as output from this script (nsddevices) _and_ exiting this script with a "return 0". -Brian Finley //end snip If I read that correctly, it would indicate GPFS uses those additional labels (at least dmm vs generic) as a mechanism to determine which device files to prefer when scanning a system and finding the same NSD available via different devices (i.e. /dev/mapper/foo vs /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed device, I guess it would just ignore the /dev/sd${foo} devices when it scans them. Also, the difference only seems to matter if you're not explicitly creating a list a Brian F. indicates. If you simply generate the list and exit (via return 0), GPFS wouldn't continue scanning, and then find the associated /dev/sd${foo} devices to begin with, and therefore wouldn't need a label like dmm to prioritize it over say a generic device. - Eric On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister wrote: > Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > Devtype to the ?correct? setting, for example: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > My question is what is the correct setting? > > > > And what does GPFS do differently with each setting? > > > > Thanks, > > -Bryan > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. > Lang > Sent: Thursday, January 18, 2018 9:20 AM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ________________________________ > > So is there a way to change it if it?s set incorrectly? > > > > jeff > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty > Sent: Wednesday, January 17, 2018 6:28 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > > > Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our > case the device is setup as dmm > > > > [root at service5 ~]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > --------------------------------------------------------------------------------------------------- > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service5.pok.stglabs.ibm.com server node > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service6.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > service5.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > service6.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > service5.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > service6.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-3 dmm > service5.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-4 dmm > service6.pok.stglabs.ibm.com server node > > [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > [root at service5 ~]# > > > > If you run an tspreparedisk -s it will show you all of the paths. > > > > [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > 0972B6CD587CD8E0 /dev/sda generic > 0972B6CD587CD8E0 /dev/sdk generic > 0972B6CD587CD8E0 /dev/sdu generic > 0972B6CD587CD8E0 /dev/sdah generic > 0972B6CD587CD8E0 /dev/dm-0 dmm > [root at service5 ~]# > > > > Jim > > > > > > > > Jim > > On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: > > > > > > Hi all, > > > > We are reviewing some of our configurations and were not sure what to make > of the NSD Device Types that GPFS uses and what, if anything, do they change > about how GPFS accesses/recovers/manages/etc the underlying storage based on > this setting. > > > > The documentation doesn?t say much about it other than to consult the > /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > section: > > > > # Known disk types currently are: > > # > > # powerdisk - EMC power path disk > > # vpath - IBM virtual path disk > > # dmm - Device-Mapper Multipath (DMM) > > # dlmfdrv - Hitachi dlm > > # hdisk - AIX hard disk > > # lv - AIX logical volume. Historical usage only. > > # Not allowed as a new device to mmcrnsd. > > # gpt - GPFS partition on Windows disk > > # generic - Device having no unique failover or multipathing > > # characteristic (predominantly Linux devices). > > # dasd - DASD device (for Linux on z Systems) > > > > We have our storage under Linux Device-Mapper Multipath control (two device > paths to all storage, active/passive) and are accessible under /dev/mapper, > but the NSD types are current set to ?generic? not ?dmm?. This is > configured in the /var/mmfs/etc/nsddevices file: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > Can somebody from IBM explain what the correct setting should be and what > differences GPFS does with ?generic? vs. ?dmm? vs. others? > > > > Thanks in advance! > > -Bryan > > > > ________________________________ > > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From er.a.ross at gmail.com Thu Jan 18 20:22:09 2018 From: er.a.ross at gmail.com (Eric Ross) Date: Thu, 18 Jan 2018 14:22:09 -0600 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <149af11d97db41b0a343932cd936eafc@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: I would say that's what it *appears* to be doing. The sys admin in me would still want some "official" word from IBM, as I wouldn't my assumption to be the cause of a meltdown at 16.50 on a Friday when everyone is heading out for the weekend ;) Cheers, -Eric On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister wrote: > Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross > Sent: Thursday, January 18, 2018 2:01 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file > > Note: External Email > ------------------------------------------------- > > Bryan, > > While waiting on the "official" word as to what each setting does > differently, I remembered there was a brief explanation of the > differences (at least of dmm vs generic) in the > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit > (https://github.com/finley/GPFS-Goodies) I use to use when I was at > IBM. > > //snip > > dmm vs. generic is used by GPFS to prioritize internal order of > searching through available disks, then later GPFS discards other > disk device names that it finds that match as the same NSD device > by a different path. For this reason, dmm vs. generic is an > important distinction if you are not explicitly producing the > entire and exclusive set of disks that GPFS should use, as output > from this script (nsddevices) _and_ exiting this script with a > "return 0". -Brian Finley > > //end snip > > If I read that correctly, it would indicate GPFS uses those additional > labels (at least dmm vs generic) as a mechanism to determine which > device files to prefer when scanning a system and finding the same NSD > available via different devices (i.e. /dev/mapper/foo vs > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed > device, I guess it would just ignore the /dev/sd${foo} devices when it > scans them. Also, the difference only seems to matter if you're not > explicitly creating a list a Brian F. indicates. If you simply > generate the list and exit (via return 0), GPFS wouldn't continue > scanning, and then find the associated /dev/sd${foo} devices to begin > with, and therefore wouldn't need a label like dmm to prioritize it > over say a generic device. > > > - Eric > > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister > wrote: >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the >> Devtype to the ?correct? setting, for example: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> My question is what is the correct setting? >> >> >> >> And what does GPFS do differently with each setting? >> >> >> >> Thanks, >> >> -Bryan >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. >> Lang >> Sent: Thursday, January 18, 2018 9:20 AM >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> Note: External Email >> >> ________________________________ >> >> So is there a way to change it if it?s set incorrectly? >> >> >> >> jeff >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty >> Sent: Wednesday, January 17, 2018 6:28 PM >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> >> >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our >> case the device is setup as dmm >> >> >> >> [root at service5 ~]# mmlsnsd -X >> >> Disk name NSD volume ID Device Devtype Node name >> Remarks >> --------------------------------------------------------------------------------------------------- >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service5.pok.stglabs.ibm.com server node >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service6.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm >> service5.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm >> service6.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm >> service5.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm >> service6.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-3 dmm >> service5.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-4 dmm >> service6.pok.stglabs.ibm.com server node >> >> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: >> [root at service5 ~]# >> >> >> >> If you run an tspreparedisk -s it will show you all of the paths. >> >> >> >> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 >> 0972B6CD587CD8E0 /dev/sda generic >> 0972B6CD587CD8E0 /dev/sdk generic >> 0972B6CD587CD8E0 /dev/sdu generic >> 0972B6CD587CD8E0 /dev/sdah generic >> 0972B6CD587CD8E0 /dev/dm-0 dmm >> [root at service5 ~]# >> >> >> >> Jim >> >> >> >> >> >> >> >> Jim >> >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister >> wrote: >> >> >> >> >> >> Hi all, >> >> >> >> We are reviewing some of our configurations and were not sure what to make >> of the NSD Device Types that GPFS uses and what, if anything, do they change >> about how GPFS accesses/recovers/manages/etc the underlying storage based on >> this setting. >> >> >> >> The documentation doesn?t say much about it other than to consult the >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this >> section: >> >> >> >> # Known disk types currently are: >> >> # >> >> # powerdisk - EMC power path disk >> >> # vpath - IBM virtual path disk >> >> # dmm - Device-Mapper Multipath (DMM) >> >> # dlmfdrv - Hitachi dlm >> >> # hdisk - AIX hard disk >> >> # lv - AIX logical volume. Historical usage only. >> >> # Not allowed as a new device to mmcrnsd. >> >> # gpt - GPFS partition on Windows disk >> >> # generic - Device having no unique failover or multipathing >> >> # characteristic (predominantly Linux devices). >> >> # dasd - DASD device (for Linux on z Systems) >> >> >> >> We have our storage under Linux Device-Mapper Multipath control (two device >> paths to all storage, active/passive) and are accessible under /dev/mapper, >> but the NSD types are current set to ?generic? not ?dmm?. This is >> configured in the /var/mmfs/etc/nsddevices file: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> Can somebody from IBM explain what the correct setting should be and what >> differences GPFS does with ?generic? vs. ?dmm? vs. others? >> >> >> >> Thanks in advance! >> >> -Bryan >> >> >> >> ________________________________ >> >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ________________________________ >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From yguvvala at cambridgecomputer.com Thu Jan 18 22:10:35 2018 From: yguvvala at cambridgecomputer.com (Yugendra Guvvala) Date: Thu, 18 Jan 2018 17:10:35 -0500 Subject: [gpfsug-discuss] Excluding Filesets on snapshots Message-ID: Hi, We have 5 dependent Filesets linked to root fileset (mount point) and we are creating a new independent Fileset which will be linked to root fileset (mount point). We want to take snapshots of the dependent filesets and independent fileset on different periodic intervals. Is there a way to exclude independent fileset and take snapshot of all dependent filesets ? -- Thanks, Yugi -------------- next part -------------- An HTML attachment was scrubbed... URL: From rohwedder at de.ibm.com Fri Jan 19 08:07:17 2018 From: rohwedder at de.ibm.com (Markus Rohwedder) Date: Fri, 19 Jan 2018 09:07:17 +0100 Subject: [gpfsug-discuss] Excluding Filesets on snapshots In-Reply-To: References: Message-ID: Hello Yugendra, Snapshots are taken per Inode space. Both the root fileset and the independent fileset you created own separate inode spaces, even if the independent fileset is nested in the root fileset. Taking a snapshot of the root fileset should exclude the independent fileset. e.g: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.03.52' -j 'root' Taking a filesystem snapshot however would include all inode spaces of the file system and would therefore include the independent fileset. e.g.: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.04.47' Mit freundlichen Gr??en / Kind regards Dr. Markus Rohwedder Spectrum Scale GUI Development Phone: +49 7034 6430190 IBM Deutschland E-Mail: rohwedder at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina K?deritz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Yugendra Guvvala To: gpfsug main discussion list Date: 01/18/2018 11:11 PM Subject: [gpfsug-discuss] Excluding Filesets on snapshots Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, We have 5 dependent Filesets linked to root fileset (mount point) and we are creating a new independent Fileset which will be linked to root fileset (mount point). We want to take snapshots of the dependent filesets and independent fileset on different periodic intervals. Is there a way to exclude independent fileset and take snapshot of all dependent filesets ? -- Thanks, Yugi _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=vZOWTPN66Z-dgh-Xs1VV4BI9dhIVi3BUFi7Zt1gPE2E&s=3HAnqhprHHsQe1cvwJIYK4cFJwG7BcNK6hKxXaDQ7sA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14162451.gif Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From jonbernard at gmail.com Fri Jan 19 15:57:11 2018 From: jonbernard at gmail.com (Jon Bernard) Date: Fri, 19 Jan 2018 10:57:11 -0500 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: A few years ago, Yuri noted that "dmm" has a special meaning to GPFS code. SCSI3 PR code keys off this disk type. https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014924464#77777777-0000-0000-0000-000014924464 https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#scsi3 Jon On Thu, Jan 18, 2018 at 3:22 PM, Eric Ross wrote: > I would say that's what it *appears* to be doing. The sys admin in me > would still want some "official" word from IBM, as I wouldn't my > assumption to be the cause of a meltdown at 16.50 on a Friday when > everyone is heading out for the weekend ;) > > Cheers, > -Eric > > On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister > wrote: > > Great! So this is just for selecting devices and according to my > `mmlsnsd -X` output it's using the correct ones, so I can probably return > to ignoring this parameter! > > -Bryan > > > > -----Original Message----- > > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org] On Behalf Of Eric Ross > > Sent: Thursday, January 18, 2018 2:01 PM > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ------------------------------------------------- > > > > Bryan, > > > > While waiting on the "official" word as to what each setting does > > differently, I remembered there was a brief explanation of the > > differences (at least of dmm vs generic) in the > > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit > > (https://github.com/finley/GPFS-Goodies) I use to use when I was at > > IBM. > > > > //snip > > > > dmm vs. generic is used by GPFS to prioritize internal order of > > searching through available disks, then later GPFS discards other > > disk device names that it finds that match as the same NSD device > > by a different path. For this reason, dmm vs. generic is an > > important distinction if you are not explicitly producing the > > entire and exclusive set of disks that GPFS should use, as output > > from this script (nsddevices) _and_ exiting this script with a > > "return 0". -Brian Finley > > > > //end snip > > > > If I read that correctly, it would indicate GPFS uses those additional > > labels (at least dmm vs generic) as a mechanism to determine which > > device files to prefer when scanning a system and finding the same NSD > > available via different devices (i.e. /dev/mapper/foo vs > > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed > > device, I guess it would just ignore the /dev/sd${foo} devices when it > > scans them. Also, the difference only seems to matter if you're not > > explicitly creating a list a Brian F. indicates. If you simply > > generate the list and exit (via return 0), GPFS wouldn't continue > > scanning, and then find the associated /dev/sd${foo} devices to begin > > with, and therefore wouldn't need a label like dmm to prioritize it > > over say a generic device. > > > > > > - Eric > > > > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister > > wrote: > >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > >> Devtype to the ?correct? setting, for example: > >> > >> > >> > >> if [[ $osName = Linux ]] > >> > >> then > >> > >> : # Add function to discover disks in the Linux environment. > >> > >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > >> > >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > >> > >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > >> > >> fi > >> > >> > >> > >> My question is what is the correct setting? > >> > >> > >> > >> And what does GPFS do differently with each setting? > >> > >> > >> > >> Thanks, > >> > >> -Bryan > >> > >> > >> > >> From: gpfsug-discuss-bounces at spectrumscale.org > >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey > R. > >> Lang > >> Sent: Thursday, January 18, 2018 9:20 AM > >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > >> nsddevices file > >> > >> > >> > >> Note: External Email > >> > >> ________________________________ > >> > >> So is there a way to change it if it?s set incorrectly? > >> > >> > >> > >> jeff > >> > >> > >> > >> From: gpfsug-discuss-bounces at spectrumscale.org > >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim > Doherty > >> Sent: Wednesday, January 17, 2018 6:28 PM > >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > >> nsddevices file > >> > >> > >> > >> > >> > >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In > our > >> case the device is setup as dmm > >> > >> > >> > >> [root at service5 ~]# mmlsnsd -X > >> > >> Disk name NSD volume ID Device Devtype Node name > >> Remarks > >> ------------------------------------------------------------ > --------------------------------------- > >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume4 0972B6CE587CF625 /dev/dm-3 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume4 0972B6CE587CF625 /dev/dm-4 dmm > >> service6.pok.stglabs.ibm.com server node > >> > >> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata: > 0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6. > pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system: > service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > >> [root at service5 ~]# > >> > >> > >> > >> If you run an tspreparedisk -s it will show you all of the paths. > >> > >> > >> > >> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > >> 0972B6CD587CD8E0 /dev/sda generic > >> 0972B6CD587CD8E0 /dev/sdk generic > >> 0972B6CD587CD8E0 /dev/sdu generic > >> 0972B6CD587CD8E0 /dev/sdah generic > >> 0972B6CD587CD8E0 /dev/dm-0 dmm > >> [root at service5 ~]# > >> > >> > >> > >> Jim > >> > >> > >> > >> > >> > >> > >> > >> Jim > >> > >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > >> wrote: > >> > >> > >> > >> > >> > >> Hi all, > >> > >> > >> > >> We are reviewing some of our configurations and were not sure what to > make > >> of the NSD Device Types that GPFS uses and what, if anything, do they > change > >> about how GPFS accesses/recovers/manages/etc the underlying storage > based on > >> this setting. > >> > >> > >> > >> The documentation doesn?t say much about it other than to consult the > >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > >> section: > >> > >> > >> > >> # Known disk types currently are: > >> > >> # > >> > >> # powerdisk - EMC power path disk > >> > >> # vpath - IBM virtual path disk > >> > >> # dmm - Device-Mapper Multipath (DMM) > >> > >> # dlmfdrv - Hitachi dlm > >> > >> # hdisk - AIX hard disk > >> > >> # lv - AIX logical volume. Historical usage only. > >> > >> # Not allowed as a new device to mmcrnsd. > >> > >> # gpt - GPFS partition on Windows disk > >> > >> # generic - Device having no unique failover or multipathing > >> > >> # characteristic (predominantly Linux devices). > >> > >> # dasd - DASD device (for Linux on z Systems) > >> > >> > >> > >> We have our storage under Linux Device-Mapper Multipath control (two > device > >> paths to all storage, active/passive) and are accessible under > /dev/mapper, > >> but the NSD types are current set to ?generic? not ?dmm?. This is > >> configured in the /var/mmfs/etc/nsddevices file: > >> > >> > >> > >> if [[ $osName = Linux ]] > >> > >> then > >> > >> : # Add function to discover disks in the Linux environment. > >> > >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > >> > >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > >> > >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > >> > >> fi > >> > >> > >> > >> Can somebody from IBM explain what the correct setting should be and > what > >> differences GPFS does with ?generic? vs. ?dmm? vs. others? > >> > >> > >> > >> Thanks in advance! > >> > >> -Bryan > >> > >> > >> > >> ________________________________ > >> > >> > >> Note: This email is for the confidential use of the named addressee(s) > only > >> and may contain proprietary, confidential or privileged information. If > you > >> are not the intended recipient, you are hereby notified that any review, > >> dissemination or copying of this email is strictly prohibited, and to > please > >> notify the sender immediately and destroy this email and any > attachments. > >> Email transmission cannot be guaranteed to be secure or error-free. The > >> Company, therefore, does not make any guarantees as to the completeness > or > >> accuracy of this email or any attachments. This email is for > informational > >> purposes only and does not constitute a recommendation, offer, request > or > >> solicitation of any kind to buy, sell, subscribe, redeem or perform any > type > >> of transaction of a financial product. > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> > >> ________________________________ > >> > >> Note: This email is for the confidential use of the named addressee(s) > only > >> and may contain proprietary, confidential or privileged information. If > you > >> are not the intended recipient, you are hereby notified that any review, > >> dissemination or copying of this email is strictly prohibited, and to > please > >> notify the sender immediately and destroy this email and any > attachments. > >> Email transmission cannot be guaranteed to be secure or error-free. The > >> Company, therefore, does not make any guarantees as to the completeness > or > >> accuracy of this email or any attachments. This email is for > informational > >> purposes only and does not constitute a recommendation, offer, request > or > >> solicitation of any kind to buy, sell, subscribe, redeem or perform any > type > >> of transaction of a financial product. > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ________________________________ > > > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Fri Jan 19 16:10:14 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 19 Jan 2018 16:10:14 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: <1b8e86ededfb41de966d23df47b3439f@jumptrading.com> I was thinking that (SCSI3 Persistent Reserve) might be used with dmm after I saw something related to IBM RDAC software in this posting by Scott Fadden: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+%28GPFS%29/page/Device+Naming+and+Discovery+in+GPFS Thanks Jon, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jon Bernard Sent: Friday, January 19, 2018 9:57 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ A few years ago, Yuri noted that "dmm" has a special meaning to GPFS code. SCSI3 PR code keys off this disk type. https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014924464#77777777-0000-0000-0000-000014924464 https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#scsi3 Jon On Thu, Jan 18, 2018 at 3:22 PM, Eric Ross > wrote: I would say that's what it *appears* to be doing. The sys admin in me would still want some "official" word from IBM, as I wouldn't my assumption to be the cause of a meltdown at 16.50 on a Friday when everyone is heading out for the weekend ;) Cheers, -Eric On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister > wrote: > Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross > Sent: Thursday, January 18, 2018 2:01 PM > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file > > Note: External Email > ------------------------------------------------- > > Bryan, > > While waiting on the "official" word as to what each setting does > differently, I remembered there was a brief explanation of the > differences (at least of dmm vs generic) in the > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit > (https://github.com/finley/GPFS-Goodies) I use to use when I was at > IBM. > > //snip > > dmm vs. generic is used by GPFS to prioritize internal order of > searching through available disks, then later GPFS discards other > disk device names that it finds that match as the same NSD device > by a different path. For this reason, dmm vs. generic is an > important distinction if you are not explicitly producing the > entire and exclusive set of disks that GPFS should use, as output > from this script (nsddevices) _and_ exiting this script with a > "return 0". -Brian Finley > > //end snip > > If I read that correctly, it would indicate GPFS uses those additional > labels (at least dmm vs generic) as a mechanism to determine which > device files to prefer when scanning a system and finding the same NSD > available via different devices (i.e. /dev/mapper/foo vs > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed > device, I guess it would just ignore the /dev/sd${foo} devices when it > scans them. Also, the difference only seems to matter if you're not > explicitly creating a list a Brian F. indicates. If you simply > generate the list and exit (via return 0), GPFS wouldn't continue > scanning, and then find the associated /dev/sd${foo} devices to begin > with, and therefore wouldn't need a label like dmm to prioritize it > over say a generic device. > > > - Eric > > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister > > wrote: >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the >> Devtype to the ?correct? setting, for example: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> My question is what is the correct setting? >> >> >> >> And what does GPFS do differently with each setting? >> >> >> >> Thanks, >> >> -Bryan >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. >> Lang >> Sent: Thursday, January 18, 2018 9:20 AM >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> Note: External Email >> >> ________________________________ >> >> So is there a way to change it if it?s set incorrectly? >> >> >> >> jeff >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty >> Sent: Wednesday, January 17, 2018 6:28 PM >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> >> >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our >> case the device is setup as dmm >> >> >> >> [root at service5 ~]# mmlsnsd -X >> >> Disk name NSD volume ID Device Devtype Node name >> Remarks >> --------------------------------------------------------------------------------------------------- >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service5.pok.stglabs.ibm.com server node >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service6.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm >> service5.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm >> service6.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm >> service5.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm >> service6.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-3 dmm >> service5.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-4 dmm >> service6.pok.stglabs.ibm.com server node >> >> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: >> [root at service5 ~]# >> >> >> >> If you run an tspreparedisk -s it will show you all of the paths. >> >> >> >> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 >> 0972B6CD587CD8E0 /dev/sda generic >> 0972B6CD587CD8E0 /dev/sdk generic >> 0972B6CD587CD8E0 /dev/sdu generic >> 0972B6CD587CD8E0 /dev/sdah generic >> 0972B6CD587CD8E0 /dev/dm-0 dmm >> [root at service5 ~]# >> >> >> >> Jim >> >> >> >> >> >> >> >> Jim >> >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister >> > wrote: >> >> >> >> >> >> Hi all, >> >> >> >> We are reviewing some of our configurations and were not sure what to make >> of the NSD Device Types that GPFS uses and what, if anything, do they change >> about how GPFS accesses/recovers/manages/etc the underlying storage based on >> this setting. >> >> >> >> The documentation doesn?t say much about it other than to consult the >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this >> section: >> >> >> >> # Known disk types currently are: >> >> # >> >> # powerdisk - EMC power path disk >> >> # vpath - IBM virtual path disk >> >> # dmm - Device-Mapper Multipath (DMM) >> >> # dlmfdrv - Hitachi dlm >> >> # hdisk - AIX hard disk >> >> # lv - AIX logical volume. Historical usage only. >> >> # Not allowed as a new device to mmcrnsd. >> >> # gpt - GPFS partition on Windows disk >> >> # generic - Device having no unique failover or multipathing >> >> # characteristic (predominantly Linux devices). >> >> # dasd - DASD device (for Linux on z Systems) >> >> >> >> We have our storage under Linux Device-Mapper Multipath control (two device >> paths to all storage, active/passive) and are accessible under /dev/mapper, >> but the NSD types are current set to ?generic? not ?dmm?. This is >> configured in the /var/mmfs/etc/nsddevices file: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> Can somebody from IBM explain what the correct setting should be and what >> differences GPFS does with ?generic? vs. ?dmm? vs. others? >> >> >> >> Thanks in advance! >> >> -Bryan >> >> >> >> ________________________________ >> >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ________________________________ >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Fri Jan 19 21:38:03 2018 From: ewahl at osc.edu (Edward Wahl) Date: Fri, 19 Jan 2018 16:38:03 -0500 Subject: [gpfsug-discuss] policy ilm features? Message-ID: <20180119163803.79fddbeb@osc.edu> This one has been on my list a long time so I figured I'd ask here first before I open an apar or request an enhancement (most likely). Is there a way using the policy engine to determine the following? -metadata replication total/current -unbalanced file Looking to catch things like this that stand out on my filesystem without having to run several hundred million 'mmlsattr's. metadata replication: 1 max 2 flags: unbalanced Ed -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From makaplan at us.ibm.com Sat Jan 20 16:06:58 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sat, 20 Jan 2018 13:06:58 -0300 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180119163803.79fddbeb@osc.edu> References: <20180119163803.79fddbeb@osc.edu> Message-ID: Hint. RTFineManual, particularly the Admin guide, and look for MISC_ATTRIBUTES, Regarding metadata replication, one first has to ask, which metadata is replicated when and what if anything the mmchattr -m does or changes... https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+(GPFS)/page/Configuring+GPFS+for+Reliability --marc K. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hmorales at optimizeit.co Sun Jan 21 21:19:29 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Sun, 21 Jan 2018 16:19:29 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem Message-ID: Hello, I have a GPFS cluster with two filesystems. The disks associated to one filesystem reside on an old storage and the other filesystem disks reside on a much more modern storage system. I have successfully moved data from one fs to the other but there are questions about data integrity that still need verification so the old filesystem needs somehow to be preserved. My question is: Can I remove the old filesystem LUNs association to the NSDs servers without removing spectrum scale filesystems, so that later on, if necessary, I could associate them back and the old filesystem would be operating as normal? If possible: what would be the general steps to achieve this? ----- Thank you, Harold. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Sun Jan 21 21:35:44 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sun, 21 Jan 2018 21:35:44 +0000 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: <1d625007-9b14-a39b-5b30-027cb03a383c@strath.ac.uk> On 21/01/18 21:19, Harold Morales wrote: > Hello, > > I have a GPFS cluster with two filesystems. The disks associated to one > filesystem reside on an old storage and the other filesystem disks > reside on a much more modern storage system. I have successfully moved > data from one fs to the other but there are questions about data > integrity that still need verification so the old filesystem needs > somehow to be preserved. > > My question is: Can I remove the old filesystem LUNs association to the > NSDs servers without removing spectrum scale filesystems, so that later > on, if necessary, I could associate them back and the old filesystem > would be operating as normal? If possible: what would be the general > steps to achieve this? > I would take a look at mmexportfs/mmimportfs myself to be on the safe side. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From hmorales at optimizeit.co Sun Jan 21 23:35:01 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Sun, 21 Jan 2018 18:35:01 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: <1d625007-9b14-a39b-5b30-027cb03a383c@strath.ac.uk> References: <1d625007-9b14-a39b-5b30-027cb03a383c@strath.ac.uk> Message-ID: Thank you. The filesystems belong both to the same cluster. My question is whether LUNs can be unmapped from the host even though filesystems have not been configured and when they are mapped back the filesystem is going to operate correctly. El 21/01/2018 5:35 p.m., "Jonathan Buzzard" escribi?: On 21/01/18 21:19, Harold Morales wrote: > Hello, > > I have a GPFS cluster with two filesystems. The disks associated to one > filesystem reside on an old storage and the other filesystem disks reside > on a much more modern storage system. I have successfully moved data from > one fs to the other but there are questions about data integrity that still > need verification so the old filesystem needs somehow to be preserved. > > My question is: Can I remove the old filesystem LUNs association to the > NSDs servers without removing spectrum scale filesystems, so that later on, > if necessary, I could associate them back and the old filesystem would be > operating as normal? If possible: what would be the general steps to > achieve this? > > I would take a look at mmexportfs/mmimportfs myself to be on the safe side. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Sun Jan 21 23:43:47 2018 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Sun, 21 Jan 2018 23:43:47 +0000 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Mon Jan 22 03:41:58 2018 From: ulmer at ulmer.org (Stephen Ulmer) Date: Sun, 21 Jan 2018 22:41:58 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: Harold, The way I read your question, no one has actually answered it fully: You want to put the old file system in cold storage for forensic purposes ? exactly as it is. You want the NSDs to go away until and unless you need them in the future. QUICK AND DIRTY - YOU SHOULD NOT DO THIS: Set the old file system to NOT mount automatically. Make sure it is unmounted everywhere(!!!!!). Make sure none of those NSDs are being used as quorum tie-breakers, etc. Print your resume. Unmap the LUNs. This will leave all of the information ABOUT the filesystem in the Spectrum Scale configuration, but it won?t be available. You?ll get a bunch of errors that the NSDs are gone. Lots of them. It will make starting up Spectrum Scale take a long time while it looks for and fails to find them. That will be mostly noise. I?m not sure how you could corrupt the file system when the LUNs are no longer accessible and there can?t be any writes pending. I have done this recently to keep an old file system around after a migration (the customer required an "A/B" switch when installing new storage). This was with a very old version of GPFS that could not be upgraded. I would not do this unless the time frame to keep this configuration is very short ? I only did it because I didn?t have a choice. PROBABLY BETTER - READ THIS ONE TWICE: Take a look at mmexportfs (this was already suggested). The point of this command to to be able to carry the Spectrum Scale "configuration" for a file system AND the LUNs full of data around and plug them into a cluster later. I think a misunderstanding here led to your objection about this method: It doesn?t have to be a different cluster ? you can use the same one it came from. The advantage to this approach is that it leaves you with a more hygienic cluster ? no "missing" storage errors. The ONLY disadvantage that I can think of at the moment is that the file system configuration MIGHT have some lint removed during the import/export process. I?m not sure that *any* checking is done during the export, but I?d guess that the import involves at least some validation of the imported file. I hope. So if you think the file system *configuration* is wonky, you should call support and see if they will look at your export, or get a snap while you?ve still got the file system in your cluster. If you think that the *structure* of the file system on disk (or maybe just the copy method you?re using) might be wonky, then learn how to use mmexportfs. Note that the learning can certainly include asking questions here. When you want to look at the old file system again, you WILL have to re-map the LUNs, and use mmimportfs (and the file you made with mmexportfs). Then the file system will be part of your cluster again. STANDARD DISCLAIMERS APPLY: Your mileage may vary. Following this advice could cause nausea, vomiting, sweating, weight gain, sleeplessness, unemployment, weight loss, temporary tooth loss, redundancy, olfactory hallucinations, oozing (generalized), redundancy, uneven tire wear, excessive body oder, scoliosis, halitosis, ring worm, and intermittent acute ignorance. Good luck! :) Liberty, -- Stephen > On Jan 21, 2018, at 6:43 PM, Andrew Beattie > wrote: > > Harold, > > How big is the old file system, Spectrum Scale is going to throw a bunch of errors if you remove the luns from the old file system while attempting to keep the file system "data" on the luns. its likely to cause you all sorts of errors and potential data corruption. Its not something that I would recommend. > > Can you do a backup of the old filesystem so you can do a restore of data if you need to? > > Regards, > Andrew Beattie > Software Defined Storage - IT Specialist > Phone: 614-2133-7927 > E-mail: abeattie at au1.ibm.com > > > ----- Original message ----- > From: Harold Morales > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem > Date: Mon, Jan 22, 2018 7:20 AM > > Hello, > > I have a GPFS cluster with two filesystems. The disks associated to one filesystem reside on an old storage and the other filesystem disks reside on a much more modern storage system. I have successfully moved data from one fs to the other but there are questions about data integrity that still need verification so the old filesystem needs somehow to be preserved. > > My question is: Can I remove the old filesystem LUNs association to the NSDs servers without removing spectrum scale filesystems, so that later on, if necessary, I could associate them back and the old filesystem would be operating as normal? If possible: what would be the general steps to achieve this? > > ----- > Thank you, > > Harold. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=deVdfZ4CCXfI09tUiJGGP1c17jRhhwjx2TcB12uunoc&s=y3EUXWX1ecxWLR1HG0Ohwn9xsPKHZ6Pdodxoz44HV7A&e= > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon Jan 22 09:57:04 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 22 Jan 2018 09:57:04 +0000 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: <73d242a4-ee85-a9af-22c0-5c4fe23e3367@strath.ac.uk> On 22/01/18 03:41, Stephen Ulmer wrote: > Harold, > > The way I read your question, no one has actually answered it fully: > > You want to put the old file system in cold storage for forensic > purposes ? exactly as it is. You want the NSDs to go away until and > unless you need them in the future. > Hum, I would have said I did answer it fully even if I didn't go into detail. The only in my view sensible approach is to do mmexportfs so that should you need access to the data then you can get to it by reimporting it using mmimportfs. In the interim you can unmap all the NSD's from the cluster without causing GPFS to go care in the slightest. Otherwise you are doing something that is in my view at the best fool hardy and more generally down right idiotic. Personally I would outright refuse to do it without someone else higher up signing a written disclaimer that I was not responsible should it all go pear shaped. Note that the mmexportfs takes a few seconds at most to complete, and likewise with the mmimport. I am however somewhat perplexed by the data integrity side of the issue. If you suspect the data is corrupt in the old file system then having it around for reference purposes is surely not going to help. What are you going to do mount it again to find the file is corrupt; how is that going to help? If you suspect that whatever method you used to move the files from the old file system to the new one may have introduced corruption, then I would suggest that the best way out of that is to do an rsync with the -c flag so that it takes an MD5 checksum of the files on both file systems. Once that is complete you can safely ditch the old file system completely knowing that you have recovered it as best as possible. You could probably kick a bunch of rsyncs of in parallel to speed this method up. In fact a "rsync -c" would be a gold standard of look it's absolutely the same on the old and new file systems and remove all doubts about the transfer introducing corruption. At that point if someone comes and says your transfer corrupted my files, you can with justification say they where corrupt in the old file system and I have done my best to transfer them over. Note if any user was deliberately creating MD5 collisions then they get everything they deserve in my view, and accidental collisions in MD5 are about as likely as ?. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From john.hearns at asml.com Mon Jan 22 08:29:42 2018 From: john.hearns at asml.com (John Hearns) Date: Mon, 22 Jan 2018 08:29:42 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180119163803.79fddbeb@osc.edu> References: <20180119163803.79fddbeb@osc.edu> Message-ID: Ed, This is not a perfect answer. You need to look at policies for this. I have been doing something similar recently. Something like: RULE 'list_file' EXTERNAL LIST 'all-files' EXEC '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files' SHOW( varchar(kb_allocated) || ' ' || varchar(file_size) || ' ' || varchar(misc_attributes) || ' ' || name || ' ' || fileset_name ) WHERE REGEX(misc_attributes,'[J]') So this policy shows the kbytes allocates, file size, the miscellaneous attributes, name and fileset name For all files with miscellaneous attributes of 'J' which means 'Some data blocks might be ill replicated' -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Edward Wahl Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] policy ilm features? This one has been on my list a long time so I figured I'd ask here first before I open an apar or request an enhancement (most likely). Is there a way using the policy engine to determine the following? -metadata replication total/current -unbalanced file Looking to catch things like this that stand out on my filesystem without having to run several hundred million 'mmlsattr's. metadata replication: 1 max 2 flags: unbalanced Ed -- Ed Wahl Ohio Supercomputer Center 614-292-9302 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D&reserved=0 -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. From yguvvala at cambridgecomputer.com Mon Jan 22 21:13:15 2018 From: yguvvala at cambridgecomputer.com (Yugendra Guvvala) Date: Mon, 22 Jan 2018 16:13:15 -0500 Subject: [gpfsug-discuss] Excluding Filesets on snapshots In-Reply-To: References: Message-ID: Thank you that works, we have to specify root fileset to get a snapshot. not specifiying root:snapshotname will include all all file sets. example : mmcrsnapshot device root:snapshot_name On Fri, Jan 19, 2018 at 3:07 AM, Markus Rohwedder wrote: > Hello Yugendra, > > Snapshots are taken per Inode space. Both the root fileset and the > independent fileset you created own separate inode spaces, > even if the independent fileset is nested in the root fileset. > > Taking a snapshot of the root fileset should exclude the independent > fileset. > e.g: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.03.52' -j 'root' > > Taking a filesystem snapshot however would include all inode spaces of the > file system and would therefore include the independent fileset. > e.g.: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.04.47' > > > Mit freundlichen Gr??en / Kind regards > > *Dr. Markus Rohwedder* > > Spectrum Scale GUI Development > ------------------------------ > Phone: +49 7034 6430190 <+49%207034%206430190> IBM Deutschland > E-Mail: rohwedder at de.ibm.com Am Weiher 24 > 65451 Kelsterbach > Germany > ------------------------------ > IBM Deutschland Research & Development GmbH / Vorsitzender des > Aufsichtsrats: Martina K?deritz > Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, > HRB 243294 > > [image: Inactive hide details for Yugendra Guvvala ---01/18/2018 11:11:55 > PM---Hi, We have 5 dependent Filesets linked to root fileset]Yugendra > Guvvala ---01/18/2018 11:11:55 PM---Hi, We have 5 dependent Filesets linked > to root fileset (mount point) and we > > From: Yugendra Guvvala > To: gpfsug main discussion list > Date: 01/18/2018 11:11 PM > Subject: [gpfsug-discuss] Excluding Filesets on snapshots > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi, > > We have 5 dependent Filesets linked to root fileset (mount point) and we > are creating a new independent Fileset which will be linked to root fileset > (mount point). We want to take snapshots of the dependent filesets and > independent fileset on different periodic intervals. Is there a way to > exclude independent fileset and take snapshot of all dependent filesets ? > > > -- > Thanks, > Yugi > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r= > IbxtjdkPAM2Sbon4Lbbi4w&m=vZOWTPN66Z-dgh-Xs1VV4BI9dhIVi3BUFi7Zt1gPE2E&s= > 3HAnqhprHHsQe1cvwJIYK4cFJwG7BcNK6hKxXaDQ7sA&e= > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Thanks, *Yugendra Guvvala | HPC Technologist ** |** Cambridge Computer ** |** "Artists in Data Storage" * Direct: 781-250-3273 | Cell: 806-773-4464 | yguvvala at cambridgecomputer.com | www.cambridgecomputer.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14162451.gif Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From hmorales at optimizeit.co Tue Jan 23 04:23:47 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Mon, 22 Jan 2018 23:23:47 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: <73d242a4-ee85-a9af-22c0-5c4fe23e3367@strath.ac.uk> References: <73d242a4-ee85-a9af-22c0-5c4fe23e3367@strath.ac.uk> Message-ID: Thanks Stephen and Jonathan for pointing this important things out. Now I begin to understand the importance of mmimportfs/mmexportfs for operations like the one I discuss here. Will let you know how it goes. 2018-01-22 4:57 GMT-05:00 Jonathan Buzzard : > On 22/01/18 03:41, Stephen Ulmer wrote: > >> Harold, >> >> The way I read your question, no one has actually answered it fully: >> >> You want to put the old file system in cold storage for forensic purposes >> ? exactly as it is. You want the NSDs to go away until and unless you need >> them in the future. >> >> > Hum, I would have said I did answer it fully even if I didn't go into > detail. The only in my view sensible approach is to do mmexportfs so that > should you need access to the data then you can get to it by reimporting it > using mmimportfs. In the interim you can unmap all the NSD's from the > cluster without causing GPFS to go care in the slightest. > > Otherwise you are doing something that is in my view at the best fool > hardy and more generally down right idiotic. Personally I would outright > refuse to do it without someone else higher up signing a written disclaimer > that I was not responsible should it all go pear shaped. Note that the > mmexportfs takes a few seconds at most to complete, and likewise with the > mmimport. > > I am however somewhat perplexed by the data integrity side of the issue. > If you suspect the data is corrupt in the old file system then having it > around for reference purposes is surely not going to help. What are you > going to do mount it again to find the file is corrupt; how is that going > to help? > > If you suspect that whatever method you used to move the files from the > old file system to the new one may have introduced corruption, then I would > suggest that the best way out of that is to do an rsync with the -c flag so > that it takes an MD5 checksum of the files on both file systems. Once that > is complete you can safely ditch the old file system completely knowing > that you have recovered it as best as possible. You could probably kick a > bunch of rsyncs of in parallel to speed this method up. > > In fact a "rsync -c" would be a gold standard of look it's absolutely the > same on the old and new file systems and remove all doubts about the > transfer introducing corruption. At that point if someone comes and says > your transfer corrupted my files, you can with justification say they where > corrupt in the old file system and I have done my best to transfer them > over. Note if any user was deliberately creating MD5 collisions then they > get everything they deserve in my view, and accidental collisions in MD5 > are about as likely as ?. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chetkulk at in.ibm.com Tue Jan 23 09:30:10 2018 From: chetkulk at in.ibm.com (Chetan R Kulkarni) Date: Tue, 23 Jan 2018 15:00:10 +0530 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: Message-ID: Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Tue Jan 23 11:10:57 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 23 Jan 2018 11:10:57 +0000 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: Message-ID: Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services to halt/timeout/fail/stop/go slow/cause an earthquake or is it completely transparent? My existing config is: FILE access configuration : AD PARAMETERS VALUES ------------------------------------------------- ENABLE_NFS_KERBEROS true SERVERS server.domain USER_NAME username NETBIOS_NAME store IDMAP_ROLE master IDMAP_RANGE 10000000-299999999 IDMAP_RANGE_SIZE 1000000 UNIXMAP_DOMAINS DOMAIN(500 - 2000000) LDAPMAP_DOMAINS none OBJECT access not configured PARAMETERS VALUES Enabled services: SMB SMB is running Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Chetan R Kulkarni Sent: 23 January 2018 09:30 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. [Inactive hide details for "Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a f]"Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nf From: "Sobey, Richard A" > To: "'gpfsug-discuss at spectrumscale.org'" > Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 23 17:16:36 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 23 Jan 2018 17:16:36 +0000 Subject: [gpfsug-discuss] Metadata only system pool Message-ID: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_johnson at brown.edu Tue Jan 23 17:23:59 2018 From: david_johnson at brown.edu (david_johnson at brown.edu) Date: Tue, 23 Jan 2018 12:23:59 -0500 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: If the new files need indirect blocks or extended attributes that don?t fit in the basic inode, additional metadata space would need to be allocated. There might be other reasons but these come to mind immediately. -- ddj Dave Johnson > On Jan 23, 2018, at 12:16 PM, Buterbaugh, Kevin L wrote: > > Hi All, > > I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: > > Inode Information > ----------------- > Number of used inodes: 218635454 > Number of free inodes: 131364674 > Number of allocated inodes: 350000128 > Maximum number of inodes: 350000128 > > I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. > > Now my system pool is almost ?full?: > > (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) > > But again, what - outside of me creating more inodes - would cause that to change?? > > Thanks? > > Kevin > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Jan 23 17:25:11 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 23 Jan 2018 17:25:11 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: <00afb095d1314045963e740f624882b8@jumptrading.com> Directory entries are also stored in the system pool and existing directories must grow in size (indirect blocks) to hold additional files/dir references as new files/dirs are created. That would be my initial guess anyways. The ?pool total? output below seems to indicate that you only have 34Mb of free space? that is definitely not good, -B From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L Sent: Tuesday, January 23, 2018 11:17 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] Metadata only system pool Note: External Email ________________________________ Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Tue Jan 23 17:25:48 2018 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 23 Jan 2018 12:25:48 -0500 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at calicolabs.com Tue Jan 23 17:27:57 2018 From: alex at calicolabs.com (Alex Chekholko) Date: Tue, 23 Jan 2018 09:27:57 -0800 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: 2.8TB seems quite high for only 350M inodes. Are you sure you only have metadata in there? On Tue, Jan 23, 2018 at 9:25 AM, Frederick Stock wrote: > One possibility is the creation/expansion of directories or allocation of > indirect blocks for large files. > > Not sure if this is the issue here but at one time inode allocation was > considered slow and so folks may have pre-allocated inodes to avoid that > overhead during file creation. To my understanding inode creation time is > not so slow that users need to pre-allocate inodes. Yes, there are likely > some applications where pre-allocating may be necessary but I expect they > would be the exception. I mention this because you have a lot of free > inodes and of course once they are allocated they cannot be de-allocated. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821> > stockf at us.ibm.com > > > > From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/23/2018 12:17 PM > Subject: [gpfsug-discuss] Metadata only system pool > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi All, > > I was under the (possibly false) impression that if you have a filesystem > where the system pool contains metadata only then the only thing that would > cause the amount of free space in that pool to change is the creation of > more inodes ? is that correct? In other words, given that I have a > filesystem with 130 million free (but allocated) inodes: > > Inode Information > ----------------- > Number of used inodes: 218635454 > Number of free inodes: 131364674 > Number of allocated inodes: 350000128 > Maximum number of inodes: 350000128 > > I would not expect that a user creating a few hundred or thousands of > files could cause a ?no space left on device? error (which I?ve got one > user getting). There?s plenty of free data space, BTW. > > Now my system pool is almost ?full?: > > (pool total) 2.878T 34M ( 0%) > 140.9M ( 0%) > > But again, what - outside of me creating more inodes - would cause that to > change?? > > Thanks? > > Kevin > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > *Kevin.Buterbaugh at vanderbilt.edu* - > (615)875-9633 <(615)%20875-9633> > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m= > gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_ > vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cphoffma at uoregon.edu Tue Jan 23 17:27:52 2018 From: cphoffma at uoregon.edu (Chris Hoffman) Date: Tue, 23 Jan 2018 17:27:52 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu>, Message-ID: <1516728472608.73429@uoregon.edu> If you are still running out of space when mmdf shows preallocated inodes left I'd check MaxInodes vs AllocInodes for that fileset. You can run: mmlsfileset -L Thanks, Chris? ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederick Stock Sent: Tuesday, January 23, 2018 9:25 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Metadata only system pool One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ... is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a "no space left on device" error (which I've got one user getting). There's plenty of free data space, BTW. Now my system pool is almost "full": (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks... Kevin - Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 23 17:37:51 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 23 Jan 2018 17:37:51 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: Hi All, I do have metadata replication set to two, so Alex, does that make more sense? And I had forgotten about indirect blocks for large files, which actually makes sense with the user in question ? my apologies for that ? due to a very gravely ill pet and a recovering at home from pneumonia family member I?m way more sleep deprived right now than I?d like. :-( Fred - I think you?ve already answered this ? but mmchfs can only create / allocate more inodes ? it cannot be used to shrink the number of inodes? That would make sense, and if that?s the case then I can allocate more NSDs to the system pool. Thanks? Kevin On Jan 23, 2018, at 11:27 AM, Alex Chekholko > wrote: 2.8TB seems quite high for only 350M inodes. Are you sure you only have metadata in there? On Tue, Jan 23, 2018 at 9:25 AM, Frederick Stock > wrote: One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list > Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C1607a3fe872e4241587b08d56286a746%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636523252830007825&sdata=rIFx3lzbAIH5SZtFxJsVqWMMSo%2F0LssNc4K4tZH3uQc%3D&reserved=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Tue Jan 23 18:18:43 2018 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 23 Jan 2018 13:18:43 -0500 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: You are correct about mmchfs, you can increase the inode maximum but once an inode is allocated it cannot be de-allocated in the sense that the space can be recovered. You can of course decreased the inode maximum to a value equal to the used and allocated inodes but that would not help you here. Providing more metadata space via additional NSDs seems your most expedient option to address the issue. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 01:10 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I do have metadata replication set to two, so Alex, does that make more sense? And I had forgotten about indirect blocks for large files, which actually makes sense with the user in question ? my apologies for that ? due to a very gravely ill pet and a recovering at home from pneumonia family member I?m way more sleep deprived right now than I?d like. :-( Fred - I think you?ve already answered this ? but mmchfs can only create / allocate more inodes ? it cannot be used to shrink the number of inodes? That would make sense, and if that?s the case then I can allocate more NSDs to the system pool. Thanks? Kevin On Jan 23, 2018, at 11:27 AM, Alex Chekholko wrote: 2.8TB seems quite high for only 350M inodes. Are you sure you only have metadata in there? On Tue, Jan 23, 2018 at 9:25 AM, Frederick Stock wrote: One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C1607a3fe872e4241587b08d56286a746%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636523252830007825&sdata=rIFx3lzbAIH5SZtFxJsVqWMMSo%2F0LssNc4K4tZH3uQc%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=LN14zBEOYVrP2YRk3of_f08Ok5f256m7SYf1xL0qDvU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From UWEFALKE at de.ibm.com Tue Jan 23 19:03:40 2018 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Tue, 23 Jan 2018 20:03:40 +0100 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Tue Jan 23 19:12:50 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 23 Jan 2018 16:12:50 -0300 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: If one were starting over, it might make sense to use a smaller inode size. I believe we still support 512, 1K, 2K. Tradeoff with the fact that inodes can store data and EAs. From: "Uwe Falke" To: gpfsug main discussion list Date: 01/23/2018 04:04 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 23 19:25:54 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 23 Jan 2018 19:25:54 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: <9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> Hi All, This is all making sense and I appreciate everyone?s responses ? and again I apologize for not thinking about the indirect blocks. Marc - we specifically chose 4K inodes when we created this filesystem a little over a year ago so that small files could fit in the inode and therefore be stored on the metadata SSDs. This is more of a curiosity question ? is it documented somewhere how a 4K inode is used? I understand that for very small files up to 3.5K of that can be for data, but what about for large files? I.e., how much of that 4K is used for block addresses (3.5K plus whatever portion was already allocated to block addresses??) ? or what I?m really asking is, given 4K inodes and a 1M block size how big does a file have to be before it will need to use indirect blocks? Thanks again? Kevin On Jan 23, 2018, at 1:12 PM, Marc A Kaplan > wrote: If one were starting over, it might make sense to use a smaller inode size. I believe we still support 512, 1K, 2K. Tradeoff with the fact that inodes can store data and EAs. From: "Uwe Falke" > To: gpfsug main discussion list > Date: 01/23/2018 04:04 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C77fefde14ec54e04b35708d5629550bd%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636523315820548341&sdata=rbazh5e%2BxgHGvgF65VHTs9Hf4kk9EtUizsb19l5rr7U%3D&reserved=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From UWEFALKE at de.ibm.com Tue Jan 23 20:35:56 2018 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Tue, 23 Jan 2018 21:35:56 +0100 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> <9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> Message-ID: Hi, Kevin, (reproducing mainly what I've learned from Yuri) a 512B inode can hold about 32 Disk Addresses (DAs) addressing 32 physical disk blocks (but mind: if max replication - not necessarily actual repl. -- is set to more than 1, that mans for example 16 logical blocks (R=2). A DA is 12 bytes. With a 4k inode you can expect the other 3.5kiB to contain also DAs throughout (about 298) So with R=2 (even if actual replication is r=1), a 4k inode can hold max about 330 DAs addressing 165 logical blocks The max file size w/out indirect addressing is thus 165 x blocksize (e.g. 1320MiB @8MiB) However, there could be other data structures in an inode (EAs) reducing the space available for DAs Hence, YMMV Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 08:26 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, This is all making sense and I appreciate everyone?s responses ? and again I apologize for not thinking about the indirect blocks. Marc - we specifically chose 4K inodes when we created this filesystem a little over a year ago so that small files could fit in the inode and therefore be stored on the metadata SSDs. This is more of a curiosity question ? is it documented somewhere how a 4K inode is used? I understand that for very small files up to 3.5K of that can be for data, but what about for large files? I.e., how much of that 4K is used for block addresses (3.5K plus whatever portion was already allocated to block addresses??) ? or what I?m really asking is, given 4K inodes and a 1M block size how big does a file have to be before it will need to use indirect blocks? Thanks again? Kevin On Jan 23, 2018, at 1:12 PM, Marc A Kaplan wrote: If one were starting over, it might make sense to use a smaller inode size. I believe we still support 512, 1K, 2K. Tradeoff with the fact that inodes can store data and EAs. From: "Uwe Falke" To: gpfsug main discussion list Date: 01/23/2018 04:04 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C77fefde14ec54e04b35708d5629550bd%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636523315820548341&sdata=rbazh5e%2BxgHGvgF65VHTs9Hf4kk9EtUizsb19l5rr7U%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Tue Jan 23 21:13:20 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 23 Jan 2018 18:13:20 -0300 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu><9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> Message-ID: One way to know for sure it to do some experiments and dump some inodes with the command tsdbfs filesystem_name inode inode_number Of course improper use of tsdbfs command can destroy or corrupt your filesystems. So I take no responsibility for that. To stay safe, only use it on test filesystems you can afford to lose. (If you are root `dd if=/dev/zero of=/dev/some-disk-you-should-not-wipe` is also something you should not do!) -------------- next part -------------- An HTML attachment was scrubbed... URL: From hmorales at optimizeit.co Wed Jan 24 01:39:52 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Tue, 23 Jan 2018 20:39:52 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale Message-ID: Hello, Is there any proven strategy to replicate disks from one local storage to one storage on another site and have this as a spectrum scale backup strategy. Let me explain: Context: - site A simple three node cluster configured (AIX nodes). two failure groups. two filesystems. No quota, no ILM. - Site B the same cluster config as in site A configured (AIX nodes) same hdisk device names used as in site A for each and every NSDs - NO TCPIP connection between those two sites. Same ip addresses available locally on each one. and same non-scale based filesystems. Same scale version installed. (esentially site B is a copy from site A). My question is, can I replicate the spectrum scale disks from site A to site B storage to another (HP 3PAR) and maintain consistency between scale sites? can this be considered a valid DR configuration? People here don't want a stretch cluster (without any apparent reason, which talks very bad about them). Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Jan 24 06:22:33 2018 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Wed, 24 Jan 2018 06:22:33 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: How are you proposing to replicate the data between the sites? You mention no IP connection, so were you planning some other form of data shipping? I would suggest it would be a very high risk strategy to try this. Stretched cluster or ASYNC-DR are more appropriate for this type of DR solution. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of hmorales at optimizeit.co [hmorales at optimizeit.co] Sent: 24 January 2018 01:39 To: gpfsug main discussion list Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale Hello, Is there any proven strategy to replicate disks from one local storage to one storage on another site and have this as a spectrum scale backup strategy. Let me explain: Context: - site A simple three node cluster configured (AIX nodes). two failure groups. two filesystems. No quota, no ILM. - Site B the same cluster config as in site A configured (AIX nodes) same hdisk device names used as in site A for each and every NSDs - NO TCPIP connection between those two sites. Same ip addresses available locally on each one. and same non-scale based filesystems. Same scale version installed. (esentially site B is a copy from site A). My question is, can I replicate the spectrum scale disks from site A to site B storage to another (HP 3PAR) and maintain consistency between scale sites? can this be considered a valid DR configuration? People here don't want a stretch cluster (without any apparent reason, which talks very bad about them). Thanks. From hmorales at optimizeit.co Wed Jan 24 06:33:10 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Wed, 24 Jan 2018 01:33:10 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Wed Jan 24 07:03:46 2018 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 24 Jan 2018 07:03:46 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: Message-ID: Hi They are not even open for a routed L3 network? How they talk between DCs today then? I would really go L3 and AFM here if the sole purpose here is to have a DR -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous > On 24 Jan 2018, at 8.33, Harold Morales wrote: > > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Wed Jan 24 07:07:46 2018 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 24 Jan 2018 07:07:46 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm ? Seems to cover what you?re looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales : > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > IP connection between storage boxes but not between servers on both sites, > there is a dark fiber connecting both sites. Here they dont want to explore > the idea of a scaled-based. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alevin at gmail.com Wed Jan 24 07:30:15 2018 From: alevin at gmail.com (Alex Levin) Date: Wed, 24 Jan 2018 02:30:15 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Hi, We are using a similar type of replication. I assume the site B is the cold site prepared for DR The storage layer is EMC VMAX and the LUNs are replicated with SRDF. All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication group to ensure consistency. The cluster name, IP addresses , hostnames of the cluster nodes are different on another site - it can be a pre-configured cluster without gpfs filesystems or with another filesystem. Same names and addresses shouldn't be a problem. Additionally to the replicated LUNs/NSDs you need to deliver copy of /var/mmfs/gen/mmsdrfs file from A to B site. There is no need to replicate it in real-time, only after the change of the cluster configuration. To activate site B - present replicated LUNs to the nodes in the DR cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" Tested with multiples LUNs and filesystems on various workloads - seems to be working --Alex On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales wrote: > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > IP connection between storage boxes but not between servers on both sites, > there is a dark fiber connecting both sites. Here they dont want to explore > the idea of a scaled-based. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chetkulk at in.ibm.com Wed Jan 24 09:37:01 2018 From: chetkulk at in.ibm.com (Chetan R Kulkarni) Date: Wed, 24 Jan 2018 15:07:01 +0530 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: Message-ID: Hi, One can't enable/disable smb/nfs service if file authentication is already configured. Hence, with your already existing config; you can't enable NFS service directly. You need to re-configure file authentication (i.e. remove file auth, enable nfs service; configure file auth). May be following sequence of commands will help you. mmuserauth service remove --data-access-method file mmces service enable nfs mmces service list -a # check nfs is enabled and running mmuserauth service create --data-access-method file --type ad ..... # please complete this command as per your set up details (the way you already configured) FYI, Enabling NFS service won't affect other services. Thanks, Chetan. From: "Sobey, Richard A" To: gpfsug main discussion list Date: 01/23/2018 04:42 PM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services to halt/timeout/fail/stop/go slow/cause an earthquake or is it completely transparent? My existing config is: FILE access configuration : AD PARAMETERS VALUES ------------------------------------------------- ENABLE_NFS_KERBEROS true SERVERS server.domain USER_NAME username NETBIOS_NAME store IDMAP_ROLE master IDMAP_RANGE 10000000-299999999 IDMAP_RANGE_SIZE 1000000 UNIXMAP_DOMAINS DOMAIN(500 - 2000000) LDAPMAP_DOMAINS none OBJECT access not configured PARAMETERS VALUES Enabled services: SMB SMB is running Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Chetan R Kulkarni Sent: 23 January 2018 09:30 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. Inactive hide details for "Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a f"Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nf From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=mn6WfAqcjuBRZT4DaqmRBJ7KKacO2ma_baqc0GPm0PU&s=7ItSqugWx9SDxGAB2Odvj6oWUoFCiCzOH7cHdvRszY4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Wed Jan 24 16:13:01 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 24 Jan 2018 16:13:01 +0000 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , Message-ID: Gosh.. Seriously? I need downtime to enable NFS? Can that change in future releases? I have major issues configuring Ad file auth and I would not be confident getting my SMB working again in a timely manner. Thanks for letting me know anyway. Get Outlook for Android ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Chetan R Kulkarni Sent: Wednesday, January 24, 2018 9:37:01 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, One can't enable/disable smb/nfs service if file authentication is already configured. Hence, with your already existing config; you can't enable NFS service directly. You need to re-configure file authentication (i.e. remove file auth, enable nfs service; configure file auth). May be following sequence of commands will help you. mmuserauth service remove --data-access-method file mmces service enable nfs mmces service list -a # check nfs is enabled and running mmuserauth service create --data-access-method file --type ad ..... # please complete this command as per your set up details (the way you already configured) FYI, Enabling NFS service won't affect other services. Thanks, Chetan. [Inactive hide details for "Sobey, Richard A" ---01/23/2018 04:42:33 PM---Many thanks Chetan. Next question, does enabling the N]"Sobey, Richard A" ---01/23/2018 04:42:33 PM---Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services From: "Sobey, Richard A" To: gpfsug main discussion list Date: 01/23/2018 04:42 PM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services to halt/timeout/fail/stop/go slow/cause an earthquake or is it completely transparent? My existing config is: FILE access configuration : AD PARAMETERS VALUES ------------------------------------------------- ENABLE_NFS_KERBEROS true SERVERS server.domain USER_NAME username NETBIOS_NAME store IDMAP_ROLE master IDMAP_RANGE 10000000-299999999 IDMAP_RANGE_SIZE 1000000 UNIXMAP_DOMAINS DOMAIN(500 - 2000000) LDAPMAP_DOMAINS none OBJECT access not configured PARAMETERS VALUES Enabled services: SMB SMB is running Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Chetan R Kulkarni Sent: 23 January 2018 09:30 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. [Inactive hide details for "Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a f]"Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nf From: "Sobey, Richard A" > To: "'gpfsug-discuss at spectrumscale.org'" > Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=mn6WfAqcjuBRZT4DaqmRBJ7KKacO2ma_baqc0GPm0PU&s=7ItSqugWx9SDxGAB2Odvj6oWUoFCiCzOH7cHdvRszY4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From valdis.kletnieks at vt.edu Wed Jan 24 16:28:21 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 24 Jan 2018 11:28:21 -0500 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , Message-ID: <17153.1516811301@turing-police.cc.vt.edu> On Wed, 24 Jan 2018 16:13:01 +0000, "Sobey, Richard A" said: > Gosh.. Seriously? I need downtime to enable NFS? Wait till you get to the part where 'mmnfs export change /foo/whatever --nfsadd (whatever)' bounces your nfs-ganesha services on all protocol nodes - at the same time. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From valdis.kletnieks at vt.edu Wed Jan 24 16:31:26 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 24 Jan 2018 11:31:26 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: <17310.1516811486@turing-police.cc.vt.edu> On Tue, 23 Jan 2018 20:39:52 -0500, Harold Morales said: > - site A simple three node cluster configured (AIX nodes). two failure > groups. two filesystems. No quota, no ILM. > - Site B the same cluster config as in site A configured (AIX nodes) same > hdisk device names used as in site A for each and every NSDs > - NO TCPIP connection between those two sites. Same ip addresses available > locally on each one. and same non-scale based filesystems. Same scale > version installed. (esentially site B is a copy from site A). You're going to have a *really* bad time with semantic stability at site B, as while it's reading the replica from site A, it will continually be reading metadata and data blocks that it didn't write there. (Thought experiment - what happens when site B thinks there's 3 files in a directory, and a replication block from A changes that to 2 without B's knowledge?) Be ready to be dealing with filesystem issues at B - it's *always* bad karma to do writes to a filesystem that the operating system doesn't know about.... If B is a *cold* backup site, this should work just fine and you can ignore all this. But if B has the filesystem mounted while A is making changes, it *will* go pear shaped on you. > People here don't want a stretch cluster (without any apparent reason, which > talks very bad about them). For what it's worth, we're running a stretch cluster with about 95 cable miles separating the two halves. Biggest problem we have is that a network hiccup can cause the cluster to partition and down the NSD's at the remote site (yes, it's designed that way - 5 nodes at each site, and on a partition the main site has enough quorum nodes to stay up, but the remote site doesn't. So it downs the NSDs' there. Just means a 'mmchdisk start -a' once the cluster recovers all the nodes at the far end (which is a lot less painful than the alternative, which is a split-brain situation). (And yeah, we 're working on getting reliable alternate network connections. Just a bit hard to find another 10G link to light up that isn't fate-sharing *and* plays nice with the rest of our BGP routing and can be fixed to have failover fast enough for GPFS to not notice the blink. We're in the part of Virginia that doesn't have dark fiber all over the place.) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From stockf at us.ibm.com Wed Jan 24 16:36:55 2018 From: stockf at us.ibm.com (Frederick Stock) Date: Wed, 24 Jan 2018 11:36:55 -0500 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: <17153.1516811301@turing-police.cc.vt.edu> References: , <17153.1516811301@turing-police.cc.vt.edu> Message-ID: Thankfully the issue of Ganesha being restarted across the cluster when an export is changed has been addressed in the 5.0 release of Spectrum Scale. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list Date: 01/24/2018 11:34 AM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org On Wed, 24 Jan 2018 16:13:01 +0000, "Sobey, Richard A" said: > Gosh.. Seriously? I need downtime to enable NFS? Wait till you get to the part where 'mmnfs export change /foo/whatever --nfsadd (whatever)' bounces your nfs-ganesha services on all protocol nodes - at the same time. [attachment "attxn3je.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Wed Jan 24 19:16:44 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 24 Jan 2018 19:16:44 +0000 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , <17153.1516811301@turing-police.cc.vt.edu>, Message-ID: I saw that yes. Maybe I will wait until I upgrade to v5 and sort out NFS then - one downtime window to rule them all. Richard Get Outlook for Android ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederick Stock Sent: Wednesday, January 24, 2018 4:36:55 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Thankfully the issue of Ganesha being restarted across the cluster when an export is changed has been addressed in the 5.0 release of Spectrum Scale. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list Date: 01/24/2018 11:34 AM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ On Wed, 24 Jan 2018 16:13:01 +0000, "Sobey, Richard A" said: > Gosh.. Seriously? I need downtime to enable NFS? Wait till you get to the part where 'mmnfs export change /foo/whatever --nfsadd (whatever)' bounces your nfs-ganesha services on all protocol nodes - at the same time. [attachment "attxn3je.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Wed Jan 24 19:31:41 2018 From: chair at spectrumscale.org (Simon Thompson (Spectrum Scale UG Chair)) Date: Wed, 24 Jan 2018 19:31:41 +0000 Subject: [gpfsug-discuss] UK April Meeting Agenda Message-ID: <7dd288e6-679e-4bd8-9803-b5f21607e92a@email.android.com> An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Wed Jan 24 20:32:00 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 24 Jan 2018 15:32:00 -0500 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , <17153.1516811301@turing-police.cc.vt.edu> Message-ID: <50389.1516825920@turing-police.cc.vt.edu> On Wed, 24 Jan 2018 11:36:55 -0500, "Frederick Stock" said: > Thankfully the issue of Ganesha being restarted across the cluster when an > export is changed has been addressed in the 5.0 release of Spectrum Scale. Thanks for the info. Now all I need is a version of LTFS/EE that supports 5.0. :) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From janfrode at tanso.net Thu Jan 25 08:27:37 2018 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 25 Jan 2018 09:27:37 +0100 Subject: [gpfsug-discuss] AFM and RHEL 7.4 Message-ID: The FAQ has a note stating: 1. AFM, Asynch Disaster Recovery with AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4. Could someone please clarify this sentence ? It can't be right that none of these features are supported with RHEL 7.4, or .. ? -jf -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Thu Jan 25 09:53:06 2018 From: john.hearns at asml.com (John Hearns) Date: Thu, 25 Jan 2018 09:53:06 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Jan Frode, thankyou for that link. I have a general question regarding remote GPFS filesystems. If we have two clusters, in separate locations on separate Infiniband fabrics, we can set up a remote relationship between filesystems. As Valdis discusses, what happens if the IP link between the clusters goes down or is unstable? Can nodes in one cluster vote out nodes in the other cluster? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: Wednesday, January 24, 2018 8:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm ? Seems to cover what you?re looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales >: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Thu Jan 25 10:08:38 2018 From: YARD at il.ibm.com (Yaron Daniel) Date: Thu, 25 Jan 2018 12:08:38 +0200 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Hi You can do remote mount between GPFS clusters. https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_admmcch.htm Where is you daemon communication network ? IP ir IP Over IB ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: John Hearns To: gpfsug main discussion list Date: 01/25/2018 11:53 AM Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Sent by: gpfsug-discuss-bounces at spectrumscale.org Jan Frode, thankyou for that link. I have a general question regarding remote GPFS filesystems. If we have two clusters, in separate locations on separate Infiniband fabrics, we can set up a remote relationship between filesystems. As Valdis discusses, what happens if the IP link between the clusters goes down or is unstable? Can nodes in one cluster vote out nodes in the other cluster? From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: Wednesday, January 24, 2018 8:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm ? Seems to cover what you?re looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales : Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=BMyrARi4hlfjG-EugDznaiyWSqErGF5FyFpQQ-o4ScU&s=R0w70yvIjZaXpcs4P2mGJNQYBSNlUp3aZcCNks-sveU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4376 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 5093 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4746 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4557 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 11294 bytes Desc: not available URL: From Achim.Rehor at de.ibm.com Thu Jan 25 10:20:41 2018 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Thu, 25 Jan 2018 11:20:41 +0100 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: From john.hearns at asml.com Thu Jan 25 10:44:28 2018 From: john.hearns at asml.com (John Hearns) Date: Thu, 25 Jan 2018 10:44:28 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Achim, Thankyou for your clear reply. Yes indeed we are using AFM at the moment. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Achim Rehor Sent: Thursday, January 25, 2018 11:21 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale John, yes, they definitely can! Nodes in a remote cluster are tob viewed just as local nodes in terms of taking part in the mechanisms of access to data. Token management will be done just as with local nodes. So if one node in cluster A recognizes a communication issue with a node in cluster B, it will let the clustermgr know, and that one then decides on whether to expel one or the other. Having a remote cluster connected relies on a stable and low latency network, just as a local cluster does. if your network is not reliable, you woudl go for AFM or other replication mechanisms (as the thread title implies;) ) Mit freundlichen Gr??en / Kind regards Achim Rehor ________________________________ Software Technical Support Specialist AIX/ Emea HPC Support [cid:image001.gif at 01D395D1.DA27C7E0] IBM Certified Advanced Technical Expert - Power Systems with AIX TSCC Software Service, Dept. 7922 Global Technology Services ________________________________ Phone: +49-7034-274-7862 IBM Deutschland E-Mail: Achim.Rehor at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany ________________________________ IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martin Hartmann (Vorsitzender), Norbert Janzen, Stefan Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 WEEE-Reg.-Nr. DE 99369940 From: John Hearns > To: gpfsug main discussion list > Date: 25/01/2018 10:53 Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Jan Frode, thankyou for that link. I have a general question regarding remote GPFS filesystems. If we have two clusters, in separate locations on separate Infiniband fabrics, we can set up a remote relationship between filesystems. As Valdis discusses, what happens if the IP link between the clusters goes down or is unstable? Can nodes in one cluster vote out nodes in the other cluster? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: Wednesday, January 24, 2018 8:08 AM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm? Seems to cover what you're looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales >: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 7182 bytes Desc: image001.gif URL: From olaf.weiser at de.ibm.com Thu Jan 25 13:17:52 2018 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 25 Jan 2018 14:17:52 +0100 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 14941 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: From hmorales at optimizeit.co Fri Jan 26 18:29:09 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Fri, 26 Jan 2018 13:29:09 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Hi Alex, This set up seems close to what I am trying to achieve. With regards to this kind of replication: any prereqs need to be met in the target environment for this to work? for example, should disk devices naming on AIX be the same as in the source environment? when importing the mmsdrfs file, how is Scale going to know which disks should it assign to the cluster? by their hdisk name alone? Thanks again, 2018-01-24 2:30 GMT-05:00 Alex Levin : > Hi, > > We are using a similar type of replication. > I assume the site B is the cold site prepared for DR > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > group to ensure consistency. > > The cluster name, IP addresses , hostnames of the cluster nodes are > different on another site - it can be a pre-configured cluster without > gpfs filesystems or with another filesystem. > Same names and addresses shouldn't be a problem. > > Additionally to the replicated LUNs/NSDs you need to deliver copy > of /var/mmfs/gen/mmsdrfs file from A to B site. > There is no need to replicate it in real-time, only after the change of > the cluster configuration. > > To activate site B - present replicated LUNs to the nodes in the DR > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > Tested with multiples LUNs and filesystems on various workloads - seems > to be working > > --Alex > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > wrote: > >> Thanks for answering. >> >> Essentially, the idea being explored is to replicate LUNs between >> identical storage hardware (HP 3PAR volumesrein) on both sites. There is an >> IP connection between storage boxes but not between servers on both sites, >> there is a dark fiber connecting both sites. Here they dont want to explore >> the idea of a scaled-based. >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gcorneau at us.ibm.com Fri Jan 26 20:21:15 2018 From: gcorneau at us.ibm.com (Glen Corneau) Date: Fri, 26 Jan 2018 14:21:15 -0600 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Scale will walk across all discovered disks upon start time and attempt to read the NSD identifiers from the disks. Once it finds them, it makes a local map file that correlates the NSD id and the hdiskX identifier. The names do not have to be the same as either the source cluster or even from node-to-node. The main thing to keep in mind is to keep the file system definitions in sync between the source and destination clusters. The "syncFSconfig" user exit is the best way to do it because it's automatic. You generally shouldn't be shuffling the mmsdrfs file between sites, that's what the "mmfsctl syncFSconfig" does for you, on a per-file system basis. GPFS+AIX customers have been using this kind of storage replication for over 10 years, it's business as usual. ------------------ Glen Corneau Power Systems Washington Systems Center gcorneau at us.ibm.com From: Harold Morales To: gpfsug main discussion list Date: 01/26/2018 12:30 PM Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Alex, This set up seems close to what I am trying to achieve. With regards to this kind of replication: any prereqs need to be met in the target environment for this to work? for example, should disk devices naming on AIX be the same as in the source environment? when importing the mmsdrfs file, how is Scale going to know which disks should it assign to the cluster? by their hdisk name alone? Thanks again, 2018-01-24 2:30 GMT-05:00 Alex Levin : Hi, We are using a similar type of replication. I assume the site B is the cold site prepared for DR The storage layer is EMC VMAX and the LUNs are replicated with SRDF. All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication group to ensure consistency. The cluster name, IP addresses , hostnames of the cluster nodes are different on another site - it can be a pre-configured cluster without gpfs filesystems or with another filesystem. Same names and addresses shouldn't be a problem. Additionally to the replicated LUNs/NSDs you need to deliver copy of /var/mmfs/gen/mmsdrfs file from A to B site. There is no need to replicate it in real-time, only after the change of the cluster configuration. To activate site B - present replicated LUNs to the nodes in the DR cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" Tested with multiples LUNs and filesystems on various workloads - seems to be working --Alex On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales wrote: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=d-vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m=VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh-GeVJiVS49opBwwCFs-FKQrQ7rc&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 26117 bytes Desc: not available URL: From hmorales at optimizeit.co Fri Jan 26 20:47:03 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Fri, 26 Jan 2018 15:47:03 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Thanks for participating in the discussion. Immediately after replication I am getting the error documented below, I have moved the mmsdrfs file (upon deleting the previous filesystem definitions, because I have managed to configure exactly equal my source and target clusters). Even then, I am getting the following error: GPFS: 6027-419 Failed to read a file system descriptor. There is an input or output error. mmlsfs: 6027-1639 Command failed. Examine previous That's the same error that upon first replicating without taking any other action. I think I am missing something really important but still dont know what it is. 2018-01-26 15:21 GMT-05:00 Glen Corneau : > Scale will walk across all discovered disks upon start time and attempt to > read the NSD identifiers from the disks. Once it finds them, it makes a > local map file that correlates the NSD id and the hdiskX identifier. The > names do not have to be the same as either the source cluster or even from > node-to-node. > > The main thing to keep in mind is to keep the file system definitions in > sync between the source and destination clusters. The "syncFSconfig" user > exit is the best way to do it because it's automatic. You generally > shouldn't be shuffling the mmsdrfs file between sites, that's what the > "mmfsctl syncFSconfig" does for you, on a per-file system basis. > > GPFS+AIX customers have been using this kind of storage replication for > over 10 years, it's business as usual. > > ------------------ > Glen Corneau > Power Systems > Washington Systems Center > gcorneau at us.ibm.com > > > > > > From: Harold Morales > To: gpfsug main discussion list > Date: 01/26/2018 12:30 PM > Subject: Re: [gpfsug-discuss] storage-based replication for > Spectrum Scale > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > With regards to this kind of replication: any prereqs need to be met in > the target environment for this to work? for example, should disk devices > naming on AIX be the same as in the source environment? when importing the > mmsdrfs file, how is Scale going to know which disks should it assign to > the cluster? by their hdisk name alone? > > Thanks again, > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin <*alevin at gmail.com* > >: > Hi, > > We are using a similar type of replication. > I assume the site B is the cold site prepared for DR > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > group to ensure consistency. > > The cluster name, IP addresses , hostnames of the cluster nodes are > different on another site - it can be a pre-configured cluster without > gpfs filesystems or with another filesystem. > Same names and addresses shouldn't be a problem. > > Additionally to the replicated LUNs/NSDs you need to deliver copy > of /var/mmfs/gen/mmsdrfs file from A to B site. > There is no need to replicate it in real-time, only after the change of > the cluster configuration. > > To activate site B - present replicated LUNs to the nodes in the DR > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > Tested with multiples LUNs and filesystems on various workloads - seems > to be working > > --Alex > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales <*hmorales at optimizeit.co* > > wrote: > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > IP connection between storage boxes but not between servers on both sites, > there is a dark fiber connecting both sites. Here they dont want to explore > the idea of a scaled-based. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at *spectrumscale.org* > > *http://gpfsug.org/mailman/listinfo/gpfsug-discuss* > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at *spectrumscale.org* > > *http://gpfsug.org/mailman/listinfo/gpfsug-discuss* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=d-vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m= > VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- > GeVJiVS49opBwwCFs-FKQrQ7rc&e= > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 26117 bytes Desc: not available URL: From sxiao at us.ibm.com Fri Jan 26 21:04:03 2018 From: sxiao at us.ibm.com (Steve Xiao) Date: Fri, 26 Jan 2018 16:04:03 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: When using this method of replication, you need to either issue "mmfsctl suspend or suspend-write" command before replication or setup a single consistency group for all LUNs. This is needed to ensure replica contain a consistent copy of GPFS data. Steve Y. Xiao gpfsug-discuss-bounces at spectrumscale.org wrote on 01/26/2018 03:21:23 PM: > From: gpfsug-discuss-request at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Date: 01/26/2018 03:21 PM > Subject: gpfsug-discuss Digest, Vol 72, Issue 69 > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: storage-based replication for Spectrum Scale (Harold Morales) > 2. Re: storage-based replication for Spectrum Scale (Glen Corneau) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 26 Jan 2018 13:29:09 -0500 > From: Harold Morales > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > Scale > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Hi Alex, This set up seems close to what I am trying to achieve. > > With regards to this kind of replication: any prereqs need to be met in the > target environment for this to work? for example, should disk devices > naming on AIX be the same as in the source environment? when importing the > mmsdrfs file, how is Scale going to know which disks should it assign to > the cluster? by their hdisk name alone? > > Thanks again, > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > > > Hi, > > > > We are using a similar type of replication. > > I assume the site B is the cold site prepared for DR > > > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > > group to ensure consistency. > > > > The cluster name, IP addresses , hostnames of the cluster nodes are > > different on another site - it can be a pre-configured cluster without > > gpfs filesystems or with another filesystem. > > Same names and addresses shouldn't be a problem. > > > > Additionally to the replicated LUNs/NSDs you need to deliver copy > > of /var/mmfs/gen/mmsdrfs file from A to B site. > > There is no need to replicate it in real-time, only after the change of > > the cluster configuration. > > > > To activate site B - present replicated LUNs to the nodes in the DR > > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > > > Tested with multiples LUNs and filesystems on various workloads - seems > > to be working > > > > --Alex > > > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > > wrote: > > > >> Thanks for answering. > >> > >> Essentially, the idea being explored is to replicate LUNs between > >> identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > >> IP connection between storage boxes but not between servers on both sites, > >> there is a dark fiber connecting both sites. Here they dont want to explore > >> the idea of a scaled-based. > >> > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > >> > >> > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180126_1503995b_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=SKdMmQae8uzHNWZq3vuRTp5UVwYFeeusLAxtbaposX0&e=> > > ------------------------------ > > Message: 2 > Date: Fri, 26 Jan 2018 14:21:15 -0600 > From: "Glen Corneau" > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > Scale > Message-ID: > 006FCEEE at notes.na.collabserv.com> > > Content-Type: text/plain; charset="us-ascii" > > Scale will walk across all discovered disks upon start time and attempt to > read the NSD identifiers from the disks. Once it finds them, it makes a > local map file that correlates the NSD id and the hdiskX identifier. The > names do not have to be the same as either the source cluster or even from > node-to-node. > > The main thing to keep in mind is to keep the file system definitions in > sync between the source and destination clusters. The "syncFSconfig" user > exit is the best way to do it because it's automatic. You generally > shouldn't be shuffling the mmsdrfs file between sites, that's what the > "mmfsctl syncFSconfig" does for you, on a per-file system basis. > > GPFS+AIX customers have been using this kind of storage replication for > over 10 years, it's business as usual. > > ------------------ > Glen Corneau > Power Systems > Washington Systems Center > gcorneau at us.ibm.com > > > > > > From: Harold Morales > To: gpfsug main discussion list > Date: 01/26/2018 12:30 PM > Subject: Re: [gpfsug-discuss] storage-based replication for > Spectrum Scale > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > With regards to this kind of replication: any prereqs need to be met in > the target environment for this to work? for example, should disk devices > naming on AIX be the same as in the source environment? when importing the > mmsdrfs file, how is Scale going to know which disks should it assign to > the cluster? by their hdisk name alone? > > Thanks again, > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > Hi, > > We are using a similar type of replication. > I assume the site B is the cold site prepared for DR > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > group to ensure consistency. > > The cluster name, IP addresses , hostnames of the cluster nodes are > different on another site - it can be a pre-configured cluster without > gpfs filesystems or with another filesystem. > Same names and addresses shouldn't be a problem. > > Additionally to the replicated LUNs/NSDs you need to deliver copy > of /var/mmfs/gen/mmsdrfs file from A to B site. > There is no need to replicate it in real-time, only after the change of > the cluster configuration. > > To activate site B - present replicated LUNs to the nodes in the DR > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > Tested with multiples LUNs and filesystems on various workloads - seems > to be working > > --Alex > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > wrote: > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is > an IP connection between storage boxes but not between servers on both > sites, there is a dark fiber connecting both sites. Here they dont want to > explore the idea of a scaled-based. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=d- > vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m=VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- > GeVJiVS49opBwwCFs-FKQrQ7rc&e= > > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180126_e291af63_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=bYnf-7v0CxYUkGth-QaVeUQdIlG8f1Gro-hwOxok7Qw&e=> > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: not available > Type: image/jpeg > Size: 26117 bytes > Desc: not available > URL: u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180126_e291af63_attachment.jpe&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=jYdnqhQBlnpf58oxunzBcTs9XdcbeOtLDQdgnASidDA&e=> > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > End of gpfsug-discuss Digest, Vol 72, Issue 69 > ********************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alevin at gmail.com Sat Jan 27 05:02:40 2018 From: alevin at gmail.com (Alex Levin) Date: Sat, 27 Jan 2018 00:02:40 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Steve, I've read that "mmfsctl suspend or suspend-write" should be executed, but in real life it is impossible for DR scenario. We tested both cases, the graceful one when the failover to another site is planned, applications stopped and i/o suspended and the case when there was no advanced notice about the disaster in the primary site. Both worked and for the second case various loads were simulated including heavy writes and reads/writes. In disaster model as expected some data were lost (due to incomplete writes, replication latency ... ), but mmfsck was always able to repair and the applications databases located on the affected filesystem were in an acceptable state. it is possible that we were just lucky and the test case didn't cover all possible scenarios. Harold, In our case, it is Linux, not AIX, but it shouldn't matter And our DR cluster is fully configured ( different IPs, hostnames and cluster name ) and running without filesystems at all or with a differently named filesystem. Before running mmimport , make sure that all expected LUNs are visible and writable. You can verify if the device is correct by reading first blocks of the device ( for example dd if=/dev/NSD_LUN_device bs=1M count=16 | strings ) not sure where you are " moved the mmsdrfs" you don't need to move/modify mmsdrfs file on the target ( disaster recovery ) cluster Just copy the one from primary to /tmp or /var/tmp and try to run mmimportfs fs_name -i copy_of_mmsdrfs /tmp/mmsdrfs Glen, as Harold has no IP connectivity between the clusters "mmfsctl syncFSconfig" is not an option ... Thanks --Alex On Fri, Jan 26, 2018 at 4:04 PM, Steve Xiao wrote: > When using this method of replication, you need to either issue "mmfsctl > suspend or suspend-write" command before replication or setup a single > consistency group for all LUNs. This is needed to ensure replica contain > a consistent copy of GPFS data. > > Steve Y. Xiao > > gpfsug-discuss-bounces at spectrumscale.org wrote on 01/26/2018 03:21:23 PM: > > > From: gpfsug-discuss-request at spectrumscale.org > > To: gpfsug-discuss at spectrumscale.org > > Date: 01/26/2018 03:21 PM > > Subject: gpfsug-discuss Digest, Vol 72, Issue 69 > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Send gpfsug-discuss mailing list submissions to > > gpfsug-discuss at spectrumscale.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > or, via email, send a message with subject or body 'help' to > > gpfsug-discuss-request at spectrumscale.org > > > > You can reach the person managing the list at > > gpfsug-discuss-owner at spectrumscale.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of gpfsug-discuss digest..." > > > > > > Today's Topics: > > > > 1. Re: storage-based replication for Spectrum Scale (Harold Morales) > > 2. Re: storage-based replication for Spectrum Scale (Glen Corneau) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Fri, 26 Jan 2018 13:29:09 -0500 > > From: Harold Morales > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > > Scale > > Message-ID: > > > > Content-Type: text/plain; charset="utf-8" > > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > > > With regards to this kind of replication: any prereqs need to be met in > the > > target environment for this to work? for example, should disk devices > > naming on AIX be the same as in the source environment? when importing > the > > mmsdrfs file, how is Scale going to know which disks should it assign to > > the cluster? by their hdisk name alone? > > > > Thanks again, > > > > > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > > > > > Hi, > > > > > > We are using a similar type of replication. > > > I assume the site B is the cold site prepared for DR > > > > > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX > replication > > > group to ensure consistency. > > > > > > The cluster name, IP addresses , hostnames of the cluster nodes are > > > different on another site - it can be a pre-configured cluster without > > > gpfs filesystems or with another filesystem. > > > Same names and addresses shouldn't be a problem. > > > > > > Additionally to the replicated LUNs/NSDs you need to deliver copy > > > of /var/mmfs/gen/mmsdrfs file from A to B site. > > > There is no need to replicate it in real-time, only after the change of > > > the cluster configuration. > > > > > > To activate site B - present replicated LUNs to the nodes in the DR > > > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > > > > > Tested with multiples LUNs and filesystems on various workloads - > seems > > > to be working > > > > > > --Alex > > > > > > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales < > hmorales at optimizeit.co> > > > wrote: > > > > > >> Thanks for answering. > > >> > > >> Essentially, the idea being explored is to replicate LUNs between > > >> identical storage hardware (HP 3PAR volumesrein) on both sites. There > is an > > >> IP connection between storage boxes but not between servers on both > sites, > > >> there is a dark fiber connecting both sites. Here they dont want to > explore > > >> the idea of a scaled-based. > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > >> > > >> > > > > > > _______________________________________________ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_ > attachments_20180126_1503995b_attachment-2D0001.html&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=SKdMmQae8uzHNWZq3vuRTp5UVwYFeeusLAxtbaposX0&e=> > > > > ------------------------------ > > > > Message: 2 > > Date: Fri, 26 Jan 2018 14:21:15 -0600 > > From: "Glen Corneau" > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > > Scale > > Message-ID: > > > 006FCEEE at notes.na.collabserv.com> > > > > Content-Type: text/plain; charset="us-ascii" > > > > Scale will walk across all discovered disks upon start time and attempt > to > > read the NSD identifiers from the disks. Once it finds them, it makes a > > local map file that correlates the NSD id and the hdiskX identifier. > The > > names do not have to be the same as either the source cluster or even > from > > node-to-node. > > > > The main thing to keep in mind is to keep the file system definitions in > > sync between the source and destination clusters. The "syncFSconfig" > user > > exit is the best way to do it because it's automatic. You generally > > shouldn't be shuffling the mmsdrfs file between sites, that's what the > > "mmfsctl syncFSconfig" does for you, on a per-file system basis. > > > > GPFS+AIX customers have been using this kind of storage replication for > > over 10 years, it's business as usual. > > > > ------------------ > > Glen Corneau > > Power Systems > > Washington Systems Center > > gcorneau at us.ibm.com > > > > > > > > > > > > From: Harold Morales > > To: gpfsug main discussion list > > Date: 01/26/2018 12:30 PM > > Subject: Re: [gpfsug-discuss] storage-based replication for > > Spectrum Scale > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > > > With regards to this kind of replication: any prereqs need to be met in > > the target environment for this to work? for example, should disk > devices > > naming on AIX be the same as in the source environment? when importing > the > > mmsdrfs file, how is Scale going to know which disks should it assign to > > the cluster? by their hdisk name alone? > > > > Thanks again, > > > > > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > > Hi, > > > > We are using a similar type of replication. > > I assume the site B is the cold site prepared for DR > > > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX > replication > > group to ensure consistency. > > > > The cluster name, IP addresses , hostnames of the cluster nodes are > > different on another site - it can be a pre-configured cluster without > > gpfs filesystems or with another filesystem. > > Same names and addresses shouldn't be a problem. > > > > Additionally to the replicated LUNs/NSDs you need to deliver copy > > of /var/mmfs/gen/mmsdrfs file from A to B site. > > There is no need to replicate it in real-time, only after the change of > > the cluster configuration. > > > > To activate site B - present replicated LUNs to the nodes in the DR > > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > > > Tested with multiples LUNs and filesystems on various workloads - seems > > to be working > > > > --Alex > > > > > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > > > wrote: > > Thanks for answering. > > > > Essentially, the idea being explored is to replicate LUNs between > > identical storage hardware (HP 3PAR volumesrein) on both sites. There is > > an IP connection between storage boxes but not between servers on both > > sites, there is a dark fiber connecting both sites. Here they dont want > to > > explore the idea of a scaled-based. > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=d- > > vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m= > VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- > > GeVJiVS49opBwwCFs-FKQrQ7rc&e= > > > > > > > > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_ > attachments_20180126_e291af63_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=bYnf-7v0CxYUkGth-QaVeUQdIlG8f1Gro-hwOxok7Qw&e=> > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: not available > > Type: image/jpeg > > Size: 26117 bytes > > Desc: not available > > URL: > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_ > attachments_20180126_e291af63_attachment.jpe&d=DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=jYdnqhQBlnpf58oxunzBcTs9XdcbeOtLDQdgnASidDA&e=> > > > > ------------------------------ > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > > End of gpfsug-discuss Digest, Vol 72, Issue 69 > > ********************************************** > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hmorales at optimizeit.co Sat Jan 27 09:11:44 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Sat, 27 Jan 2018 04:11:44 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Thanks for your insights. Alex, we did as you mentioned but after using mmimportfs there are a lot of errors on all commands having to do with filesystems: GPFS: 6027-419 Failed to read a file system descriptor. There is an input or output error that occurs for: mmlsfs mmlsdisk mmdf obviously, fs won't mount. 2018-01-27 0:02 GMT-05:00 Alex Levin : > Steve, > I've read that "mmfsctl suspend or suspend-write" should be executed, > but in real life it is impossible for DR scenario. > > We tested both cases, > the graceful one when the failover to another site is planned, > applications stopped and i/o suspended > and the case when there was no advanced notice about the disaster in the > primary site. > > Both worked and for the second case various loads were simulated including > heavy writes and reads/writes. > In disaster model as expected some data were lost (due to incomplete > writes, replication latency ... ), > but mmfsck was always able to repair and the applications databases > located on the affected filesystem were in an acceptable state. > > > it is possible that we were just lucky and the test case didn't cover all > possible scenarios. > > Harold, > In our case, it is Linux, not AIX, but it shouldn't matter > And our DR cluster is fully configured ( different IPs, hostnames and > cluster name ) and running without filesystems at all or with > a differently named filesystem. > > Before running mmimport , make sure that all expected LUNs are visible > and writable. > You can verify if the device is correct by reading first blocks of the > device ( for example dd if=/dev/NSD_LUN_device bs=1M count=16 | strings ) > > not sure where you are " moved the mmsdrfs" > you don't need to move/modify mmsdrfs file on the target ( disaster > recovery ) cluster > > Just copy the one from primary to /tmp or /var/tmp and try to run mmimportfs > fs_name -i copy_of_mmsdrfs /tmp/mmsdrfs > > > > Glen, as Harold has no IP connectivity between the clusters "mmfsctl > syncFSconfig" is not an option ... > > Thanks > --Alex > > > > > On Fri, Jan 26, 2018 at 4:04 PM, Steve Xiao wrote: > >> When using this method of replication, you need to either issue "mmfsctl >> suspend or suspend-write" command before replication or setup a single >> consistency group for all LUNs. This is needed to ensure replica contain >> a consistent copy of GPFS data. >> >> Steve Y. Xiao >> >> gpfsug-discuss-bounces at spectrumscale.org wrote on 01/26/2018 03:21:23 PM: >> >> > From: gpfsug-discuss-request at spectrumscale.org >> > To: gpfsug-discuss at spectrumscale.org >> > Date: 01/26/2018 03:21 PM >> > Subject: gpfsug-discuss Digest, Vol 72, Issue 69 >> > Sent by: gpfsug-discuss-bounces at spectrumscale.org >> > >> > Send gpfsug-discuss mailing list submissions to >> > gpfsug-discuss at spectrumscale.org >> > >> > To subscribe or unsubscribe via the World Wide Web, visit >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > or, via email, send a message with subject or body 'help' to >> > gpfsug-discuss-request at spectrumscale.org >> > >> > You can reach the person managing the list at >> > gpfsug-discuss-owner at spectrumscale.org >> > >> > When replying, please edit your Subject line so it is more specific >> > than "Re: Contents of gpfsug-discuss digest..." >> > >> > >> > Today's Topics: >> > >> > 1. Re: storage-based replication for Spectrum Scale (Harold Morales) >> > 2. Re: storage-based replication for Spectrum Scale (Glen Corneau) >> > >> > >> > ---------------------------------------------------------------------- >> > >> > Message: 1 >> > Date: Fri, 26 Jan 2018 13:29:09 -0500 >> > From: Harold Morales >> > To: gpfsug main discussion list >> > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum >> > Scale >> > Message-ID: >> > >> > Content-Type: text/plain; charset="utf-8" >> >> > >> > Hi Alex, This set up seems close to what I am trying to achieve. >> > >> > With regards to this kind of replication: any prereqs need to be met in >> the >> > target environment for this to work? for example, should disk devices >> > naming on AIX be the same as in the source environment? when importing >> the >> > mmsdrfs file, how is Scale going to know which disks should it assign to >> > the cluster? by their hdisk name alone? >> > >> > Thanks again, >> > >> > >> > >> > 2018-01-24 2:30 GMT-05:00 Alex Levin : >> > >> > > Hi, >> > > >> > > We are using a similar type of replication. >> > > I assume the site B is the cold site prepared for DR >> > > >> > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. >> > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX >> replication >> > > group to ensure consistency. >> > > >> > > The cluster name, IP addresses , hostnames of the cluster nodes are >> > > different on another site - it can be a pre-configured cluster without >> > > gpfs filesystems or with another filesystem. >> > > Same names and addresses shouldn't be a problem. >> > > >> > > Additionally to the replicated LUNs/NSDs you need to deliver copy >> > > of /var/mmfs/gen/mmsdrfs file from A to B site. >> > > There is no need to replicate it in real-time, only after the change >> of >> > > the cluster configuration. >> > > >> > > To activate site B - present replicated LUNs to the nodes in the DR >> > > cluster and run mmimportfs as "mmimportfs fs_name -i >> copy_of_mmsdrfs" >> > > >> > > Tested with multiples LUNs and filesystems on various workloads - >> seems >> > > to be working >> > > >> > > --Alex >> > > >> > > >> > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales < >> hmorales at optimizeit.co> >> > > wrote: >> > > >> > >> Thanks for answering. >> > >> >> > >> Essentially, the idea being explored is to replicate LUNs between >> > >> identical storage hardware (HP 3PAR volumesrein) on both sites. >> There is an >> > >> IP connection between storage boxes but not between servers on both >> sites, >> > >> there is a dark fiber connecting both sites. Here they dont want to >> explore >> > >> the idea of a scaled-based. >> > >> >> > >> >> > >> _______________________________________________ >> > >> gpfsug-discuss mailing list >> > >> gpfsug-discuss at spectrumscale.org >> > >> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> >> > >> >> > > >> > > _______________________________________________ >> > > gpfsug-discuss mailing list >> > > gpfsug-discuss at spectrumscale.org >> > > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > > >> > > >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: > > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_1503995b_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=SKdMmQae8uzHNWZq3vuRTp5UVwYFeeusLAxtbaposX0&e=> >> > >> > ------------------------------ >> > >> > Message: 2 >> > Date: Fri, 26 Jan 2018 14:21:15 -0600 >> > From: "Glen Corneau" >> > To: gpfsug main discussion list >> > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum >> > Scale >> > Message-ID: >> > > > 006FCEEE at notes.na.collabserv.com> >> > >> > Content-Type: text/plain; charset="us-ascii" >> > >> > Scale will walk across all discovered disks upon start time and attempt >> to >> > read the NSD identifiers from the disks. Once it finds them, it makes >> a >> > local map file that correlates the NSD id and the hdiskX identifier. >> The >> > names do not have to be the same as either the source cluster or even >> from >> > node-to-node. >> > >> > The main thing to keep in mind is to keep the file system definitions >> in >> > sync between the source and destination clusters. The "syncFSconfig" >> user >> > exit is the best way to do it because it's automatic. You generally >> > shouldn't be shuffling the mmsdrfs file between sites, that's what the >> > "mmfsctl syncFSconfig" does for you, on a per-file system basis. >> > >> > GPFS+AIX customers have been using this kind of storage replication for >> > over 10 years, it's business as usual. >> > >> > ------------------ >> > Glen Corneau >> > Power Systems >> > Washington Systems Center >> > gcorneau at us.ibm.com >> > >> > >> > >> > >> > >> > From: Harold Morales >> > To: gpfsug main discussion list >> > Date: 01/26/2018 12:30 PM >> > Subject: Re: [gpfsug-discuss] storage-based replication for >> > Spectrum Scale >> > Sent by: gpfsug-discuss-bounces at spectrumscale.org >> > >> > >> > >> > Hi Alex, This set up seems close to what I am trying to achieve. >> > >> > With regards to this kind of replication: any prereqs need to be met in >> > the target environment for this to work? for example, should disk >> devices >> > naming on AIX be the same as in the source environment? when importing >> the >> > mmsdrfs file, how is Scale going to know which disks should it assign >> to >> > the cluster? by their hdisk name alone? >> > >> > Thanks again, >> > >> > >> > >> > 2018-01-24 2:30 GMT-05:00 Alex Levin : >> > Hi, >> > >> > We are using a similar type of replication. >> > I assume the site B is the cold site prepared for DR >> > >> > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. >> > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX >> replication >> > group to ensure consistency. >> > >> > The cluster name, IP addresses , hostnames of the cluster nodes are >> > different on another site - it can be a pre-configured cluster without >> > gpfs filesystems or with another filesystem. >> > Same names and addresses shouldn't be a problem. >> > >> > Additionally to the replicated LUNs/NSDs you need to deliver copy >> > of /var/mmfs/gen/mmsdrfs file from A to B site. >> > There is no need to replicate it in real-time, only after the change of >> > the cluster configuration. >> > >> > To activate site B - present replicated LUNs to the nodes in the DR >> > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" >> > >> > Tested with multiples LUNs and filesystems on various workloads - >> seems >> > to be working >> > >> > --Alex >> > >> > >> > >> > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales >> >> > wrote: >> > Thanks for answering. >> > >> > Essentially, the idea being explored is to replicate LUNs between >> > identical storage hardware (HP 3PAR volumesrein) on both sites. There >> is >> > an IP connection between storage boxes but not between servers on both >> > sites, there is a dark fiber connecting both sites. Here they dont want >> to >> > explore the idea of a scaled-based. >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=d- >> > vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m=VbfWaftYSVjx8fMb >> 2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- >> > GeVJiVS49opBwwCFs-FKQrQ7rc&e= >> > >> > >> > >> > >> > >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: > > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_e291af63_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=bYnf-7v0CxYUkGth-QaVeUQdIlG8f1Gro-hwOxok7Qw&e=> >> > -------------- next part -------------- >> > A non-text attachment was scrubbed... >> > Name: not available >> > Type: image/jpeg >> > Size: 26117 bytes >> > Desc: not available >> > URL: > > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_e291af63_attachment.jpe&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=jYdnqhQBlnpf58oxunzBcTs9XdcbeOtLDQdgnASidDA&e=> >> > >> > ------------------------------ >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > >> > End of gpfsug-discuss Digest, Vol 72, Issue 69 >> > ********************************************** >> > >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egonle at aim.com Sat Jan 27 15:18:24 2018 From: egonle at aim.com (Eg. Bo.) Date: Sat, 27 Jan 2018 10:18:24 -0500 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Message-ID: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> Hello, while operating a GPFS filesystem for HPC some workload of few clients is able to eat available storage and NSD performance. Compute Nodes and NSD servers are connected to Infiniband.This question somehow is weird but is there a way to limit performance / I/O / bandwidth consumption for nodes so that other node still get "good" performance? Thanks, Eg. Bo.egonle at aim.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Sun Jan 28 15:53:47 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sun, 28 Jan 2018 12:53:47 -0300 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth In-Reply-To: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> References: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> Message-ID: Depends... While the current version of mmchqos is primarily geared towards controlling "maintenance" tasks. It can be used to limit (cap) the IOP rate for IOs coming from a particular GFPFS client node. You would need a version that supports the -N option on the mmchqos command and then configure each node that will mount the FS. Caution: try some experiments on a test system -- incorrect configuration can severely impact performance to the point where it can appear that things are totally "stuck"! mmchqos FS --disable --reset should get you back to the default config for gpfs/QOS. From: "Eg. Bo." To: gpfsug-discuss at spectrumscale.org Date: 01/27/2018 12:23 PM Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, while operating a GPFS filesystem for HPC some workload of few clients is able to eat available storage and NSD performance. Compute Nodes and NSD servers are connected to Infiniband. This question somehow is weird but is there a way to limit performance / I/O / bandwidth consumption for nodes so that other node still get "good" performance? Thanks, Eg. Bo. egonle at aim.com _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=r0AtLTBmDo_uZhIdntBO8k2-1Alg_2LBLPoJamV-zsY&s=BXGAuET7K4UhA24P72F4YRsAdcdJR3H02gqGOERENlE&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From vpuvvada at in.ibm.com Mon Jan 29 07:04:56 2018 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Mon, 29 Jan 2018 12:34:56 +0530 Subject: [gpfsug-discuss] AFM and RHEL 7.4 In-Reply-To: References: Message-ID: This is testing effort, AFM with RHEL 7.4 will be officially supported in next PTF releases 5.0.0.1 and 4.2.3.7. ~Venkat (vpuvvada at in.ibm.com) From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 01/25/2018 01:57 PM Subject: [gpfsug-discuss] AFM and RHEL 7.4 Sent by: gpfsug-discuss-bounces at spectrumscale.org The FAQ has a note stating: 1. AFM, Asynch Disaster Recovery with AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4. Could someone please clarify this sentence ? It can't be right that none of these features are supported with RHEL 7.4, or .. ? -jf_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=WsrKUTbd_huFePIa8dbAt4aezWpKHV902b-BlZTwGiA&s=OMRkxSzqA8U8J8mo3ap8I5TKscWunNEA2KZV3Z5_6HA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From richardb+gpfsUG at ellexus.com Mon Jan 29 10:27:37 2018 From: richardb+gpfsUG at ellexus.com (Richard Booth) Date: Mon, 29 Jan 2018 10:27:37 +0000 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Message-ID: Hi Without wishing to blow our own trumpet, Ellexus tools do this with GPFS, There is a youtube video that shows our tool Mistral managing a fair usage policy, link below, https://youtu.be/LJFcBUkcn3c Richard ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 27 Jan 2018 10:18:24 -0500 > From: "Eg. Bo." > To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth > Message-ID: <1613832bc3c-1721-1396f9 at webjas-vaa235.srv.aolmail.net> > Content-Type: text/plain; charset="utf-8" > > > Hello, > > > while operating a GPFS filesystem for HPC some workload of few clients is > able to eat available storage and NSD performance. Compute Nodes and NSD > servers are connected to Infiniband.This question somehow is weird but is > there a way to limit performance / I/O / bandwidth consumption for nodes so > that other node still get "good" performance? > > > > > Thanks, > > > Eg. Bo.egonle at aim.com > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: 20180127/dcc7cd79/attachment-0001.html> > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 72, Issue 74 > ********************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Mon Jan 29 09:54:46 2018 From: john.hearns at asml.com (John Hearns) Date: Mon, 29 Jan 2018 09:54:46 +0000 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth In-Reply-To: References: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> Message-ID: Hello Egonle. I looked at this scenario recently (H Mark!) Maybe you could send me an email off-list? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Marc A Kaplan Sent: Sunday, January 28, 2018 4:54 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] limit / fair share of GPFS bandwidth Depends... While the current version of mmchqos is primarily geared towards controlling "maintenance" tasks. It can be used to limit (cap) the IOP rate for IOs coming from a particular GFPFS client node. You would need a version that supports the -N option on the mmchqos command and then configure each node that will mount the FS. Caution: try some experiments on a test system -- incorrect configuration can severely impact performance to the point where it can appear that things are totally "stuck"! mmchqos FS --disable --reset should get you back to the default config for gpfs/QOS. From: "Eg. Bo." > To: gpfsug-discuss at spectrumscale.org Date: 01/27/2018 12:23 PM Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hello, while operating a GPFS filesystem for HPC some workload of few clients is able to eat available storage and NSD performance. Compute Nodes and NSD servers are connected to Infiniband. This question somehow is weird but is there a way to limit performance / I/O / bandwidth consumption for nodes so that other node still get "good" performance? Thanks, Eg. Bo. egonle at aim.com_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=r0AtLTBmDo_uZhIdntBO8k2-1Alg_2LBLPoJamV-zsY&s=BXGAuET7K4UhA24P72F4YRsAdcdJR3H02gqGOERENlE&e= -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkr at lbl.gov Wed Jan 31 21:20:04 2018 From: kkr at lbl.gov (Kristy Kallback-Rose) Date: Wed, 31 Jan 2018 13:20:04 -0800 Subject: [gpfsug-discuss] US Spectrum Scale/GPFS User Group - Save the Date ~May 16 - ~May 17 - Boston Message-ID: <46BCDF1D-72C8-4FD1-8375-DD4207732F78@lbl.gov> Please mark your calendars, we?re planning for our next US Spectrum Scale/GPFS User Group meeting. We should be able to provide more exact details in a week or so, but loosely the event will be two days in Boston around May 16 - May 17, 2018. Send agenda suggestions now. Feel free to reply to the list to generate discussion. Best, Kristy From chris.schlipalius at pawsey.org.au Wed Jan 31 23:21:02 2018 From: chris.schlipalius at pawsey.org.au (Chris Schlipalius) Date: Thu, 01 Feb 2018 07:21:02 +0800 Subject: [gpfsug-discuss] 2018 March 26th Singapore Spectrum Scale User Group event announced - Final call for user stories - submission deadline 9th Feb Message-ID: Hello, First, let me apologise to all who have responded. This is a follow up call out for user use cases or presentations for those wishing to present at this Usergroup ? namely client use cases or client stories, from those who haven?t yet contacted me to organise to do so. At the inaugural Spectrum Scale Usergroup Singapore on the Monday 26th March 2018, Sentosa, Singapore. This is being held in conjunction with SCA18 https://sc-asia.org/ All current Spectrum Scale User Group event details can be found here: http://goo.gl/dXtqvS Feel free to circulate this event link to all that may need it. Please ensure a speaker?s ticket is reserved, and please email me the title and duration of the talk and speakers name details so I can add this to the agenda on Eventbrite and promote on the discussion list and spectrumscale.org website. Accommodation and conference registration - both are open ? please see https://www.sc-asia.org/accommodation/ for the accommodation booking form, and https://www.sc-asia.org/register/ for the early bird conference registration (early bird rate closes soon 9th Feb). Please reserve a ticket in the Eventbrite link above ASAP and if you wish, email me separately if you have accommodation sorted, or need Sentosa accommodation (in addition to completing the form linked above). As I am in contact with the organisers of the event and I can also ensure there are places for presenters at this group and attendees. The user group location is still located at Resorts World Convention We are looking forwards to a great new Usergroup soon in a fabulous venue. Thanks again to NSCC and IBM for helping to arrange the venue and event booking. Regards, Chris Schlipalius Team Lead, Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO) 12 Burvill Court Kensington WA 6151 Australia From mail at arif-ali.co.uk Tue Jan 2 10:29:11 2018 From: mail at arif-ali.co.uk (Arif Ali) Date: Tue, 2 Jan 2018 10:29:11 +0000 Subject: [gpfsug-discuss] Testing to see the mailing list is working -- please ignore Message-ID: <42f7a66e-ff42-ef26-8ece-888fdc932f8e@arif-ali.co.uk> Just a quick test message, to ensure mailing list is working, as we've not heard since 22nd December, So please ignore -- regards, Arif Ali -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From r.sobey at imperial.ac.uk Wed Jan 3 10:37:54 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 3 Jan 2018 10:37:54 +0000 Subject: [gpfsug-discuss] mmchfileset syntax Message-ID: Hi all, Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. mmchfileset gpfs filesetname -J /gpfs/new_path mmchfileset: Options -J and FilesetName cannot be specified at the same time. Which leads to what?s the point of the following command as the error suggests it will work: mmchfileset device -J /gpfs/new_path ?as not specifying a FilesetName to the input of mmchfileset seems odd. I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jan 3 11:49:42 2018 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 3 Jan 2018 11:49:42 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: Message-ID: -J is to define which fileset you are talking about. The alternative is give the fileset name straight after the ?device? argument. Once the fileset is unambiguous, you then use other command line option to change things e.g. -j to change the *name* of this fileset. -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. Daniel Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com > On 3 Jan 2018, at 10:38, Sobey, Richard A wrote: > > Hi all, > > Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. > > mmchfileset gpfs filesetname -J /gpfs/new_path > mmchfileset: Options -J and FilesetName cannot be specified at the same time. > > Which leads to what?s the point of the following command as the error suggests it will work: > > mmchfileset device -J /gpfs/new_path > > ?as not specifying a FilesetName to the input of mmchfileset seems odd. > > I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? > > Cheers > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Wed Jan 3 12:37:46 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 3 Jan 2018 12:37:46 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: References: Message-ID: Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and mmchfileset gets the filesetname from that). Ultimately though I cannot change the mount point of the fileset without mmunlink / mmlink, right? Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger Sent: 03 January 2018 11:50 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmchfileset syntax -J is to define which fileset you are talking about. The alternative is give the fileset name straight after the ?device? argument. Once the fileset is unambiguous, you then use other command line option to change things e.g. -j to change the *name* of this fileset. -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. Daniel [/spectrum_storage-banne] [Spectrum Scale Logo] Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com On 3 Jan 2018, at 10:38, Sobey, Richard A > wrote: Hi all, Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. mmchfileset gpfs filesetname -J /gpfs/new_path mmchfileset: Options -J and FilesetName cannot be specified at the same time. Which leads to what?s the point of the following command as the error suggests it will work: mmchfileset device -J /gpfs/new_path ?as not specifying a FilesetName to the input of mmchfileset seems odd. I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? Cheers Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From tconnelly at pixitmedia.com Wed Jan 3 13:27:13 2018 From: tconnelly at pixitmedia.com (Tim Connelly) Date: Wed, 3 Jan 2018 13:27:13 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: References: Message-ID: >From the mmlinkfileset docs; "The user may use the mv command on the directory to move to a new location in the parent fileset, but the mv command is not allowed to move the junction to a different fileset." # mmlsfileset mmfs1 |grep timt sas1-timtest Linked /mmfs1/another_test/timtest # mv /mmfs1/another_test/timtest /mmfs1/another_test/timtest2 # mmlsfileset mmfs1 |grep timt sas1-timtest Linked /mmfs1/another_test/timtest2 Hope this helps! Tim On 3 January 2018 at 12:37, Sobey, Richard A wrote: > Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and > mmchfileset gets the filesetname from that). > > > > Ultimately though I cannot change the mount point of the fileset without > mmunlink / mmlink, right? > > > > Thanks > > Richard > > > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: > gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Daniel Kidger > *Sent:* 03 January 2018 11:50 > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] mmchfileset syntax > > > > -J is to define which fileset you are talking about. > > The alternative is give the fileset name straight after the ?device? > argument. > > > > Once the fileset is unambiguous, you then use other command line option to > change things > > e.g. -j to change the *name* of this fileset. > > -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the > mount point either. > > > > Daniel > > [image: /spectrum_storage-banne] > > > > > [image: Spectrum Scale Logo] > > > > > > *Dr Daniel Kidger* > IBM Technical Sales Specialist > Software Defined Solution Sales > > + <+%2044-7818%20522%20266> 44-(0)7818 522 266 <+%2044-7818%20522%20266> > daniel.kidger at uk.ibm.com > > > On 3 Jan 2018, at 10:38, Sobey, Richard A wrote: > > Hi all, > > > > Attempting to change the junction path of a fileset and I?m getting some > strange output. It may just not be possible but the documentation on > mmchfileset itself doesn?t make it clear. > > > > mmchfileset gpfs filesetname -J /gpfs/new_path > > mmchfileset: Options -J and FilesetName cannot be specified at the same > time. > > > > Which leads to what?s the point of the following command as the error > suggests it will work: > > > > mmchfileset device -J /gpfs/new_path > > > > ?as not specifying a FilesetName to the input of mmchfileset seems odd. > > > > I do of course know that I can just unlink and relink but hey.. trying to > save a few precious seconds ? > > > > Cheers > > Richard > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jan 3 13:13:03 2018 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 3 Jan 2018 13:13:03 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: Message-ID: Richard, The documentation says that that you can simply use ?mv? on a live fileset too. I have just tried this and yes it appears to work. I can move a fileset into another subdirectory of the same filesystem and mmlsfileset confirms the new mount point. Daniel Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com > On 3 Jan 2018, at 12:37, Sobey, Richard A wrote: > > Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and mmchfileset gets the filesetname from that). > > Ultimately though I cannot change the mount point of the fileset without mmunlink / mmlink, right? > > Thanks > Richard > > > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger > Sent: 03 January 2018 11:50 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] mmchfileset syntax > > -J is to define which fileset you are talking about. > The alternative is give the fileset name straight after the ?device? argument. > > Once the fileset is unambiguous, you then use other command line option to change things > e.g. -j to change the *name* of this fileset. > -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. > > > Daniel > > > > > > > Dr Daniel Kidger > IBM Technical Sales Specialist > Software Defined Solution Sales > > + 44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > On 3 Jan 2018, at 10:38, Sobey, Richard A wrote: > > Hi all, > > Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. > > mmchfileset gpfs filesetname -J /gpfs/new_path > mmchfileset: Options -J and FilesetName cannot be specified at the same time. > > Which leads to what?s the point of the following command as the error suggests it will work: > > mmchfileset device -J /gpfs/new_path > > ?as not specifying a FilesetName to the input of mmchfileset seems odd. > > I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? > > Cheers > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Wed Jan 3 13:58:20 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 3 Jan 2018 13:58:20 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: References: Message-ID: Thank you Daniel and Tim, most helpful. Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger Sent: 03 January 2018 13:13 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmchfileset syntax Richard, The documentation says that that you can simply use ?mv? on a live fileset too. I have just tried this and yes it appears to work. I can move a fileset into another subdirectory of the same filesystem and mmlsfileset confirms the new mount point. Daniel [Image removed by sender. /spectrum_storage-banne] [Image removed by sender. Spectrum Scale Logo] Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com On 3 Jan 2018, at 12:37, Sobey, Richard A > wrote: Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and mmchfileset gets the filesetname from that). Ultimately though I cannot change the mount point of the fileset without mmunlink / mmlink, right? Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger Sent: 03 January 2018 11:50 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] mmchfileset syntax -J is to define which fileset you are talking about. The alternative is give the fileset name straight after the ?device? argument. Once the fileset is unambiguous, you then use other command line option to change things e.g. -j to change the *name* of this fileset. -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. Daniel [Image removed by sender. /spectrum_storage-banne] [Image removed by sender. Spectrum Scale Logo] Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com On 3 Jan 2018, at 10:38, Sobey, Richard A > wrote: Hi all, Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. mmchfileset gpfs filesetname -J /gpfs/new_path mmchfileset: Options -J and FilesetName cannot be specified at the same time. Which leads to what?s the point of the following command as the error suggests it will work: mmchfileset device -J /gpfs/new_path ?as not specifying a FilesetName to the input of mmchfileset seems odd. I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? Cheers Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ~WRD029.jpg Type: image/jpeg Size: 823 bytes Desc: ~WRD029.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2519 bytes Desc: image001.jpg URL: From hoov at us.ibm.com Wed Jan 3 17:10:51 2018 From: hoov at us.ibm.com (Theodore Hoover Jr) Date: Wed, 3 Jan 2018 17:10:51 +0000 Subject: [gpfsug-discuss] Fw: Spectrum Scale on AWS - Join Sponsor User Program Message-ID: An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Thu Jan 4 11:16:57 2018 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 4 Jan 2018 11:16:57 +0000 Subject: [gpfsug-discuss] Use AFM for migration of many small files In-Reply-To: References: <467FB293-D33B-46F4-BA1B-A5CB4D28DDE6@psi.ch> Message-ID: <1515064616.28898.32.camel@qmul.ac.uk> We are doing something very similar using 4.2.3-4 and GPFS 4.2.1-1 on the nfs backend. Did you have any success? The plan is to load the new cache from the old GPFS then dump once the cache is full. We've already increase numThreashThreads from 4 to 8 and seen only marginal improvements, we could attempt to increase this further. I'm also wondering if its worth increasing the Refresh Intervals to speed up read of already cache files. At this stage we want to fill the cache and don't care about write back until we switch the target to the new NFS/GPFS from our old GPFS storage to a new box back at our off-site location, (otherwise known as the office) [root at afmgateway1 scratch]# mmlsfileset home home -L --afm Filesets in file system 'home': Attributes for fileset home: ============================= Status Linked Path /data2/home Id 42 Root inode 1343225859 Parent Id 0 Created Wed Jan 3 12:32:33 2018 Comment Inode space 41 Maximum number of inodes 100000000 Allocated inodes 15468544 Permission change flag chmodAndSetacl afm-associated Yes Target nfs://afm1/gpfs/data1/afm/home Mode single-writer File Lookup Refresh Interval 30 (default) File Open Refresh Interval 30 (default) Dir Lookup Refresh Interval 60 (default) Dir Open Refresh Interval 60 (default) Async Delay 15 (default) Last pSnapId 0 Display Home Snapshots no Number of Gateway Flush Threads 8 Prefetch Threshold 0 (default) Eviction Enabled no Thanks in advance. Peter Childs On Tue, 2017-09-05 at 19:57 +0530, Venkateswara R Puvvada wrote: Which version of Spectrum Scale ? What is the fileset mode ? >We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here >I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. How was the performance measured ? If parallel IO is enabled, AFM uses multiple gateway nodes to prefetch the large files (if file size if more than 1GB). Performance difference between small and lager file is huge (1000MB - 150MB = 850MB) here, and generally it is not the case. How many files were present in list file for prefetch ? Could you also share full internaldump from the gateway node ? >I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few >read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. AFM prefetches the files on multiple threads. Default flush threads for prefetch are 36 (fileset.afmNumFlushThreads (default 4) + afmNumIOFlushThreads (default 32)). >Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and >I need to look elsewhere to get better performance for prefetch of many smaller files? See above, AFM reads files on multiple threads parallelly. Try increasing the afmNumFlushThreads on fileset and verify if it improves the performance. ~Venkat (vpuvvada at in.ibm.com) From: "Billich Heinrich Rainer (PSI)" To: "gpfsug-discuss at spectrumscale.org" Date: 09/04/2017 10:18 PM Subject: [gpfsug-discuss] Use AFM for migration of many small files Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and I need to look elsewhere to get better performance for prefetch of many smaller files? We will migrate several filesets in parallel, but still with individual filesets up to 350TB in size 150MB/s isn?t fun. Also just about 150 files/s seconds looks poor. The setup is quite new, hence there may be other places to look at. It?s all RHEL7 an spectrum scale 4.2.2-3 on the afm cache. Thank you, Heiner --, Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://urldefense.proofpoint.com/v2/url?u=https-3A__www.psi.ch&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=eHcVdovN10-m-Qk0Ln2qvol3pkKNFwrzz2wgf1zXVXE&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=LbRyuSM_djs0FDXr27hPottQHAn3OGcivpyRcIDBN3U&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Peter Childs ITS Research Storage Queen Mary, University of London -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Thu Jan 4 17:57:14 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Thu, 4 Jan 2018 17:57:14 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Message-ID: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bipcuds at gmail.com Thu Jan 4 20:34:30 2018 From: bipcuds at gmail.com (Keith Ball) Date: Thu, 4 Jan 2018 15:34:30 -0500 Subject: [gpfsug-discuss] Conflicting RHEL compatibility information in the Spectrum Scale FAQ Message-ID: Hi All, Here is a repeat post of my question on GSS 2.0.7 compatibility with RHEL 6.7. FAQ is contradictory. I know GSS/ESS is usually quite sensitive to use a specific RHEL minor version). And at this point, RHEL 6.5 isn't really supported anymore (no updates provided). On Tue, Dec 19, 2017 at 6:08 PM, Keith Ball wrote: I was recently trying to determine the latest RHEL release that will work > with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code > uses Scale 4.1.0.8. > > A specific X.Y GSS code build, from my experience, is intended to use a > specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), > which no longer has EUS support from RedHat (only 6.7 still does). > > GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be > used, which is a surprising statement given GSS/ESS code's sensitivity to > OS levels (any ESS I have ever seen has never been supported on more than > one version of RHEL). > > According to the Scale FAQ (https://www.ibm.com/support/ > knowledgecenter/STXKQY/gpfsclustersfaq.html#linux), A 2.2, Table 27, > Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and > 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 > is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 > release" ... which contradicts Table 27! > > Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install > GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is > supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option > here). > > Many Thanks, > Keith > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skylar2 at u.washington.edu Fri Jan 5 00:52:31 2018 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Thu, 4 Jan 2018 16:52:31 -0800 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: <85c8b317-a7cb-ce84-81bd-0a7e8b75bc7e@u.washington.edu> While I'm not fully versed on the vulnerability or the proposed fixes, my understanding is that most of the performance impact from the fix is around having kernel memory completely separately from a process's user-space, which means every system call will have cache/TLB misses. This might mean clusters which are using RDMA won't have as big of a performance hit versus ones that aren't able to use RDMA, since they can do a lot more in user-space without involving the kernel. On 01/04/2018 09:57 AM, Buterbaugh, Kevin L wrote: > Happy New Year everyone, > > I?m sure that everyone is aware of Meltdown and Spectre by now ? we, > like many other institutions, will be patching for it at the earliest > possible opportunity. > > Our understanding is that the most serious of the negative performance > impacts of these patches will be for things like I/O (disk / network) > ? given that, we are curious if IBM has any plans for a GPFS update > that could help mitigate those impacts? ?Or is there simply nothing > that can be done? > > If there is a GPFS update planned for this we?d be interested in > knowing so that we could coordinate the kernel and GPFS upgrades on > our cluster. > > Thanks? > > Kevin > > P.S. ?The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a > good year for everyone despite how it?s starting out. ?:-O > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu > ?- (615)875-9633 > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Thu Jan 4 22:55:18 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Thu, 4 Jan 2018 17:55:18 -0500 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=m7Pdt9KL82CJT_AT-PwkmO3PbHg88-IQ7Jq-dwhDOdY&s=5i66Rx3vse5LcaN4-WlyCwi_TDTOQGQR2-X_XyjbBpw&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From richardb+gpfsUG at ellexus.com Fri Jan 5 08:09:38 2018 From: richardb+gpfsUG at ellexus.com (Richard Booth) Date: Fri, 5 Jan 2018 08:09:38 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Message-ID: Dear Kevin The company I work for might be able to help get back the performance lost from the fixes for Meltdown and Spectre. I work for Ellexus, an I/O profiling company. Our tools could help identify any inefficient I/O patterns in your applications to reduce the impact of this. We can set up demos and a free trial if you're interested in seeing what we do and how we can help. Kind regards Richard Message: 1 Date: Thu, 4 Jan 2018 17:57:14 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Message-ID: <5D655862-7F60-47F6-8BD2-A5298F73F70F at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -- Richard Booth Account Manager, Ellexus Ltd richard.booth at ellexus.com +44 (0)1223 42 16 46 Ellexus is the I/O profiling company. www.ellexus.com *Ellexus Ltd is a limited company registered in England & Wales* *Company registration no. 07166034* *Registered address: 198 High Street, Tonbridge, Kent TN9 1BE, UK* *Operating address: St John's Innovation Centre, Cowley Road, Cambridge CB4 0WS, UK* -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Fri Jan 5 12:39:11 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 5 Jan 2018 18:09:11 +0530 Subject: [gpfsug-discuss] Password to GUI forgotten In-Reply-To: <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu> References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu><2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com><26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu><348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu><1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov> <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu> Message-ID: Hi Kevin, If you are stuck then please open a PMR and work with the IBM support folks to get this resolved. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Buterbaugh, Kevin L" To: "Hanley, Jesse A." Cc: gpfsug main discussion list Date: 12/19/2017 01:42 AM Subject: Re: [gpfsug-discuss] Password to GUI forgotten Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Jesse, Thanks for the suggestion ? I find the following error very interesting: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/rmuser admin EFSSP0010C CLI parser: The object "admin" specified for "userID" does not exist. /root root at testnsd1# That says to me that I don?t have an admin user, which - if true - would explain why not a single password I can think of works. ;-) But as I mentioned in my original post I had this up and working earlier this fall. While I can?t prove anything, I can?t imagine a scenario where I would deliberately choose a non-default username. So if ?admin? has been the default login for the GPFS GUI all along then I am really mystified. Thanks! Kevin On Dec 18, 2017, at 1:58 PM, Hanley, Jesse A. wrote: Kevin, I ran into this a couple times using 4.2.3. This is what we used to get around it: /usr/lpp/mmfs/gui/cli/rmuser admin /usr/lpp/mmfs/gui/cli/mkuser admin -p -g Administrator,SecurityAdmin You may need to run the initgui command if those objects are present. That typically gets run on first login to the GUI. Thanks, -- Jesse From: on behalf of "Buterbaugh, Kevin L" Reply-To: gpfsug main discussion list Date: Monday, December 18, 2017 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten Hi All, Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up. Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this. Richards? suggestion below errors for me on the ?-g Administrator? part. Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked. No matter what I do, I cannot log in to the GUI. Thanks for the suggestions, though? Kevin On Dec 7, 2017, at 6:10 AM, Sobey, Richard A wrote: Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error. That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself. [root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin EFSSG0019I The user admin2 has been successfully created. EFSSG1000I The command completed successfully. Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A Sent: 07 December 2017 11:57 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out. Get Outlook for Android From: gpfsug-discuss-bounces at spectrumscale.org < gpfsug-discuss-bounces at spectrumscale.org> on behalf of Buterbaugh, Kevin L Sent: Wednesday, December 6, 2017 10:41:12 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist. /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist. /root root at testnsd1# I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case. Thanks! Kevin On Dec 6, 2017, at 4:32 PM, Joshua Kwedar wrote: Hmm.. odd. Here?s what the lsuser output should look like. # /usr/lpp/mmfs/gui/cli/lsuser Name Long name Password status Group names Failed login attempts admin active Administrator,SecurityAdmin 0 EFSSG1000I The command completed successfully. Can you try something like? # /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin From: on behalf of "Buterbaugh, Kevin L" Reply-To: gpfsug main discussion list Date: Wednesday, December 6, 2017 at 5:15 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, Sorry - should?ve mentioned that: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 EFSSG0001C Cannot validate option: login /root root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts: /root root at testnsd1# Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November. Thanks... Kevin On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) wrote: The GUI password can be changed via command line using chuser. /usr/lpp/mmfs/gui/cli/chuser Usage is as follows (where userID = admin) chuser userID {-p | -l | -a | -d | -g | --expirePassword} [-o ] Josh K On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L < Kevin.Buterbaugh at Vanderbilt.Edu> wrote: Hi All, So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall. However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI. :-( Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch? If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route. Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-) Thanks? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=B07CFG_8Wsf_UKWDoD32OlUNVRy3yFTESgTgZPki4Wg&s=A5cFcjLgIwulsbnh_cPIBawmU32lkOd4dqyl9_yzELA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jan 5 14:44:04 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 5 Jan 2018 14:44:04 +0000 Subject: [gpfsug-discuss] Password to GUI forgotten In-Reply-To: References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu> <2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com> <26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu> <348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu> <1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov> <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu> Message-ID: Hi GPFS team, I did open a PMR and they (mainly Matthais) did help me get that issue resolved. Thanks for following up! Kevin On Jan 5, 2018, at 6:39 AM, IBM Spectrum Scale > wrote: Hi Kevin, If you are stuck then please open a PMR and work with the IBM support folks to get this resolved. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Buterbaugh, Kevin L" > To: "Hanley, Jesse A." > Cc: gpfsug main discussion list > Date: 12/19/2017 01:42 AM Subject: Re: [gpfsug-discuss] Password to GUI forgotten Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi Jesse, Thanks for the suggestion ? I find the following error very interesting: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/rmuser admin EFSSP0010C CLI parser: The object "admin" specified for "userID" does not exist. /root root at testnsd1# That says to me that I don?t have an admin user, which - if true - would explain why not a single password I can think of works. ;-) But as I mentioned in my original post I had this up and working earlier this fall. While I can?t prove anything, I can?t imagine a scenario where I would deliberately choose a non-default username. So if ?admin? has been the default login for the GPFS GUI all along then I am really mystified. Thanks! Kevin On Dec 18, 2017, at 1:58 PM, Hanley, Jesse A. > wrote: Kevin, I ran into this a couple times using 4.2.3. This is what we used to get around it: /usr/lpp/mmfs/gui/cli/rmuser admin /usr/lpp/mmfs/gui/cli/mkuser admin -p -g Administrator,SecurityAdmin You may need to run the initgui command if those objects are present. That typically gets run on first login to the GUI. Thanks, -- Jesse From: > on behalf of "Buterbaugh, Kevin L" > Reply-To: gpfsug main discussion list > Date: Monday, December 18, 2017 at 2:52 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Password to GUI forgotten Hi All, Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up. Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this. Richards? suggestion below errors for me on the ?-g Administrator? part. Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked. No matter what I do, I cannot log in to the GUI. Thanks for the suggestions, though? Kevin On Dec 7, 2017, at 6:10 AM, Sobey, Richard A > wrote: Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error. That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself. [root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin EFSSG0019I The user admin2 has been successfully created. EFSSG1000I The command completed successfully. Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org[mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A Sent: 07 December 2017 11:57 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Password to GUI forgotten This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out. Get Outlook for Android ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org> on behalf of Buterbaugh, Kevin L > Sent: Wednesday, December 6, 2017 10:41:12 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist. /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist. /root root at testnsd1# I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case. Thanks! Kevin On Dec 6, 2017, at 4:32 PM, Joshua Kwedar > wrote: Hmm.. odd. Here?s what the lsuser output should look like. # /usr/lpp/mmfs/gui/cli/lsuser Name Long name Password status Group names Failed login attempts admin active Administrator,SecurityAdmin 0 EFSSG1000I The command completed successfully. Can you try something like? # /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin From: > on behalf of "Buterbaugh, Kevin L" > Reply-To: gpfsug main discussion list > Date: Wednesday, December 6, 2017 at 5:15 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, Sorry - should?ve mentioned that: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 EFSSG0001C Cannot validate option: login /root root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts: /root root at testnsd1# Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November. Thanks... Kevin On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) > wrote: The GUI password can be changed via command line using chuser. /usr/lpp/mmfs/gui/cli/chuser Usage is as follows (where userID = admin) chuser userID {-p | -l | -a | -d | -g | --expirePassword} [-o ] Josh K On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L > wrote: Hi All, So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall. However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI. :-( Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch? If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route. Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-) Thanks? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=B07CFG_8Wsf_UKWDoD32OlUNVRy3yFTESgTgZPki4Wg&s=A5cFcjLgIwulsbnh_cPIBawmU32lkOd4dqyl9_yzELA&e= ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Jan 6 10:46:16 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Sat, 6 Jan 2018 16:16:16 +0530 Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ In-Reply-To: References: Message-ID: Hi Lyle, Can you please forward this to the right folks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Keith Ball To: gpfsug-discuss at spectrumscale.org Date: 12/20/2017 04:39 AM Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was recently trying to determine the latest RHEL release that will work with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code uses Scale 4.1.0.8. A specific X.Y GSS code build, from my experience, is intended to use a specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), which no longer has EUS support from RedHat (only 6.7 still does). GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be used, which is a surprising statement given GSS/ESS code's sensitivity to OS levels (any ESS I have ever seen has never been supported on more than one version of RHEL). According to the Scale FAQ ( https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux ), A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27! Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option here). Many Thanks, Keith_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=u7hLzIRwApuSZgw6uZ_j3uewYEyRTPxRmWJhpM_UAwY&s=bBgN8s6sKOU36_ivUi5NHGarxlF3TNLPyXS-7PA34F0&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Jan 6 11:17:34 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Sat, 6 Jan 2018 16:47:34 +0530 Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group without takeover In-Reply-To: References: Message-ID: Hi Veera, Can you please help in answering Damir's query. --------------------------- My question is, is there a way of brining back the IO server into the mix without the recoverygroup takeover happening? Could I just start a gpfs and have it back in the mix as a backup server for the recoverygroup and if so, how do you do that. Right now that server is designated as primary server for the recovery group. I would like to have both IO servers in the mix for redundancy purposes. --------------------------- Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Damir Krstic To: gpfsug main discussion list Date: 12/22/2017 11:15 PM Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group without takeover Sent by: gpfsug-discuss-bounces at spectrumscale.org It's been a very frustrating couple of months with our 2 ESS systems. IBM tells us we had blueflame bug and they came on site and updated our ESS to the latest version back in middle of November. Wednesday night one of the NSD servers in one of our ESS building blocks kernel panicked. No idea why and none of the logs are insightful. We have a PMR open with IBM. I am not very confident we will get to the bottom of what's causing kernel panics on our IO servers. The system has gone down over 4 times now in 2 months. When we tried brining it back up, it rejoined the recovery group and the IO on the entire cluster locked up until we were able to find couple of compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on those nodes resolved the issue of the filesystem locking up. So far we have never been successful in brining back an IO server and not having a filesystem lock up until we find a node with pending state with tscomm. Anyway, the system was stable for few minutes until the same IO server that went down on Wednesday night went into an arbitrating mode. It never recovered. We stopped gpfs on that server and IO recovered again. We left gpfs down and cluster seems to be OK. My question is, is there a way of brining back the IO server into the mix without the recoverygroup takeover happening? Could I just start a gpfs and have it back in the mix as a backup server for the recoverygroup and if so, how do you do that. Right now that server is designated as primary server for the recovery group. I would like to have both IO servers in the mix for redundancy purposes. This ESS situation is beyond frustrating and I don't see end in sight. Any help is appreciated._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=vGabwIUw-ziMAtM7VfTppRp3S16NsgGOk5qMe50gtIQ&s=eplQuGhWVZMQ3tBLeqhCKpZ0w0rIiU-2R-UuqHYSsVA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Jan 6 10:21:14 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Sat, 6 Jan 2018 15:51:14 +0530 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35 In-Reply-To: <44E99F55-25CC-48DB-9AD6-E7D6794694DC@nasa.gov> References: , <4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz> <44E99F55-25CC-48DB-9AD6-E7D6794694DC@nasa.gov> Message-ID: Hi Lyle, Can you forward this to the right person. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]" To: gpfsug main discussion list Date: 12/19/2017 02:01 PM Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35 Sent by: gpfsug-discuss-bounces at spectrumscale.org It?s not supported on SLES11 either. IBM didn?t (that I saw) talk much about this publicly or give customers a chance to provide feedback about the decision. I know it was raised at the UG in NY and I recall a number of people saying it would be a significant issue for them (myself included) as is the fact they no longer support Debian with scale 5.0. I?d raised the issue on the mailing list after the UG trying to start the discussion but IBM said they weren?t ready to talk about it publicly and I can only guess they had already set their sights and didn?t actually want feedback. This is actually pretty frustrating. I?m tempted to open an RFE but most of my RFEs either have been rejected or just sit idle so I?m not clear there?s a benefit. On December 19, 2017 at 03:08:27 EST, atmane khiredine wrote: IBM Spectrum Scale V5.0 not support RHEL 6.x Only RHEL 7.1 or later https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz ________________________________________ De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org] Envoy? : lundi 18 d?cembre 2017 22:46 ? : gpfsug-discuss at spectrumscale.org Objet : gpfsug-discuss Digest, Vol 71, Issue 35 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: FW: Spectrum Scale 5.0 now available on Fix Central (Michael L Taylor) 2. Re: gpfs 4.2.3.5 and RHEL 7.4... (Frederick Stock) 3. Re: gpfs 4.2.3.5 and RHEL 7.4... (Eric Horst) ---------------------------------------------------------------------- Message: 1 Date: Mon, 18 Dec 2017 13:27:42 -0700 From: "Michael L Taylor" To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central Message-ID: Content-Type: text/plain; charset="us-ascii" Hi Bob, Thanks for the note on 5.0.0 One correction however.... clusters can do a rolling upgrade to 5.0.0 from any 4.2.x level (not just 4.2.3). Today's Topics: 1. FW: Spectrum Scale 5.0 now available on Fix Central (Oesterlin, Robert) ---------------------------------------------------------------------- Message: 1 Date: Mon, 18 Dec 2017 19:43:35 +0000 From: "Oesterlin, Robert" To: gpfsug main discussion list Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central Message-ID: Content-Type: text/plain; charset="utf-8" The Scale 5.0 fix level is now up on Fix Central. You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: "dW-notify at us.ibm.com" Reply-To: "dW-notify at us.ibm.com" Date: Monday, December 18, 2017 at 1:27 PM Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum. [mid:4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75 at SDEB-EXC02.meteo.dz/forums.png] gpfs at us.ibm.com< https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e= > replied to the IBM Spectrum Scale V5.0 announcements< https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e= > topic thread in the General Parallel File System - Announce (GPFS - Announce)< https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0 000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=> forum. IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central: http://www-933.ibm.com/support/fixcentral< https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e= > This topic summarizes changes to the IBM Spectrum Scale licensed program and the IBM Spectrum Scale library. Summary of changes for IBM Spectrum Scale version 5 release 0.0 -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/157250d2/attachment-0001.html > ------------------------------ Message: 2 Date: Mon, 18 Dec 2017 16:10:55 -0500 From: "Frederick Stock" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... Message-ID: Content-Type: text/plain; charset="us-ascii" Yes the integrated protocols are the Samba and Ganesha that are bundled with Spectrum Scale. These require the use of the CES component for monitoring the protocols. If you do use them then you need to wait for a release of Spectrum Scale in which the integrated protocols are also supported on RHEL 7.4. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug-discuss at spectrumscale.org Date: 12/18/2017 03:09 PM Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... Sent by: gpfsug-discuss-bounces at spectrumscale.org Currently, the IBM support matrix says: https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux that 4.2.3.5 is supported on RHEL 7.4, but with a footnote: "AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4." We don't use AFM or the install toolkit. But we *do* make fairly heavy use of mmces and nfs-ganesha - is that what they mean by "Integrated Protocols"? (We're looking at doing upgrades next month while our HPC clusters are doing their upgrades - and going to 7.4 would be nice. If there's a mine field there, I need to make sure we stay at 7.3 - plus applicable non-7.4 updates) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/7c272339/attachment-0001.html > ------------------------------ Message: 3 Date: Mon, 18 Dec 2017 21:46:02 +0000 From: Eric Horst To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... Message-ID: Content-Type: text/plain; charset="utf-8" Grr, this might explain why I experienced unhappiness when I tried to start my long-delayed AFM based migration over the weekend. I had previously tested AFM and found everything working, but 7.4 may have slipped in last month. The AFM relationship seems to work but `mmafmctl premigrate` commands fail. I would revert packages if I could figure out where the issue lies. -Eric On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock wrote: > Yes the integrated protocols are the Samba and Ganesha that are bundled > with Spectrum Scale. These require the use of the CES component for > monitoring the protocols. If you do use them then you need to wait for a > release of Spectrum Scale in which the integrated protocols are also > supported on RHEL 7.4. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821> > stockf at us.ibm.com > > > > From: valdis.kletnieks at vt.edu > To: gpfsug-discuss at spectrumscale.org > Date: 12/18/2017 03:09 PM > Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Currently, the IBM support matrix says: > > https://www.ibm.com/support/knowledgecenter/STXKQY/ > gpfsclustersfaq.html#linux > > that 4.2.3.5 is supported on RHEL 7.4, but with a footnote: > > "AFM, Integrated Protocols, and Installation Toolkit are not supported on > RHEL 7.4." > > We don't use AFM or the install toolkit. But we *do* make fairly heavy use > of mmces and nfs-ganesha - is that what they mean by "Integrated > Protocols"? > > (We're looking at doing upgrades next month while our HPC clusters are > doing > their upgrades - and going to 7.4 would be nice. If there's a mine field > there, I need to > make sure we stay at 7.3 - plus applicable non-7.4 updates) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m= > 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_ > I4NKvn8aFGXCqcJARCbu4AOE&e= > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/c01cc906/attachment.html > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 71, Issue 35 ********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=8ayFP1mF84gza8fbX73eZnWv6MoAQ-ZNctxuN0EsFjQ&s=kEhQxw29VlbFlbUt5Ml-bieqcsRiCv9aTx1cgdDS7Vk&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Mon Jan 8 10:38:14 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 8 Jan 2018 18:38:14 +0800 Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ In-Reply-To: References: Message-ID: This has been asked to Yong Ze to address through another email thread, so ok to ignore this thread request. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "IBM Spectrum Scale" To: "Lyle Gayne" Cc: gpfsug-discuss-bounces at spectrumscale.org, gpfsug main discussion list Date: 01/06/2018 06:46 PM Subject: Re: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Lyle, Can you please forward this to the right folks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Keith Ball To: gpfsug-discuss at spectrumscale.org Date: 12/20/2017 04:39 AM Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was recently trying to determine the latest RHEL release that will work with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code uses Scale 4.1.0.8. A specific X.Y GSS code build, from my experience, is intended to use a specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), which no longer has EUS support from RedHat (only 6.7 still does). GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be used, which is a surprising statement given GSS/ESS code's sensitivity to OS levels (any ESS I have ever seen has never been supported on more than one version of RHEL). According to the Scale FAQ ( https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux ), A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27! Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option here). Many Thanks, Keith_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=u7hLzIRwApuSZgw6uZ_j3uewYEyRTPxRmWJhpM_UAwY&s=bBgN8s6sKOU36_ivUi5NHGarxlF3TNLPyXS-7PA34F0&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=4uHjPqPoMOnTJmSUAO5lPaI-I9TxUZG7tosuUx85sIg&s=tPqgg4TTZ9J88YaLOrAT74IcV_33GWv11K2oywQOGDw&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From lgayne at us.ibm.com Mon Jan 8 14:53:59 2018 From: lgayne at us.ibm.com (Lyle Gayne) Date: Mon, 8 Jan 2018 09:53:59 -0500 Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ In-Reply-To: References: Message-ID: Yong Ze Chen has already been responding to these. Thanks, Lyle From: IBM Spectrum Scale/Poughkeepsie/IBM To: Lyle Gayne/Poughkeepsie/IBM at IBMUS Cc: gpfsug-discuss-bounces at spectrumscale.org, gpfsug main discussion list Date: 01/06/2018 05:46 AM Subject: Re: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: Huzefa H Pancha Hi Lyle, Can you please forward this to the right folks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Keith Ball To: gpfsug-discuss at spectrumscale.org Date: 12/20/2017 04:39 AM Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was recently trying to determine the latest RHEL release that will work with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code uses Scale 4.1.0.8. A specific X.Y GSS code build, from my experience, is intended to use a specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), which no longer has EUS support from RedHat (only 6.7 still does). GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be used, which is a surprising statement given GSS/ESS code's sensitivity to OS levels (any ESS I have ever seen has never been supported on more than one version of RHEL). According to the Scale FAQ ( https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux ), A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27! Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option here). Many Thanks, ? Keith_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=u7hLzIRwApuSZgw6uZ_j3uewYEyRTPxRmWJhpM_UAwY&s=bBgN8s6sKOU36_ivUi5NHGarxlF3TNLPyXS-7PA34F0&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Jan 8 16:52:05 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 8 Jan 2018 16:52:05 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we?d love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again? Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Stephan.Peinkofer at lrz.de Mon Jan 8 21:10:30 2018 From: Stephan.Peinkofer at lrz.de (Peinkofer, Stephan) Date: Mon, 8 Jan 2018 21:10:30 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> , Message-ID: Dear List, my very personal experience today, using the patched kernel for SLES 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS (4.2.3-4) did not even start (the kernel modules seemed to compile fine using mmbuildgpl). Interestingly, even when I disabled PTI explicitely, using the nopti kernel option, GPFS refused to start with the same error!? mmfs.log always showed something like this: ... /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: InsModWrapper[95]: eval: line 1: 3915: Memory fault ... 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading the mmfs kernel extension ... Since I had no time to investigate the issue further and raise a ticket right now, I just downgraded to the previous kernel and everything worked again. As we have to patch at least the login nodes of our HPC clusters asap, I would also appreciate if we could get a statement from IBM how the KPTI patches are expected to interact with GPFS and if there are any (general) problems, when we can expect updated GPFS packages. Many thanks in advance. Best Regards, Stephan Peinkofer ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Buterbaugh, Kevin L Sent: Monday, January 8, 2018 5:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we?d love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again? Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Mon Jan 8 22:57:20 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 8 Jan 2018 17:57:20 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) Message-ID: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> I was thinking some more about the >32 subblock feature in scale 5.0. As mentioned by IBM the biggest advantage of that feature is on filesystems with large blocks (e.g. multiple MB). The majority of our filesystems have a block size of 1MB which got me thinking... wouldn't it be nice if they had a larger block size (there seem to be compelling performance reasons for large file I/O to do this). I'm wondering, what's the feasibility is of supporting filesystem pools of varying block sizes within a single filesystem? I thought allocation maps exist on a per-pool basis which gives me some hope it's not too hard. If one could do this then, yes, you'd still need new hardware to migrate to a larger block size (and >32 subblocks), but it could be done as part of a refresh cycle *and* (and this is the part most important to me) it could be driven entirely by the policy engine which means storage admins are largely hands off and the migration is by and large transparent to the end user. This doesn't really address the need for a tool to address a filesystem migration to 4k metadata blocks (although I wonder if something could be done to create a system_4k pool that contains 4k-aligned metadata NSDs where key data structures get re-written during a restripe in a 4k-aligned manner, but that's really grasping at straws for me). -Aaorn -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From bbanister at jumptrading.com Mon Jan 8 23:48:57 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 8 Jan 2018 23:48:57 +0000 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. Cheers, -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Monday, January 08, 2018 4:57 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) Note: External Email ------------------------------------------------- I was thinking some more about the >32 subblock feature in scale 5.0. As mentioned by IBM the biggest advantage of that feature is on filesystems with large blocks (e.g. multiple MB). The majority of our filesystems have a block size of 1MB which got me thinking... wouldn't it be nice if they had a larger block size (there seem to be compelling performance reasons for large file I/O to do this). I'm wondering, what's the feasibility is of supporting filesystem pools of varying block sizes within a single filesystem? I thought allocation maps exist on a per-pool basis which gives me some hope it's not too hard. If one could do this then, yes, you'd still need new hardware to migrate to a larger block size (and >32 subblocks), but it could be done as part of a refresh cycle *and* (and this is the part most important to me) it could be driven entirely by the policy engine which means storage admins are largely hands off and the migration is by and large transparent to the end user. This doesn't really address the need for a tool to address a filesystem migration to 4k metadata blocks (although I wonder if something could be done to create a system_4k pool that contains 4k-aligned metadata NSDs where key data structures get re-written during a restripe in a 4k-aligned manner, but that's really grasping at straws for me). -Aaorn -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From aaron.s.knister at nasa.gov Tue Jan 9 00:13:40 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 8 Jan 2018 19:13:40 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: Thanks, Bryan! That's a great use case I hadn't thought of. GPFS can already support a different block size for the system pool so in my very simplistic view of the world it's already possible (unless there's some implementation detail about the system pool that lends itself to a different block size from all other pools that doesn't apply to other non-system pools differing from each other). -Aaron On 1/8/18 6:48 PM, Bryan Banister wrote: > Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. > > I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). > > NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. > > Cheers, > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister > Sent: Monday, January 08, 2018 4:57 PM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) > > Note: External Email > ------------------------------------------------- > > I was thinking some more about the >32 subblock feature in scale 5.0. As > mentioned by IBM the biggest advantage of that feature is on filesystems > with large blocks (e.g. multiple MB). The majority of our filesystems > have a block size of 1MB which got me thinking... wouldn't it be nice if > they had a larger block size (there seem to be compelling performance > reasons for large file I/O to do this). > > I'm wondering, what's the feasibility is of supporting filesystem pools > of varying block sizes within a single filesystem? I thought allocation > maps exist on a per-pool basis which gives me some hope it's not too hard. > > If one could do this then, yes, you'd still need new hardware to migrate > to a larger block size (and >32 subblocks), but it could be done as part > of a refresh cycle *and* (and this is the part most important to me) it > could be driven entirely by the policy engine which means storage admins > are largely hands off and the migration is by and large transparent to > the end user. > > This doesn't really address the need for a tool to address a filesystem > migration to 4k metadata blocks (although I wonder if something could be > done to create a system_4k pool that contains 4k-aligned metadata NSDs > where key data structures get re-written during a restripe in a > 4k-aligned manner, but that's really grasping at straws for me). > > -Aaorn > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From Greg.Lehmann at csiro.au Tue Jan 9 03:46:48 2018 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Tue, 9 Jan 2018 03:46:48 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> , Message-ID: This had me wondering, so I tried SLES 12 SP3 and thankfully GPFS v5 still runs after the kernel patch and an mmbuildgpl. It was just a test box I had at the time, so I don't have any comments on performance. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peinkofer, Stephan Sent: Tuesday, 9 January 2018 7:11 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Dear List, my very personal experience today, using the patched kernel for SLES 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS (4.2.3-4) did not even start (the kernel modules seemed to compile fine using mmbuildgpl). Interestingly, even when I disabled PTI explicitely, using the nopti kernel option, GPFS refused to start with the same error!? mmfs.log always showed something like this: ... /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: InsModWrapper[95]: eval: line 1: 3915: Memory fault ... 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading the mmfs kernel extension ... Since I had no time to investigate the issue further and raise a ticket right now, I just downgraded to the previous kernel and everything worked again. As we have to patch at least the login nodes of our HPC clusters asap, I would also appreciate if we could get a statement from IBM how the KPTI patches are expected to interact with GPFS and if there are any (general) problems, when we can expect updated GPFS packages. Many thanks in advance. Best Regards, Stephan Peinkofer ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of Buterbaugh, Kevin L > Sent: Monday, January 8, 2018 5:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we'd love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again... Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS - until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I'm sure that everyone is aware of Meltdown and Spectre by now ... we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I'm sure that everyone is aware of Meltdown and Spectre by now ... we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ... given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we'd be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks... Kevin P.S. The "Happy New Year" wasn't intended as sarcasm ... I hope it is a good year for everyone despite how it's starting out. :-O - Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From heiner.billich at psi.ch Tue Jan 9 08:24:22 2018 From: heiner.billich at psi.ch (Billich Heinrich Rainer (PSI)) Date: Tue, 9 Jan 2018 08:24:22 +0000 Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Message-ID: Hello, I just upgraded to 4.2.3-5 and now see many failures ?ib_rdma_nic_unrecognized? in mmhealth, like Component Status Status Change Reasons ------------------------------------------------------------------------------------------ NETWORK DEGRADED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized(mlx4_0/1) mlx4_0/1 FAILED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized I didn?t see this messages with 4.2.3-4. The relevant lines in /usr/lpp/mmfs/lib/mmsysmon/NetworkService.py changed between -4 and -5. What seems to happen: I have Mellanox VPI cards with one port Infiniband and one port Ethernet. mmhealth complains about the Ethernet port. Hmm ? I did specify the active Infiniband ports only in verbsPorts, I don?t see why mmhealth cares about any other ports when it checks RDMA. So probably a bug, I?ll open a PMR unless somebody points me to a different solution. I tried but I can?t hide this event in mmhealth. Cheers, Heiner -- Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://www.psi.ch From orichards at pixitmedia.com Tue Jan 9 09:09:47 2018 From: orichards at pixitmedia.com (Orlando Richards) Date: Tue, 9 Jan 2018 09:09:47 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: <22812c5d-5013-18b9-7178-7a2c620770dc@pixitmedia.com> Hi all, If it helps - 4.2.3-6 compiles and starts fine on the RHEL 7.4 patched kernel: 3.10.0-693.11.6.el7.x86_64. ______ Orlando On 09/01/2018 03:46, Greg.Lehmann at csiro.au wrote: > > This had me wondering, so I tried SLES 12 SP3 and thankfully GPFS v5 > still runs after the kernel patch and an mmbuildgpl. It was just a > test box I had at the time, so I don?t have any comments on performance. > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Peinkofer, Stephan > *Sent:* Tuesday, 9 January 2018 7:11 AM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS > > Dear List, > > my very?personal?experience today, using the patched kernel for SLES > 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS > (4.2.3-4)?did not even start (the kernel modules seemed to?compile > fine using mmbuildgpl). Interestingly, even when I disabled PTI > explicitely, using the nopti kernel option, GPFS?refused to start with > the same error!? > > mmfs.log always showed something like this: > > ... > > /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: > InsModWrapper[95]: eval: line 1: 3915: Memory fault > > ... > > 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading > the mmfs kernel extension > > ... > > Since I had no time to investigate the issue further and raise a > ticket right now, I just downgraded?to the previous kernel and > everything worked again. > > As we have to patch at least?the login nodes of our HPC clusters asap, > I would also appreciate if we could get a statement from IBM how the > KPTI patches are expected to interact with GPFS and if there are any > (general)?problems, when we can expect updated GPFS packages. > > Many thanks in advance. > Best Regards, > > Stephan Peinkofer > > ------------------------------------------------------------------------ > > *From:*gpfsug-discuss-bounces at spectrumscale.org > > > on behalf of > Buterbaugh, Kevin L > > *Sent:* Monday, January 8, 2018 5:52 PM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS > > Hi GPFS Team, > > Thanks for this response. ?If it is at all possible I know that we > (and I would suspect many others are in this same boat) would greatly > appreciate a update from IBM on how a patched kernel impacts GPFS > functionality. ?Yes, we?d love to know the performance impact of the > patches on GPFS, but that pales in significance to knowing whether > GPFS version 4.x.x.x will even *start* with the patched kernel(s). > > Thanks again? > > Kevin > > > > On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: > > Kevin, > > The team is aware of Meltdown and Spectre. Due to the late > availability of production-ready test patches (they became > available today) we started today working on evaluating the impact > of applying these patches. The focus would be both on any > potential functional impacts (especially to the kernel modules > shipped with GPFS) and on the performance degradation which > affects user/kernel mode transitions. Performance characterization > will be complex, as some system calls which may get invoked often > by the mmfsd daemon will suddenly become significantly more > expensive because of the kernel changes. Depending on the main > areas affected, code changes might be possible to alleviate the > impact, by reducing frequency of certain calls, etc. Any such > changes will be deployed over time. > > At this point, we can't say what impact this will have on > stability or Performance on systems running GPFS ? until IBM > issues an official statement on this topic. We hope to have some > basic answers soon. > > > > Regards, The Spectrum Scale (GPFS) team > > ------------------------------------------------------------------------------------------------------------------ > If you feel that your question can benefit other users of Spectrum > Scale (GPFS), then please post it to the public IBM developerWroks > Forum at > https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 > . > > > If your query concerns a potential software error in Spectrum > Scale (GPFS) and you have an IBM software maintenance contract > please contact 1-800-237-5511 in the United States or your local > IBM Service Center in other countries. > > The forum is informally monitored as time permits and should not > be used for priority messages to the Spectrum Scale (GPFS) team. > > "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 > PM---Happy New Year everyone, I?m sure that everyone is aware of > Meltdown and Spectre by now ? we, like m > > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > Date: 01/04/2018 01:11 PM > Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > ------------------------------------------------------------------------ > > > > > Happy New Year everyone, > > I?m sure that everyone is aware of Meltdown and Spectre by now ? > we, like many other institutions, will be patching for it at the > earliest possible opportunity. > > Our understanding is that the most serious of the negative > performance impacts of these patches will be for things like I/O > (disk / network) ? given that, we are curious if IBM has any plans > for a GPFS update that could help mitigate those impacts? Or is > there simply nothing that can be done? > > If there is a GPFS update planned for this we?d be interested in > knowing so that we could coordinate the kernel and GPFS upgrades > on our cluster. > > Thanks? > > Kevin > > P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it > is a good year for everyone despite how it?s starting out. :-O > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu > - (615)875-9633 > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- *Orlando Richards* VP Product Development, Pixit Media 07930742808|orichards at pixitmedia.com www.pixitmedia.com |Tw:@pixitmedia -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From MDIETZ at de.ibm.com Tue Jan 9 09:43:58 2018 From: MDIETZ at de.ibm.com (Mathias Dietz) Date: Tue, 9 Jan 2018 10:43:58 +0100 Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized In-Reply-To: References: Message-ID: Hello Heiner, with 4.2.3-5 mmhealth is always monitoring all ports of a configured IB adapter even if the port is not specified in verbsPorts. Development has implemented a fix which is planned to be part of 4.2.3-7 (February). To get rid of the false alarm in the meantime you could disable the Infiniband monitoring altogether. To disable Infiniband monitoring on a node: 1. Open the file /var/mmfs/mmsysmon/mmsysmonitor.conf 2. Locate the [network]section 3. Add below: ib_rdma_enable_monitoring=False 4. Save file and run "mmsysmoncontrol restart" If you have questions feel free to contact me directly by email. Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale RAS Architect & Release Lead Architect (4.2.3/5.0) --------------------------------------------------------------------------- IBM Deutschland Am Weiher 24 65451 Kelsterbach Phone: +49 70342744105 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ----------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk WittkoppSitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Billich Heinrich Rainer (PSI)" To: gpfsug main discussion list Date: 01/09/2018 09:31 AM Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I just upgraded to 4.2.3-5 and now see many failures ?ib_rdma_nic_unrecognized? in mmhealth, like Component Status Status Change Reasons ------------------------------------------------------------------------------------------ NETWORK DEGRADED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized(mlx4_0/1) mlx4_0/1 FAILED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized I didn?t see this messages with 4.2.3-4. The relevant lines in /usr/lpp/mmfs/lib/mmsysmon/NetworkService.py changed between -4 and -5. What seems to happen: I have Mellanox VPI cards with one port Infiniband and one port Ethernet. mmhealth complains about the Ethernet port. Hmm ? I did specify the active Infiniband ports only in verbsPorts, I don?t see why mmhealth cares about any other ports when it checks RDMA. So probably a bug, I?ll open a PMR unless somebody points me to a different solution. I tried but I can?t hide this event in mmhealth. Cheers, Heiner -- Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://www.psi.ch _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Tue Jan 9 10:19:49 2018 From: p.childs at qmul.ac.uk (Peter Childs) Date: Tue, 9 Jan 2018 10:19:49 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: <22812c5d-5013-18b9-7178-7a2c620770dc@pixitmedia.com> References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> <22812c5d-5013-18b9-7178-7a2c620770dc@pixitmedia.com> Message-ID: <1515493189.28898.65.camel@qmul.ac.uk> On Tue, 2018-01-09 at 09:09 +0000, Orlando Richards wrote: Hi all, If it helps - 4.2.3-6 compiles and starts fine on the RHEL 7.4 patched kernel: 3.10.0-693.11.6.el7.x86_64. We have 4.2.3-4 working fine on CentOS 7.4 with patched kernel 3.10-0-693.11.6.el7.x86_64. So far we have found CentOs 7.4 (patched and unpatched) to be faster than CentOS 7.3, but CentOs 7.4 patched to be slower than unpatched CentOs 7.4. But I've not seen the details as to this testing so can't really give any further details. Peter ______ Orlando On 09/01/2018 03:46, Greg.Lehmann at csiro.au wrote: This had me wondering, so I tried SLES 12 SP3 and thankfully GPFS v5 still runs after the kernel patch and an mmbuildgpl. It was just a test box I had at the time, so I don?t have any comments on performance. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peinkofer, Stephan Sent: Tuesday, 9 January 2018 7:11 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Dear List, my very personal experience today, using the patched kernel for SLES 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS (4.2.3-4) did not even start (the kernel modules seemed to compile fine using mmbuildgpl). Interestingly, even when I disabled PTI explicitely, using the nopti kernel option, GPFS refused to start with the same error!? mmfs.log always showed something like this: ... /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: InsModWrapper[95]: eval: line 1: 3915: Memory fault ... 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading the mmfs kernel extension ... Since I had no time to investigate the issue further and raise a ticket right now, I just downgraded to the previous kernel and everything worked again. As we have to patch at least the login nodes of our HPC clusters asap, I would also appreciate if we could get a statement from IBM how the KPTI patches are expected to interact with GPFS and if there are any (general) problems, when we can expect updated GPFS packages. Many thanks in advance. Best Regards, Stephan Peinkofer ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of Buterbaugh, Kevin L > Sent: Monday, January 8, 2018 5:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we?d love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again? Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Orlando Richards VP Product Development, Pixit Media 07930742808 | orichards at pixitmedia.com www.pixitmedia.com | Tw:@pixitmedia [http://pixitmedia.com/sig/pixitmedia-9-2018.png] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Peter Childs ITS Research Storage Queen Mary, University of London -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcram at ddn.com Tue Jan 9 15:30:36 2018 From: jcram at ddn.com (Jeno Cram) Date: Tue, 9 Jan 2018 15:30:36 +0000 Subject: [gpfsug-discuss] SMB with LDAP Message-ID: Has anyone had any luck implementing CES with SMB using LDAP? This link doesn?t seem to have all of the information required to getting it working properly. https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_updateldapsmb.htm 1. I?m assuming smbpasswd -a is still required for all users? 2. What keeps the LDAP/SMB passwords in sync? 3. What access does the bind user need in LDAP? 4. What special requirements are required for Windows 10 clients to connect? Jeno Cram | Lead Application Support Engineer jcram at ddn.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Jan 9 15:51:03 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 9 Jan 2018 15:51:03 +0000 Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized In-Reply-To: References: Message-ID: <7828e86d6c494dfdb95a6dbbd6439f09@jumptrading.com> I can't help but comment that it's amazing that GPFS is using a txt config file instead of requiring a command run that stores config data into a non-editable (but still editable) flat file database... Wow 2018!! Hahahahaha! -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mathias Dietz Sent: Tuesday, January 09, 2018 3:44 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Note: External Email ________________________________ Hello Heiner, with 4.2.3-5 mmhealth is always monitoring all ports of a configured IB adapter even if the port is not specified in verbsPorts. Development has implemented a fix which is planned to be part of 4.2.3-7 (February). To get rid of the false alarm in the meantime you could disable the Infiniband monitoring altogether. To disable Infiniband monitoring on a node: 1. Open the file /var/mmfs/mmsysmon/mmsysmonitor.conf 2. Locate the [network]section 3. Add below: ib_rdma_enable_monitoring=False 4. Save file and run "mmsysmoncontrol restart" If you have questions feel free to contact me directly by email. Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale RAS Architect & Release Lead Architect (4.2.3/5.0) --------------------------------------------------------------------------- IBM Deutschland Am Weiher 24 65451 Kelsterbach Phone: +49 70342744105 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ----------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk WittkoppSitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Billich Heinrich Rainer (PSI)" > To: gpfsug main discussion list > Date: 01/09/2018 09:31 AM Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hello, I just upgraded to 4.2.3-5 and now see many failures 'ib_rdma_nic_unrecognized' in mmhealth, like Component Status Status Change Reasons ------------------------------------------------------------------------------------------ NETWORK DEGRADED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized(mlx4_0/1) mlx4_0/1 FAILED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized I didn't see this messages with 4.2.3-4. The relevant lines in /usr/lpp/mmfs/lib/mmsysmon/NetworkService.py changed between -4 and -5. What seems to happen: I have Mellanox VPI cards with one port Infiniband and one port Ethernet. mmhealth complains about the Ethernet port. Hmm - I did specify the active Infiniband ports only in verbsPorts, I don't see why mmhealth cares about any other ports when it checks RDMA. So probably a bug, I'll open a PMR unless somebody points me to a different solution. I tried but I can't hide this event in mmhealth. Cheers, Heiner -- Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://www.psi.ch _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jan 9 22:44:04 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 17:44:04 -0500 Subject: [gpfsug-discuss] interrupted inode expansion Message-ID: I just ran into this in our test environment with 4.2.3.6: http://www-01.ibm.com/support/docview.wss?crawler=1&uid=isg1IV94666 Has anyone else run into this? I'm about to request an efix for it. -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Tue Jan 9 22:47:59 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 17:47:59 -0500 Subject: [gpfsug-discuss] interrupted inode expansion In-Reply-To: References: Message-ID: <0b0daec8-11aa-8d6d-961e-75120520a125@nasa.gov> Question for the Scale team-- is this bug present in the 4.1 stream (or the 5.0 stream for that matter)? On 1/9/18 5:44 PM, Aaron Knister wrote: > I just ran into this in our test environment with 4.2.3.6: > > http://www-01.ibm.com/support/docview.wss?crawler=1&uid=isg1IV94666 > > Has anyone else run into this? I'm about to request an efix for it. > > -Aaron > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From christof.schmitt at us.ibm.com Tue Jan 9 23:34:06 2018 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 9 Jan 2018 23:34:06 +0000 Subject: [gpfsug-discuss] SMB with LDAP In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Wed Jan 10 00:56:08 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 19:56:08 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: <3a1a5b22-76cc-2aa1-7d89-c03619f8d679@nasa.gov> Brian, I stole your wording and created an RFE for this: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115012 -Aaron On 1/8/18 6:48 PM, Bryan Banister wrote: > Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. > > I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). > > NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. > > Cheers, > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister > Sent: Monday, January 08, 2018 4:57 PM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) > > Note: External Email > ------------------------------------------------- > > I was thinking some more about the >32 subblock feature in scale 5.0. As > mentioned by IBM the biggest advantage of that feature is on filesystems > with large blocks (e.g. multiple MB). The majority of our filesystems > have a block size of 1MB which got me thinking... wouldn't it be nice if > they had a larger block size (there seem to be compelling performance > reasons for large file I/O to do this). > > I'm wondering, what's the feasibility is of supporting filesystem pools > of varying block sizes within a single filesystem? I thought allocation > maps exist on a per-pool basis which gives me some hope it's not too hard. > > If one could do this then, yes, you'd still need new hardware to migrate > to a larger block size (and >32 subblocks), but it could be done as part > of a refresh cycle *and* (and this is the part most important to me) it > could be driven entirely by the policy engine which means storage admins > are largely hands off and the migration is by and large transparent to > the end user. > > This doesn't really address the need for a tool to address a filesystem > migration to 4k metadata blocks (although I wonder if something could be > done to create a system_4k pool that contains 4k-aligned metadata NSDs > where key data structures get re-written during a restripe in a > 4k-aligned manner, but that's really grasping at straws for me). > > -Aaorn > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Wed Jan 10 00:57:48 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 19:57:48 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: <3a1a5b22-76cc-2aa1-7d89-c03619f8d679@nasa.gov> References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> <3a1a5b22-76cc-2aa1-7d89-c03619f8d679@nasa.gov> Message-ID: <6a1912f4-d519-f20e-ceb1-af55e2cb7f31@nasa.gov> *Bryan (sorry for the typo) On 1/9/18 7:56 PM, Aaron Knister wrote: > Brian, > > I stole your wording and created an RFE for this: > > http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115012 > > -Aaron > > On 1/8/18 6:48 PM, Bryan Banister wrote: >> Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. >> >> I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). >> >> NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. >> >> Cheers, >> -Bryan >> >> -----Original Message----- >> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister >> Sent: Monday, January 08, 2018 4:57 PM >> To: gpfsug main discussion list >> Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) >> >> Note: External Email >> ------------------------------------------------- >> >> I was thinking some more about the >32 subblock feature in scale 5.0. As >> mentioned by IBM the biggest advantage of that feature is on filesystems >> with large blocks (e.g. multiple MB). The majority of our filesystems >> have a block size of 1MB which got me thinking... wouldn't it be nice if >> they had a larger block size (there seem to be compelling performance >> reasons for large file I/O to do this). >> >> I'm wondering, what's the feasibility is of supporting filesystem pools >> of varying block sizes within a single filesystem? I thought allocation >> maps exist on a per-pool basis which gives me some hope it's not too hard. >> >> If one could do this then, yes, you'd still need new hardware to migrate >> to a larger block size (and >32 subblocks), but it could be done as part >> of a refresh cycle *and* (and this is the part most important to me) it >> could be driven entirely by the policy engine which means storage admins >> are largely hands off and the migration is by and large transparent to >> the end user. >> >> This doesn't really address the need for a tool to address a filesystem >> migration to 4k metadata blocks (although I wonder if something could be >> done to create a system_4k pool that contains 4k-aligned metadata NSDs >> where key data structures get re-written during a restripe in a >> 4k-aligned manner, but that's really grasping at straws for me). >> >> -Aaorn >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ________________________________ >> >> Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Wed Jan 10 02:01:13 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 21:01:13 -0500 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central In-Reply-To: References: Message-ID: <7fe4019d-8034-489a-631b-0c7c1134cce2@nasa.gov> Hi Steve, I can definitely understand the want for a newer compiler, but here's a question for you and the Scale team-- what requires the newer compiler? The userspace tools or the kernel module? I ask because as we all know it can be quite easy to install a modern build chain even on an older operating system or even build binaries on a new operating system that will run on an older operating system. Surely the userspace components could be built with a newer compiler and the gplbin layer would just be built with whatever compiler is present on the system. I suspect there's more complexity and subtlety to the issue but I thought I'd at least ask. I know SLES12 has been out for a while now but that doesn't mean we can just pick up and move to a new OS without some difficulty, especially if the older OS is working just fine :) -Aaron On 12/19/17 8:18 AM, Steve Duersch wrote: > As Mike Taylor pointed out in a previous post this was an incorrect > statement. > You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a > rolling upgrade. > The minReleaseLevel is not pertinent to a rolling upgrade. The running > daemon is the important part. So you can't have any 4.1.x nodes in your > cluster and do a rolling upgrade to 5.0. > > Also, Aaron, as to the OS support. This decision was not made without > some angst. As I mentioned at the user group meeting in NYC...the key > point is that we would like to get to a more current compiler. This will > allow us to take advantage of newer features and functions and hopefully > make the code better for customers. SLES 12 has been around for over 2 > years. > > I hope this helps give some thinking behind the decision. > > > Steve Duersch > Spectrum Scale > 845-433-7902 > IBM Poughkeepsie, New York > > >> Today's Topics: >> >> ? ?1. Re: Spectrum Scale 5.0 now available on Fix Central >> ? ? ? (Sobey, Richard A) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 19 Dec 2017 09:06:08 +0000 >> From: "Sobey, Richard A" >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix >> ? ?Central >> Message-ID: >> ? ? >> > >> ? ? >> Content-Type: text/plain; charset="utf-8" >> >> Hi Robert >> >> Do you mean the minReleaseLevel from mmlsconfig or just making sure >> all the nodes are running 4.2.3? >> >> Cheers! >> Richard >> >> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug- >> discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert >> Sent: 18 December 2017 19:44 >> To: gpfsug main discussion list >> Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix > Central >> >> The Scale 5.0 fix level is now up on Fix Central. >> >> You need to be at Scale 4.2.3 (cluster level) to do a rolling >> upgrade to this level. >> >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From Robert.Oesterlin at nuance.com Wed Jan 10 02:12:16 2018 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 10 Jan 2018 02:12:16 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central Message-ID: I'm in the same boat here with deprecated support for RH6. We have a number of older applications supporting product that won't run on RH7. I'm still pondering what I'm going to do here. Or stay perpetually stuck on 4.2.3 on some clusters. Bob Oesterlin Sr Principal Storage Engineer, Nuance ?On 1/9/18, 8:02 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: I know SLES12 has been out for a while now but that doesn't mean we can just pick up and move to a new OS without some difficulty, especially if the older OS is working just fine :) From aaron.s.knister at nasa.gov Wed Jan 10 02:32:24 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 21:32:24 -0500 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central In-Reply-To: References: Message-ID: I just opened an RFE for RHEL6, SLES11 and Debian support in Scale 5.0: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115019 -Aaron On 1/9/18 9:12 PM, Oesterlin, Robert wrote: > I'm in the same boat here with deprecated support for RH6. We have a number of older applications supporting product that won't run on RH7. I'm still pondering what I'm going to do here. Or stay perpetually stuck on 4.2.3 on some clusters. > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > ?On 1/9/18, 8:02 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: > > I know SLES12 has been out for a while now but that doesn't mean we can > just pick up and move to a new OS without some difficulty, especially if > the older OS is working just fine :) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From sandeep.patil at in.ibm.com Wed Jan 10 06:43:11 2018 From: sandeep.patil at in.ibm.com (Sandeep Ramesh) Date: Wed, 10 Jan 2018 12:13:11 +0530 Subject: [gpfsug-discuss] Latest Technical Blogs on Spectrum Scale In-Reply-To: References: Message-ID: Dear User Group Members, Here are list of development blogs in the last quarter. Passing it to this email group as Doris had got a feedback in the UG meetings to notify the members with the latest updates periodically. Genomic Workloads ? How To Get it Right From Infrastructure Point Of View. https://developer.ibm.com/storage/2018/01/06/genomic-workloads-get-right-infrastructure-point-view/ IBM Spectrum Scale Versus Apache Hadoop HDFS https://developer.ibm.com/storage/2018/01/10/spectrumscale_vs_hdfs/ ESS Fault Tolerance https://developer.ibm.com/storage/2018/01/09/ess-fault-tolerance/ IBM Spectrum Scale MMFSCK ? Savvy Enhancements https://developer.ibm.com/storage/2018/01/05/ibm-spectrum-scale-mmfsck-savvy-enhancements/ ESS Disk Management https://developer.ibm.com/storage/2018/01/02/ess-disk-management/ IBM Spectrum Scale Object Protocol On Ubuntu https://developer.ibm.com/storage/2018/01/01/ibm-spectrum-scale-object-protocol-ubuntu/ IBM Spectrum Scale 5.0 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2017/12/20/ibm-spectrum-scale-5-0-whats-new-object/ A Complete Guide to ? Protocol Problem Determination Guide for IBM Spectrum Scale? ? Part 1 https://developer.ibm.com/storage/2017/12/19/complete-guide-protocol-problem-determination-guide-ibm-spectrum-scale-1/ IBM Spectrum Scale installation toolkit ? enhancements over releases https://developer.ibm.com/storage/2017/12/15/ibm-spectrum-scale-installation-toolkit-enhancements-releases/ Network requirements in an Elastic Storage Server Setup https://developer.ibm.com/storage/2017/12/13/network-requirements-in-an-elastic-storage-server-setup/ Co-resident migration with Transparent cloud tierin https://developer.ibm.com/storage/2017/12/05/co-resident-migration-transparent-cloud-tierin/ IBM Spectrum Scale on Hortonworks HDP Hadoop clusters : A Complete Big Data Solution https://developer.ibm.com/storage/2017/12/05/ibm-spectrum-scale-hortonworks-hdp-hadoop-clusters-complete-big-data-solution/ Big data analytics with Spectrum Scale using remote cluster mount & multi-filesystem support https://developer.ibm.com/storage/2017/11/28/big-data-analytics-spectrum-scale-using-remote-cluster-mount-multi-filesystem-support/ IBM Spectrum Scale HDFS Transparency Short Circuit Write Support https://developer.ibm.com/storage/2017/11/28/ibm-spectrum-scale-hdfs-transparency-short-circuit-write-support/ IBM Spectrum Scale HDFS Transparency Federation Support https://developer.ibm.com/storage/2017/11/27/ibm-spectrum-scale-hdfs-transparency-federation-support/ How to configure and performance tuning different system workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-different-system-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Spark workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-spark-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning database workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-database-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Hadoop workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/24/configure-performance-tuning-hadoop-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ IBM Spectrum Scale Sharing Nothing Cluster Performance Tuning https://developer.ibm.com/storage/2017/11/24/ibm-spectrum-scale-sharing-nothing-cluster-performance-tuning/ How to Configure IBM Spectrum Scale? with NIS based Authentication. https://developer.ibm.com/storage/2017/11/21/configure-ibm-spectrum-scale-nis-based-authentication/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Cc: Doris Conti/Poughkeepsie/IBM at IBMUS Date: 11/16/2017 08:15 PM Subject: Latest Technical Blogs on Spectrum Scale Dear User Group members, Here are the Development Blogs in last 3 months on Spectrum Scale Technical Topics. Spectrum Scale Monitoring ? Know More ? https://developer.ibm.com/storage/2017/11/16/spectrum-scale-monitoring-know/ IBM Spectrum Scale 5.0 Release ? What?s coming ! https://developer.ibm.com/storage/2017/11/14/ibm-spectrum-scale-5-0-release-whats-coming/ Four Essentials things to know for managing data ACLs on IBM Spectrum Scale? from Windows https://developer.ibm.com/storage/2017/11/13/four-essentials-things-know-managing-data-acls-ibm-spectrum-scale-windows/ GSSUTILS: A new way of running SSR, Deploying or Upgrading ESS Server https://developer.ibm.com/storage/2017/11/13/gssutils/ IBM Spectrum Scale Object Authentication https://developer.ibm.com/storage/2017/11/02/spectrum-scale-object-authentication/ Video Surveillance ? Choosing the right storage https://developer.ibm.com/storage/2017/11/02/video-surveillance-choosing-right-storage/ IBM Spectrum scale object deep dive training with problem determination https://www.slideshare.net/SmitaRaut/ibm-spectrum-scale-object-deep-dive-training Spectrum Scale as preferred software defined storage for Ubuntu OpenStack https://developer.ibm.com/storage/2017/09/29/spectrum-scale-preferred-software-defined-storage-ubuntu-openstack/ IBM Elastic Storage Server 2U24 Storage ? an All-Flash offering, a performance workhorse https://developer.ibm.com/storage/2017/10/06/ess-5-2-flash-storage/ A Complete Guide to Configure LDAP-based authentication with IBM Spectrum Scale? for File Access https://developer.ibm.com/storage/2017/09/21/complete-guide-configure-ldap-based-authentication-ibm-spectrum-scale-file-access/ Deploying IBM Spectrum Scale on AWS Quick Start https://developer.ibm.com/storage/2017/09/18/deploy-ibm-spectrum-scale-on-aws-quick-start/ Monitoring Spectrum Scale Object metrics https://developer.ibm.com/storage/2017/09/14/monitoring-spectrum-scale-object-metrics/ Tier your data with ease to Spectrum Scale Private Cloud(s) using Moonwalk Universal https://developer.ibm.com/storage/2017/09/14/tier-data-ease-spectrum-scale-private-clouds-using-moonwalk-universal/ Why do I see owner as ?Nobody? for my export mounted using NFSV4 Protocol on IBM Spectrum Scale?? https://developer.ibm.com/storage/2017/09/08/see-owner-nobody-export-mounted-using-nfsv4-protocol-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory and LDAP https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-ldap/ IBM Spectrum Scale? Authentication using Active Directory and RFC2307 https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-rfc2307/ High Availability Implementation with IBM Spectrum Virtualize and IBM Spectrum Scale https://developer.ibm.com/storage/2017/08/30/high-availability-implementation-ibm-spectrum-virtualize-ibm-spectrum-scale/ 10 Frequently asked Questions on configuring Authentication using AD + AUTO ID mapping on IBM Spectrum Scale?. https://developer.ibm.com/storage/2017/08/04/10-frequently-asked-questions-configuring-authentication-using-ad-auto-id-mapping-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory https://developer.ibm.com/storage/2017/07/30/ibm-spectrum-scale-auth-using-active-directory/ Five cool things that you didn?t know Transparent Cloud Tiering on Spectrum Scale can do https://developer.ibm.com/storage/2017/07/29/five-cool-things-didnt-know-transparent-cloud-tiering-spectrum-scale-can/ IBM Spectrum Scale GUI videos https://developer.ibm.com/storage/2017/07/25/ibm-spectrum-scale-gui-videos/ IBM Spectrum Scale? Authentication ? Planning for NFS Access https://developer.ibm.com/storage/2017/07/24/ibm-spectrum-scale-planning-nfs-access/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Wed Jan 10 14:32:09 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 10 Jan 2018 14:32:09 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central In-Reply-To: References: Message-ID: <089336cb58b6443e98d28bd4d6b16724@jumptrading.com> I felt some sort of obligation to stand by my own wording... +1 vote for me!!! -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Tuesday, January 09, 2018 8:32 PM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central Note: External Email ------------------------------------------------- I just opened an RFE for RHEL6, SLES11 and Debian support in Scale 5.0: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115019 -Aaron On 1/9/18 9:12 PM, Oesterlin, Robert wrote: > I'm in the same boat here with deprecated support for RH6. We have a number of older applications supporting product that won't run on RH7. I'm still pondering what I'm going to do here. Or stay perpetually stuck on 4.2.3 on some clusters. > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > ?On 1/9/18, 8:02 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: > > I know SLES12 has been out for a while now but that doesn't mean we can > just pick up and move to a new OS without some difficulty, especially if > the older OS is working just fine :) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From Achim.Rehor at de.ibm.com Wed Jan 10 15:57:48 2018 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Wed, 10 Jan 2018 16:57:48 +0100 Subject: [gpfsug-discuss] interrupted inode expansion In-Reply-To: <0b0daec8-11aa-8d6d-961e-75120520a125@nasa.gov> References: <0b0daec8-11aa-8d6d-961e-75120520a125@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: From vpuvvada at in.ibm.com Fri Jan 12 06:50:49 2018 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Fri, 12 Jan 2018 12:20:49 +0530 Subject: [gpfsug-discuss] Use AFM for migration of many small files In-Reply-To: <1515064616.28898.32.camel@qmul.ac.uk> References: <467FB293-D33B-46F4-BA1B-A5CB4D28DDE6@psi.ch> <1515064616.28898.32.camel@qmul.ac.uk> Message-ID: >The plan is to load the new cache from the old GPFS then dump once the cache is full. >We've already increase numThreashThreads from 4 to 8 and seen only marginal improvements, we could attempt to increase this further. AFM have replication performance issues with small files on high latency networks. There is a plan to fix these issues. >I'm also wondering if its worth increasing the Refresh Intervals to speed up read of already cache files. At this stage we want to fill the cache and don't care about write back until we switch the target to the >new NFS/GPFS from our old GPFS storage to a new box back at our off-site location, (otherwise known as the office) Increasing the refresh intervals will improve the application performance at cache site. It is better to set large refresh intervals if the cache is the only writer. ~Venkat (vpuvvada at in.ibm.com) From: Peter Childs To: "gpfsug-discuss at spectrumscale.org" Date: 01/04/2018 04:47 PM Subject: Re: [gpfsug-discuss] Use AFM for migration of many small files Sent by: gpfsug-discuss-bounces at spectrumscale.org We are doing something very similar using 4.2.3-4 and GPFS 4.2.1-1 on the nfs backend. Did you have any success? The plan is to load the new cache from the old GPFS then dump once the cache is full. We've already increase numThreashThreads from 4 to 8 and seen only marginal improvements, we could attempt to increase this further. I'm also wondering if its worth increasing the Refresh Intervals to speed up read of already cache files. At this stage we want to fill the cache and don't care about write back until we switch the target to the new NFS/GPFS from our old GPFS storage to a new box back at our off-site location, (otherwise known as the office) [root at afmgateway1 scratch]# mmlsfileset home home -L --afm Filesets in file system 'home': Attributes for fileset home: ============================= Status Linked Path /data2/home Id 42 Root inode 1343225859 Parent Id 0 Created Wed Jan 3 12:32:33 2018 Comment Inode space 41 Maximum number of inodes 100000000 Allocated inodes 15468544 Permission change flag chmodAndSetacl afm-associated Yes Target nfs://afm1/gpfs/data1/afm/home Mode single-writer File Lookup Refresh Interval 30 (default) File Open Refresh Interval 30 (default) Dir Lookup Refresh Interval 60 (default) Dir Open Refresh Interval 60 (default) Async Delay 15 (default) Last pSnapId 0 Display Home Snapshots no Number of Gateway Flush Threads 8 Prefetch Threshold 0 (default) Eviction Enabled no Thanks in advance. Peter Childs On Tue, 2017-09-05 at 19:57 +0530, Venkateswara R Puvvada wrote: Which version of Spectrum Scale ? What is the fileset mode ? >We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here >I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. How was the performance measured ? If parallel IO is enabled, AFM uses multiple gateway nodes to prefetch the large files (if file size if more than 1GB). Performance difference between small and lager file is huge (1000MB - 150MB = 850MB) here, and generally it is not the case. How many files were present in list file for prefetch ? Could you also share full internaldump from the gateway node ? >I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few >read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. AFM prefetches the files on multiple threads. Default flush threads for prefetch are 36 (fileset.afmNumFlushThreads (default 4) + afmNumIOFlushThreads (default 32)). >Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and >I need to look elsewhere to get better performance for prefetch of many smaller files? See above, AFM reads files on multiple threads parallelly. Try increasing the afmNumFlushThreads on fileset and verify if it improves the performance. ~Venkat (vpuvvada at in.ibm.com) From: "Billich Heinrich Rainer (PSI)" To: "gpfsug-discuss at spectrumscale.org" Date: 09/04/2017 10:18 PM Subject: [gpfsug-discuss] Use AFM for migration of many small files Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and I need to look elsewhere to get better performance for prefetch of many smaller files? We will migrate several filesets in parallel, but still with individual filesets up to 350TB in size 150MB/s isn?t fun. Also just about 150 files/s seconds looks poor. The setup is quite new, hence there may be other places to look at. It?s all RHEL7 an spectrum scale 4.2.2-3 on the afm cache. Thank you, Heiner --, Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://urldefense.proofpoint.com/v2/url?u=https-3A__www.psi.ch&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=eHcVdovN10-m-Qk0Ln2qvol3pkKNFwrzz2wgf1zXVXE&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=LbRyuSM_djs0FDXr27hPottQHAn3OGcivpyRcIDBN3U&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=eMSpfXQgE3wMSVM0G0vCYr6LEgERhP8iGfw3uNk8lLs&s=mcQ13uhvwS4yPbA2uCmwccc7l4mjTmL2fAdPLimS0Hc&e= -- Peter Childs ITS Research Storage Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=07QQkI0Rg8NyUEgPIuJwfg3elEXqTpOjIFpy2WbaEg0&s=kGEDPbMo64yU7Tcwu61ggT89tfq_3QdX-r6NoANXh78&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From coetzee.ray at gmail.com Fri Jan 12 10:29:43 2018 From: coetzee.ray at gmail.com (Ray Coetzee) Date: Fri, 12 Jan 2018 10:29:43 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale Message-ID: I'd like to ask the group of their experiences in improving the performance of applications that use mmap calls against files on Spectrum Scale. Besides using an NFS export from CES instead of a native GPFS mount, or precaching the dataset into the pagepool, what other approaches are there to offset the performance hit of the 4K IO size? Kind regards Ray Coetzee -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Jan 12 14:45:29 2018 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 12 Jan 2018 14:45:29 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: Message-ID: what version of Scale are you using right now ? On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee wrote: > I'd like to ask the group of their experiences in improving the > performance of applications that use mmap calls against files on Spectrum > Scale. > > Besides using an NFS export from CES instead of a native GPFS mount, or > precaching the dataset into the pagepool, what other approaches are there > to offset the performance hit of the 4K IO size? > > Kind regards > > Ray Coetzee > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Thu Jan 11 14:16:26 2018 From: john.hearns at asml.com (John Hearns) Date: Thu, 11 Jan 2018 14:16:26 +0000 Subject: [gpfsug-discuss] System running out of memory - SUnreclaim is huge Message-ID: I am having problems with GPFS servers running out of memory. We have an open PMR for this, however if anyone has seen this or has any ideas I would be grateful for a heads up. Servers have 128 Gbytes f RAM, kernel 2.6.32-573.18.1.el6.x86_64, GPFS version 4.2.3.4 In the latest incident the free memory went to below 1Gbyte, and we started to have processes killed, including our monitoring setup. I shut down GPFS on that server and /proc/meminfo still shows: Slab: 111192296 kB SReclaimable: 29020 kB SUnreclaim: 111163276 kB Am I barking up a wrong tree here and pointing the finger at GPFS? Something is causing the scsi_data_buffer slab memory usage (see below). One thing I did yesterday was to change the disk scheduler for each disk from cfq to dealine (as recommended in the tuning guide) However the server was already in short memory at that point. Slabtop shows Active / Total Objects (% used) : -306803185 / -306722574 (100.0%) Active / Total Slabs (% used) : 27749714 / 27749719 (100.0%) Active / Total Caches (% used) : 115 / 198 (58.1%) Active / Total Size (% used) : 93857848.58K / 93872319.47K (100.0%) Minimum / Average / Maximum Object : 0.02K / 0.02K / 4096.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 3987822096 3987821817 0% 0.02K 27693209 144 110772836K scsi_data_buffer 91155 64448 70% 0.06K 1545 59 6180K size-64 36064 32035 88% 0.03K 322 112 1288K size-32 35505 34334 96% 0.25K 2367 15 9468K skbuff_head_cache 33876 33874 99% 8.00K 33876 1 271008K size-8192 33804 33615 99% 0.14K 1252 27 5008K sysfs_dir_cache -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Fri Jan 12 16:12:13 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 12 Jan 2018 16:12:13 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: Message-ID: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> You could put all of your data onto SSDs in a RAID1 configuration so that you don?t have insane read-modify-write penalties on writes (RAID1) and avoid horrible seek thrashing that spinning rust requires (SSD random access medium) for your 4K I/O operations. One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not to touch it?. He gave many reasons why mmap operations on a distributed file system is incredibly hard and not recommended. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sven Oehme Sent: Friday, January 12, 2018 8:45 AM To: coetzee.ray at gmail.com; gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmap performance against Spectrum Scale Note: External Email ________________________________ what version of Scale are you using right now ? On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee > wrote: I'd like to ask the group of their experiences in improving the performance of applications that use mmap calls against files on Spectrum Scale. Besides using an NFS export from CES instead of a native GPFS mount, or precaching the dataset into the pagepool, what other approaches are there to offset the performance hit of the 4K IO size? Kind regards Ray Coetzee _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From coetzee.ray at gmail.com Fri Jan 12 20:51:13 2018 From: coetzee.ray at gmail.com (Ray Coetzee) Date: Fri, 12 Jan 2018 20:51:13 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. (Without the meltdown patch) Hey Bryan, I remember that quote from Yuri, that's why I hoped some "magic" may have been done since then. Other attempts to improve performance I've tried include: - Using LROC to have a larger chance of a cache hit (Unfortunately the entire dataset is multiple TB) - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for this purpose (Job runs halved in duration but nowhere near what NFS can give) - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and some others with little improvement. For those interested, as a performance comparison. The same job when run on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe scratch filesystem and over 60min on spindle based filesystem. Kind regards Ray Coetzee Email: coetzee.ray at gmail.com On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister wrote: > You could put all of your data onto SSDs in a RAID1 configuration so that > you don?t have insane read-modify-write penalties on writes (RAID1) and > avoid horrible seek thrashing that spinning rust requires (SSD random > access medium) for your 4K I/O operations. > > > > One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not > to touch it?. He gave many reasons why mmap operations on a distributed > file system is incredibly hard and not recommended. > > -Bryan > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org] *On Behalf Of *Sven Oehme > *Sent:* Friday, January 12, 2018 8:45 AM > *To:* coetzee.ray at gmail.com; gpfsug main discussion list < > gpfsug-discuss at spectrumscale.org> > *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale > > > > *Note: External Email* > ------------------------------ > > what version of Scale are you using right now ? > > > > On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee wrote: > > I'd like to ask the group of their experiences in improving the > performance of applications that use mmap calls against files on Spectrum > Scale. > > > > Besides using an NFS export from CES instead of a native GPFS mount, or > precaching the dataset into the pagepool, what other approaches are there > to offset the performance hit of the 4K IO size? > > > > Kind regards > > Ray Coetzee > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at calicolabs.com Fri Jan 12 20:56:00 2018 From: alex at calicolabs.com (Alex Chekholko) Date: Fri, 12 Jan 2018 12:56:00 -0800 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: Hi Ray, Sounds like you are better off hosting that workflow on the other storage system. Another thing I have done in the past is work with the software maintainer to replace all mmap() with lseek(). YMMV. I have previously told users that mmap "doesn't work" on GPFS. As I understand it, programmers use mmap to try to improve performance but here it has the opposite effect. Regards, Alex On Fri, Jan 12, 2018 at 12:51 PM, Ray Coetzee wrote: > Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. > (Without the meltdown patch) > > Hey Bryan, I remember that quote from Yuri, that's why I hoped some > "magic" may have been done since then. > > Other attempts to improve performance I've tried include: > > - Using LROC to have a larger chance of a cache hit (Unfortunately the > entire dataset is multiple TB) > - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for > this purpose (Job runs halved in duration but nowhere near what NFS can > give) > - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and > some others with little improvement. > > For those interested, as a performance comparison. The same job when run > on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe > scratch filesystem and over 60min on spindle based filesystem. > > Kind regards > > Ray Coetzee > Email: coetzee.ray at gmail.com > > > On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister > wrote: > >> You could put all of your data onto SSDs in a RAID1 configuration so that >> you don?t have insane read-modify-write penalties on writes (RAID1) and >> avoid horrible seek thrashing that spinning rust requires (SSD random >> access medium) for your 4K I/O operations. >> >> >> >> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not >> to touch it?. He gave many reasons why mmap operations on a distributed >> file system is incredibly hard and not recommended. >> >> -Bryan >> >> >> >> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: >> gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >> *Sent:* Friday, January 12, 2018 8:45 AM >> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >> gpfsug-discuss at spectrumscale.org> >> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >> >> >> >> *Note: External Email* >> ------------------------------ >> >> what version of Scale are you using right now ? >> >> >> >> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >> wrote: >> >> I'd like to ask the group of their experiences in improving the >> performance of applications that use mmap calls against files on Spectrum >> Scale. >> >> >> >> Besides using an NFS export from CES instead of a native GPFS mount, or >> precaching the dataset into the pagepool, what other approaches are there >> to offset the performance hit of the 4K IO size? >> >> >> >> Kind regards >> >> Ray Coetzee >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ------------------------------ >> >> Note: This email is for the confidential use of the named addressee(s) >> only and may contain proprietary, confidential or privileged information. >> If you are not the intended recipient, you are hereby notified that any >> review, dissemination or copying of this email is strictly prohibited, and >> to please notify the sender immediately and destroy this email and any >> attachments. Email transmission cannot be guaranteed to be secure or >> error-free. The Company, therefore, does not make any guarantees as to the >> completeness or accuracy of this email or any attachments. This email is >> for informational purposes only and does not constitute a recommendation, >> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >> or perform any type of transaction of a financial product. >> > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Jan 12 20:57:24 2018 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 12 Jan 2018 20:57:24 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: is this primary read or write ? On Fri, Jan 12, 2018, 12:51 PM Ray Coetzee wrote: > Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. > (Without the meltdown patch) > > Hey Bryan, I remember that quote from Yuri, that's why I hoped some > "magic" may have been done since then. > > Other attempts to improve performance I've tried include: > > - Using LROC to have a larger chance of a cache hit (Unfortunately the > entire dataset is multiple TB) > - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for > this purpose (Job runs halved in duration but nowhere near what NFS can > give) > - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and > some others with little improvement. > > For those interested, as a performance comparison. The same job when run > on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe > scratch filesystem and over 60min on spindle based filesystem. > > Kind regards > > Ray Coetzee > Email: coetzee.ray at gmail.com > > > On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister > wrote: > >> You could put all of your data onto SSDs in a RAID1 configuration so that >> you don?t have insane read-modify-write penalties on writes (RAID1) and >> avoid horrible seek thrashing that spinning rust requires (SSD random >> access medium) for your 4K I/O operations. >> >> >> >> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not >> to touch it?. He gave many reasons why mmap operations on a distributed >> file system is incredibly hard and not recommended. >> >> -Bryan >> >> >> >> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: >> gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >> *Sent:* Friday, January 12, 2018 8:45 AM >> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >> gpfsug-discuss at spectrumscale.org> >> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >> >> >> >> *Note: External Email* >> ------------------------------ >> >> what version of Scale are you using right now ? >> >> >> >> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >> wrote: >> >> I'd like to ask the group of their experiences in improving the >> performance of applications that use mmap calls against files on Spectrum >> Scale. >> >> >> >> Besides using an NFS export from CES instead of a native GPFS mount, or >> precaching the dataset into the pagepool, what other approaches are there >> to offset the performance hit of the 4K IO size? >> >> >> >> Kind regards >> >> Ray Coetzee >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ------------------------------ >> >> Note: This email is for the confidential use of the named addressee(s) >> only and may contain proprietary, confidential or privileged information. >> If you are not the intended recipient, you are hereby notified that any >> review, dissemination or copying of this email is strictly prohibited, and >> to please notify the sender immediately and destroy this email and any >> attachments. Email transmission cannot be guaranteed to be secure or >> error-free. The Company, therefore, does not make any guarantees as to the >> completeness or accuracy of this email or any attachments. This email is >> for informational purposes only and does not constitute a recommendation, >> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >> or perform any type of transaction of a financial product. >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jan 12 22:58:39 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 12 Jan 2018 19:58:39 -0300 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: Having multiple blocksizes in the same file system would unnecessarily complicate things. Consider migrating a file from one pool to another with different blocksizes... How to represent the indirect blocks (lists of blocks allocated to the file)? Especially consider that today, migration can proceed one block at a time, during migration a file is "mis-placed" -- has blocks spread over more than one pool.... The new feature that supports more than 32 sub-blocks per block - is a step in another direction but maybe addresses some of your concerns.... We do support different blocksizes for meta-data -- but meta-data is distinct from data and never migrates out of system pool. --marc K. From: Aaron Knister To: Date: 01/08/2018 09:25 PM Subject: Re: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) Sent by: gpfsug-discuss-bounces at spectrumscale.org Thanks, Bryan! That's a great use case I hadn't thought of. GPFS can already support a different block size for the system pool so in my very simplistic view of the world it's already possible (unless there's some implementation detail about the system pool that lends itself to a different block size from all other pools that doesn't apply to other non-system pools differing from each other). -Aaron On 1/8/18 6:48 PM, Bryan Banister wrote: > Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. > > I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). > > NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. > > Cheers, > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister > Sent: Monday, January 08, 2018 4:57 PM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) > > Note: External Email > ------------------------------------------------- > > I was thinking some more about the >32 subblock feature in scale 5.0. As > mentioned by IBM the biggest advantage of that feature is on filesystems > with large blocks (e.g. multiple MB). The majority of our filesystems > have a block size of 1MB which got me thinking... wouldn't it be nice if > they had a larger block size (there seem to be compelling performance > reasons for large file I/O to do this). > > I'm wondering, what's the feasibility is of supporting filesystem pools > of varying block sizes within a single filesystem? I thought allocation > maps exist on a per-pool basis which gives me some hope it's not too hard. > > If one could do this then, yes, you'd still need new hardware to migrate > to a larger block size (and >32 subblocks), but it could be done as part > of a refresh cycle *and* (and this is the part most important to me) it > could be driven entirely by the policy engine which means storage admins > are largely hands off and the migration is by and large transparent to > the end user. > > This doesn't really address the need for a tool to address a filesystem > migration to 4k metadata blocks (although I wonder if something could be > done to create a system_4k pool that contains 4k-aligned metadata NSDs > where key data structures get re-written during a restripe in a > 4k-aligned manner, but that's really grasping at straws for me). > > -Aaorn > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=zfQZ_ymVgGc2EseA0szLiRxH-FgYnw7qMdx2qKo3Zes&s=2KsLTkZ-3MRyIMQhp8WwTn524NpfiKv8gwTy4P36xX4&e= > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=zfQZ_ymVgGc2EseA0szLiRxH-FgYnw7qMdx2qKo3Zes&s=2KsLTkZ-3MRyIMQhp8WwTn524NpfiKv8gwTy4P36xX4&e= > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=zfQZ_ymVgGc2EseA0szLiRxH-FgYnw7qMdx2qKo3Zes&s=2KsLTkZ-3MRyIMQhp8WwTn524NpfiKv8gwTy4P36xX4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Jan 12 23:14:52 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 12 Jan 2018 18:14:52 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm Message-ID: Apologies if this has been covered elsewhere (I couldn't find it if it has). I'm curious how GPFS decides where to allocate new blocks. We've got a filesystem that we added some NSDs to a while back and it hurt there for a little while because it appeared as though GPFS was choosing to allocate new blocks much more frequently on the ~100% free LUNs than the existing LUNs (I can't recall how free they were at the time). Looking at it now, though, it seems GPFS is doing the opposite. There's now a ~10% difference between the LUNs added and the existing LUNs (20% free vs 30% free) and GPFS is choosing to allocate new writes at a ratio of about 3:1 on the disks with *fewer* free blocks than on the disks with more free blocks. That's completely inconsistent with what we saw when we initially added the disks which makes me wonder how GPFS is choosing to allocate new blocks (other than the obvious bits about failure group, and replication factor). Could someone explain (or point me at a whitepaper) what factors GPFS uses when allocating blocks, particularly as it pertains to choosing one NSD over another within the same failure group. Thanks! -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From janfrode at tanso.net Sat Jan 13 09:24:54 2018 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sat, 13 Jan 2018 09:24:54 +0000 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: Message-ID: Don?t have documentation/whitepaper, but as I recall, it will first allocate round-robin over failureGroup, then round-robin over nsdServers, and then round-robin over volumes. So if these new NSDs are defined as different failureGroup from the old disks, that might explain it.. -jf l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister : > Apologies if this has been covered elsewhere (I couldn't find it if it > has). I'm curious how GPFS decides where to allocate new blocks. > > We've got a filesystem that we added some NSDs to a while back and it > hurt there for a little while because it appeared as though GPFS was > choosing to allocate new blocks much more frequently on the ~100% free > LUNs than the existing LUNs (I can't recall how free they were at the > time). Looking at it now, though, it seems GPFS is doing the opposite. > There's now a ~10% difference between the LUNs added and the existing > LUNs (20% free vs 30% free) and GPFS is choosing to allocate new writes > at a ratio of about 3:1 on the disks with *fewer* free blocks than on > the disks with more free blocks. That's completely inconsistent with > what we saw when we initially added the disks which makes me wonder how > GPFS is choosing to allocate new blocks (other than the obvious bits > about failure group, and replication factor). Could someone explain (or > point me at a whitepaper) what factors GPFS uses when allocating blocks, > particularly as it pertains to choosing one NSD over another within the > same failure group. > > Thanks! > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Sat Jan 13 13:18:36 2018 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Sat, 13 Jan 2018 13:18:36 +0000 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Jan 13 16:18:25 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 13 Jan 2018 11:18:25 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: Message-ID: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> Thanks Everyone! I whipped up a script to dump the block layout of a file and then join that with mmdf information. As part of my exploration I wrote one 2GB file to each of this particular filesystem's 4 data pools last night (using "touch $file; mmchattr $file -P $pool; dd of=$file") and have attached a dump of the layout/nsd information for each file/pool. The fields for the output are: diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, freePct, freeKbFragments, freeKbFragmentsPct Here's the highlight from pool1: 36 264 d13_06_006 23437934592 1213 4548935680 (19%) 83304320 (0%) 59 74 d10_41_025 23437934592 1011 6993759232 (30%) 58642816 (0%) For this file (And anecdotally what I've seen looking at NSD I/O data for other files written to this pool) the pattern of more blocks being allocated to the NSDs that are ~20% free vs the NSDs that are 30% free seems to be fairly consistent. Looking at a snippet of pool2: 101 238 d15_15_011 23437934592 1415 2008394752 (9%) 181699328 (1%) 102 235 d15_16_012 23437934592 1415 2009153536 (9%) 182165312 (1%) 116 248 d11_42_026 23437934592 1011 4146111488 (18%) 134941504 (1%) 117 249 d13_42_026 23437934592 1213 4147710976 (18%) 135203776 (1%) there are slightly more blocks allocated in general on the NSDs with twice the amount of free space, but it doesn't seem to be a significant amount relative to the delta in free space. The pattern from pool1 certainly doesn't hold true here. Pool4 isn't very interesting because all of the NSDs are well balanced in terms of free space (all 16% free). Pool3, however, *is* particularly interesting. Here's a snippet: 106 222 d15_24_016 23437934592 1415 2957561856 (13%) 37436768 (0%) 107 222 d15_25_017 23437934592 1415 2957537280 (13%) 37353984 (0%) 108 222 d15_26_018 23437934592 1415 2957539328 (13%) 37335872 (0%) 125 222 d11_44_028 23437934592 1011 13297235968 (57%) 20505568 (0%) 126 222 d12_44_028 23437934592 1213 13296712704 (57%) 20632768 (0%) 127 222 d12_45_029 23437934592 1213 13297404928 (57%) 20557408 (0%) GPFS consistently allocated the same number of blocks to disks with ~4x the free space than it did the other disks in the pool. Suffice it to say-- I'm *very* confused :) -Aaron On 1/13/18 8:18 AM, Daniel Kidger wrote: > Aaron, > ? > Also are your new NSDs the same size as your existing ones? > i.e. the NSDs that are at a?higher %age full might have more free blocks > than the other NSDs? > Daniel > > ? > IBM Storage Professional Badge > > ? > ? > *Dr Daniel Kidger* > IBM?Technical Sales Specialist > Software Defined Solution Sales > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > ? > ? > ? > > ----- Original message ----- > From: Jan-Frode Myklebust > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: Re: [gpfsug-discuss] pool block allocation algorithm > Date: Sat, Jan 13, 2018 9:25 AM > ? > Don?t have documentation/whitepaper, but as I recall, it will first > allocate round-robin over failureGroup, then round-robin over > nsdServers, and then round-robin over volumes. So if these new NSDs > are defined as different failureGroup from the old disks, that might > explain it.. > > > -jf > l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister > >: > > Apologies if this has been covered elsewhere (I couldn't find it > if it > has). I'm curious how GPFS decides where to allocate new blocks. > > We've got a filesystem that we added some NSDs to a while back > and it > hurt there for a little while because it appeared as though GPFS was > choosing to allocate new blocks much more frequently on the > ~100% free > LUNs than the existing LUNs (I can't recall how free they were > at the > time). Looking at it now, though, it seems GPFS is doing the > opposite. > There's now a ~10% difference between the LUNs added and the > existing > LUNs (20% free vs 30% free) and GPFS is choosing to allocate new > writes > at a ratio of about 3:1 on the disks with *fewer* free blocks > than on > the disks with more free blocks. That's completely inconsistent with > what we saw when we initially added the disks which makes me > wonder how > GPFS is choosing to allocate new blocks (other than the obvious bits > about failure group, and replication factor). Could someone > explain (or > point me at a whitepaper) what factors GPFS uses when allocating > blocks, > particularly as it pertains to choosing one NSD over another > within the > same failure group. > > Thanks! > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= > > ? > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- 1 221 d12_01_001 23437934592 1213 4550865920 (19%) 83734976 (0%) 2 266 d12_02_002 23437934592 1213 4550282240 (19%) 83712000 (0%) 3 265 d12_03_003 23437934592 1213 4550419456 (19%) 83986432 (0%) 4 265 d12_04_004 23437934592 1213 4550207488 (19%) 83382624 (0%) 5 266 d12_05_005 23437934592 1213 4551135232 (19%) 83820832 (0%) 6 264 d12_06_006 23437934592 1213 4549501952 (19%) 83537984 (0%) 31 256 d13_01_001 23437934592 1213 4550247424 (19%) 83557184 (0%) 32 266 d13_02_002 23437934592 1213 4548240384 (19%) 83289888 (0%) 33 264 d13_03_003 23437934592 1213 4548679680 (19%) 83851360 (0%) 34 266 d13_04_004 23437934592 1213 4551052288 (19%) 83524320 (0%) 35 265 d13_05_005 23437934592 1213 4550079488 (19%) 83578624 (0%) 36 264 d13_06_006 23437934592 1213 4548935680 (19%) 83304320 (0%) 59 74 d10_41_025 23437934592 1011 6993759232 (30%) 58642816 (0%) 61 216 d14_01_001 23437934592 1415 4575688704 (20%) 83600032 (0%) 62 266 d14_02_002 23437934592 1415 4574487552 (20%) 83376960 (0%) 63 265 d14_03_003 23437934592 1415 4574819328 (20%) 83378240 (0%) 64 266 d14_04_004 23437934592 1415 4575392768 (20%) 83162080 (0%) 65 264 d14_05_005 23437934592 1415 4576001024 (20%) 83447968 (0%) 66 265 d14_06_006 23437934592 1415 4575699968 (20%) 83043040 (0%) 85 72 d11_41_025 23437934592 1011 6991225856 (30%) 58576768 (0%) 86 66 d12_41_025 23437934592 1213 6992339968 (30%) 58402688 (0%) 87 71 d12_42_026 23437934592 1213 6992146432 (30%) 58284032 (0%) 88 67 d13_41_025 23437934592 1213 6993167360 (30%) 58134624 (0%) 89 75 d14_41_025 23437934592 1415 6992531456 (30%) 58169600 (0%) 90 71 d14_42_026 23437934592 1415 6993435648 (30%) 58234720 (0%) 91 265 d15_01_001 23437934592 1415 4575676416 (20%) 83394144 (0%) 92 264 d15_02_002 23437934592 1415 4576221184 (20%) 82999168 (0%) 93 264 d15_03_003 23437934592 1415 4576231424 (20%) 83179680 (0%) 94 266 d15_04_004 23437934592 1415 4577104896 (20%) 83445088 (0%) 95 265 d15_05_005 23437934592 1415 4576627712 (20%) 82928064 (0%) 96 259 d15_06_006 23437934592 1415 4576740352 (20%) 83355776 (0%) 115 73 d15_41_025 23437934592 1415 6990783488 (30%) 58595296 (0%) 145 266 d10_01_001 23437934592 1011 4588489728 (20%) 83100448 (0%) 146 266 d10_02_002 23437934592 1011 4587276288 (20%) 83239488 (0%) 147 265 d10_03_003 23437934592 1011 4586872832 (20%) 83024000 (0%) 148 264 d10_04_004 23437934592 1011 4585822208 (20%) 82897920 (0%) 149 264 d10_05_005 23437934592 1011 4586415104 (20%) 83277056 (0%) 150 266 d10_06_006 23437934592 1011 4587424768 (20%) 83076544 (0%) 151 265 d11_01_001 23437934592 1011 4586209280 (20%) 83178368 (0%) 152 265 d11_02_002 23437934592 1011 4587372544 (20%) 83091232 (0%) 153 264 d11_03_003 23437934592 1011 4587196416 (20%) 83147040 (0%) 154 264 d11_04_004 23437934592 1011 4586633216 (20%) 83049024 (0%) 155 265 d11_05_005 23437934592 1011 4586576896 (20%) 83188768 (0%) 156 264 d11_06_006 23437934592 1011 4587261952 (20%) 83393952 (0%) -------------- next part -------------- 7 238 d12_11_007 23437934592 1213 2005351424 (9%) 184092352 (1%) 8 237 d12_12_008 23437934592 1213 2005532672 (9%) 183893472 (1%) 9 243 d12_13_009 23437934592 1213 2004583424 (9%) 183000704 (1%) 10 239 d12_14_010 23437934592 1213 2004461568 (9%) 182958048 (1%) 11 228 d12_15_011 23437934592 1213 2004750336 (9%) 183160064 (1%) 12 240 d12_16_012 23437934592 1213 2008772608 (9%) 183263936 (1%) 37 228 d13_11_007 23437934592 1213 2006091776 (9%) 183041024 (1%) 38 230 d13_12_008 23437934592 1213 2003761152 (9%) 182625024 (1%) 39 229 d13_13_009 23437934592 1213 2004123648 (9%) 182907552 (1%) 40 208 d13_14_010 23437934592 1213 2006372352 (9%) 182797376 (1%) 41 233 d13_15_011 23437934592 1213 2003358720 (9%) 182322880 (1%) 42 232 d13_16_012 23437934592 1213 2004774912 (9%) 182729792 (1%) 67 234 d14_11_007 23437934592 1415 2006553600 (9%) 182736800 (1%) 68 249 d14_12_008 23437934592 1415 2010278912 (9%) 182304832 (1%) 69 226 d14_13_009 23437934592 1415 2008749056 (9%) 182854464 (1%) 70 237 d14_14_010 23437934592 1415 2009348096 (9%) 182257024 (1%) 71 249 d14_15_011 23437934592 1415 2008017920 (9%) 182217152 (1%) 72 233 d14_16_012 23437934592 1415 2008158208 (9%) 182239680 (1%) 97 237 d15_11_007 23437934592 1415 2007257088 (9%) 181898304 (1%) 98 233 d15_12_008 23437934592 1415 2008309760 (9%) 182000928 (1%) 99 229 d15_13_009 23437934592 1415 2008755200 (9%) 182380544 (1%) 100 224 d15_14_010 23437934592 1415 2008460288 (9%) 181691616 (1%) 101 238 d15_15_011 23437934592 1415 2008394752 (9%) 181699328 (1%) 102 235 d15_16_012 23437934592 1415 2009153536 (9%) 182165312 (1%) 116 248 d11_42_026 23437934592 1011 4146111488 (18%) 134941504 (1%) 117 249 d13_42_026 23437934592 1213 4147710976 (18%) 135203776 (1%) 118 248 d15_42_026 23437934592 1415 4147094528 (18%) 134748320 (1%) 119 249 d10_43_027 23437934592 1011 4148652032 (18%) 135124288 (1%) 120 247 d11_43_027 23437934592 1011 4147335168 (18%) 134808256 (1%) 121 250 d12_43_027 23437934592 1213 4147177472 (18%) 134812096 (1%) 122 248 d13_43_027 23437934592 1213 4147270656 (18%) 134516192 (1%) 123 248 d14_43_027 23437934592 1415 4148153344 (18%) 134928896 (1%) 124 248 d15_43_027 23437934592 1415 4149500928 (18%) 134818880 (1%) 157 247 d10_11_007 23437934592 1011 2008516608 (9%) 182482592 (1%) 159 248 d10_13_009 23437934592 1011 2009745408 (9%) 182393184 (1%) 161 233 d10_15_011 23437934592 1011 2007539712 (9%) 182169856 (1%) 163 234 d11_11_007 23437934592 1011 2009283584 (9%) 182173824 (1%) 164 244 d11_12_008 23437934592 1011 2009091072 (9%) 182099200 (1%) 165 248 d11_13_009 23437934592 1011 2008827904 (9%) 182029632 (1%) 166 231 d11_14_010 23437934592 1011 2010264576 (9%) 181675424 (1%) 167 235 d11_15_011 23437934592 1011 2010696704 (9%) 181854304 (1%) 168 236 d11_16_012 23437934592 1011 2008183808 (9%) 182236800 (1%) -------------- next part -------------- 13 222 d12_21_013 23437934592 1213 2957768704 (13%) 37317824 (0%) 14 223 d12_22_014 23437934592 1213 2957680640 (13%) 37608576 (0%) 15 223 d12_23_015 23437934592 1213 2958575616 (13%) 37380672 (0%) 16 223 d12_24_016 23437934592 1213 2957406208 (13%) 37403648 (0%) 17 223 d12_25_017 23437934592 1213 2957883392 (13%) 37558016 (0%) 18 222 d12_26_018 23437934592 1213 2957815808 (13%) 37556416 (0%) 43 222 d13_21_013 23437934592 1213 2957149184 (13%) 37745280 (0%) 44 222 d13_22_014 23437934592 1213 2957812736 (13%) 37520736 (0%) 45 222 d13_23_015 23437934592 1213 2957990912 (13%) 37341952 (0%) 46 222 d13_24_016 23437934592 1213 2958204928 (13%) 37477536 (0%) 47 222 d13_25_017 23437934592 1213 2958315520 (13%) 37507072 (0%) 48 222 d13_26_018 23437934592 1213 2958086144 (13%) 37407648 (0%) 73 222 d14_21_013 23437934592 1415 2957812736 (13%) 37631488 (0%) 74 223 d14_22_014 23437934592 1415 2959673344 (13%) 37427680 (0%) 75 223 d14_23_015 23437934592 1415 2957627392 (13%) 37400992 (0%) 76 223 d14_24_016 23437934592 1415 2957408256 (13%) 37235776 (0%) 77 222 d14_25_017 23437934592 1415 2957407232 (13%) 37499200 (0%) 78 222 d14_26_018 23437934592 1415 2956661760 (13%) 37633696 (0%) 103 222 d15_21_013 23437934592 1415 2957427712 (13%) 37629664 (0%) 104 222 d15_22_014 23437934592 1415 2957506560 (13%) 37571968 (0%) 105 222 d15_23_015 23437934592 1415 2958114816 (13%) 37422400 (0%) 106 222 d15_24_016 23437934592 1415 2957561856 (13%) 37436768 (0%) 107 222 d15_25_017 23437934592 1415 2957537280 (13%) 37353984 (0%) 108 222 d15_26_018 23437934592 1415 2957539328 (13%) 37335872 (0%) 125 222 d11_44_028 23437934592 1011 13297235968 (57%) 20505568 (0%) 126 222 d12_44_028 23437934592 1213 13296712704 (57%) 20632768 (0%) 127 222 d12_45_029 23437934592 1213 13297404928 (57%) 20557408 (0%) 128 222 d13_44_028 23437934592 1213 13297271808 (57%) 20502304 (0%) 129 222 d14_44_028 23437934592 1415 13294722048 (57%) 20569824 (0%) 130 222 d14_45_029 23437934592 1415 13296071680 (57%) 20667104 (0%) 131 222 d15_44_028 23437934592 1415 13297007616 (57%) 20429536 (0%) 132 223 d10_44_028 23437934592 1011 13309036544 (57%) 20299552 (0%) 133 223 d10_45_029 23437934592 1011 13309213696 (57%) 20464576 (0%) 169 223 d10_21_013 23437934592 1011 2797943808 (12%) 36881600 (0%) 170 222 d10_22_014 23437934592 1011 2798030848 (12%) 37152256 (0%) 171 222 d10_23_015 23437934592 1011 2798152704 (12%) 36959840 (0%) 172 222 d10_24_016 23437934592 1011 2798002176 (12%) 36954976 (0%) 173 222 d10_25_017 23437934592 1011 2798524416 (12%) 36737120 (0%) 174 222 d10_26_018 23437934592 1011 2797768704 (12%) 36888096 (0%) 175 222 d11_21_013 23437934592 1011 2958944256 (13%) 37504288 (0%) 176 222 d11_22_014 23437934592 1011 2957225984 (13%) 37575648 (0%) 177 222 d11_23_015 23437934592 1011 2958100480 (13%) 37546336 (0%) 178 222 d11_24_016 23437934592 1011 2958453760 (13%) 37578784 (0%) 179 222 d11_25_017 23437934592 1011 2958053376 (13%) 37331808 (0%) 180 222 d11_26_018 23437934592 1011 2958290944 (13%) 37445824 (0%) -------------- next part -------------- 19 223 d12_31_019 23437934592 1213 3803873280 (16%) 117281408 (1%) 20 222 d12_32_020 23437934592 1213 3779237888 (16%) 116652576 (0%) 21 222 d12_33_021 23437934592 1213 3779206144 (16%) 116820288 (0%) 22 222 d12_34_022 23437934592 1213 3776828416 (16%) 116108032 (0%) 23 222 d12_35_023 23437934592 1213 3776950272 (16%) 116238752 (0%) 24 222 d12_36_024 23437934592 1213 3776524288 (16%) 116599744 (0%) 49 222 d13_31_019 23437934592 1213 3778293760 (16%) 116365600 (0%) 50 222 d13_32_020 23437934592 1213 3778381824 (16%) 115960448 (0%) 51 222 d13_33_021 23437934592 1213 3777104896 (16%) 116225984 (0%) 52 222 d13_34_022 23437934592 1213 3775632384 (16%) 116496576 (0%) 53 222 d13_35_023 23437934592 1213 3776899072 (16%) 116397120 (0%) 54 222 d13_36_024 23437934592 1213 3776140288 (16%) 116400000 (0%) 79 222 d14_31_019 23437934592 1415 3844638720 (16%) 113531904 (0%) 80 222 d14_32_020 23437934592 1415 3817139200 (16%) 113600928 (0%) 81 222 d14_33_021 23437934592 1415 3815447552 (16%) 113358272 (0%) 82 222 d14_34_022 23437934592 1415 3814561792 (16%) 113348160 (0%) 83 222 d14_35_023 23437934592 1415 3815387136 (16%) 113582432 (0%) 84 222 d14_36_024 23437934592 1415 3815303168 (16%) 112914816 (0%) 109 222 d15_31_019 23437934592 1415 3814614016 (16%) 113298240 (0%) 110 222 d15_32_020 23437934592 1415 3814936576 (16%) 113053664 (0%) 111 222 d15_33_021 23437934592 1415 3815122944 (16%) 113438656 (0%) 112 222 d15_34_022 23437934592 1415 3813474304 (16%) 113481216 (0%) 113 222 d15_35_023 23437934592 1415 3812912128 (16%) 113618624 (0%) 114 222 d15_36_024 23437934592 1415 3813067776 (16%) 113282848 (0%) 134 223 d11_45_029 23437934592 1011 3683404800 (16%) 100860000 (0%) 135 222 d13_45_029 23437934592 1213 3682211840 (16%) 101101824 (0%) 136 223 d15_45_029 23437934592 1415 3682046976 (16%) 101375328 (0%) 137 222 d10_46_030 23437934592 1011 3684398080 (16%) 100946912 (0%) 138 222 d11_46_030 23437934592 1011 3683755008 (16%) 101295200 (0%) 139 223 d12_46_030 23437934592 1213 3682649088 (16%) 100540832 (0%) 140 223 d13_46_030 23437934592 1213 3684602880 (16%) 100489376 (0%) 141 223 d14_46_030 23437934592 1415 3681666048 (16%) 100854112 (0%) 142 223 d15_46_030 23437934592 1415 3685487616 (16%) 100715744 (0%) 181 222 d10_31_019 23437934592 1011 3816681472 (16%) 113389344 (0%) 182 222 d10_32_020 23437934592 1011 3816535040 (16%) 113652672 (0%) 183 222 d10_33_021 23437934592 1011 3817772032 (16%) 113124224 (0%) 184 222 d10_34_022 23437934592 1011 3817069568 (16%) 113225504 (0%) 185 222 d10_35_023 23437934592 1011 3816517632 (16%) 113163488 (0%) 186 222 d10_36_024 23437934592 1011 3815605248 (16%) 113475648 (0%) 187 222 d11_31_019 23437934592 1011 3815664640 (16%) 113100992 (0%) 188 222 d11_32_020 23437934592 1011 3815985152 (16%) 113325504 (0%) 189 222 d11_33_021 23437934592 1011 3815148544 (16%) 113328480 (0%) 190 223 d11_34_022 23437934592 1011 3814780928 (16%) 113457440 (0%) 191 223 d11_35_023 23437934592 1011 3815219200 (16%) 113065504 (0%) 192 223 d11_36_024 23437934592 1011 3815383040 (16%) 113170592 (0%) From aaron.s.knister at nasa.gov Sat Jan 13 16:56:52 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 13 Jan 2018 11:56:52 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> Message-ID: Sorry, I didn't explicitly say it, but the output I sent should answer both Daniel's and Jan-Frode's questions. In short, though, the new NSDs were added to existing failure groups and they should all be (except in one or two cases where we re-formatted the LUN and the size changed slightly) the same size. -Aaron On 1/13/18 11:18 AM, Aaron Knister wrote: > Thanks Everyone! I whipped up a script to dump the block layout of a > file and then join that with mmdf information. As part of my exploration > I wrote one 2GB file to each of this particular filesystem's 4 data > pools last night (using "touch $file; mmchattr $file -P $pool; dd > of=$file") and have attached a dump of the layout/nsd information for > each file/pool. The fields for the output are: > > diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, > freePct, freeKbFragments, freeKbFragmentsPct > > > Here's the highlight from pool1: > > 36 264 d13_06_006 23437934592 1213 4548935680 (19%) > 83304320 (0%) > 59 74 d10_41_025 23437934592 1011 6993759232 (30%) > 58642816 (0%) > > For this file (And anecdotally what I've seen looking at NSD I/O data > for other files written to this pool) the pattern of more blocks being > allocated to the NSDs that are ~20% free vs the NSDs that are 30% free > seems to be fairly consistent. > > > Looking at a snippet of pool2: > 101 238 d15_15_011 23437934592 1415 2008394752 (9%) > 181699328 (1%) > 102 235 d15_16_012 23437934592 1415 2009153536 (9%) > 182165312 (1%) > 116 248 d11_42_026 23437934592 1011 4146111488 (18%) > 134941504 (1%) > 117 249 d13_42_026 23437934592 1213 4147710976 (18%) > 135203776 (1%) > > there are slightly more blocks allocated in general on the NSDs with > twice the amount of free space, but it doesn't seem to be a significant > amount relative to the delta in free space. The pattern from pool1 > certainly doesn't hold true here. > > Pool4 isn't very interesting because all of the NSDs are well balanced > in terms of free space (all 16% free). > > Pool3, however, *is* particularly interesting. Here's a snippet: > > 106 222 d15_24_016 23437934592 1415 2957561856 (13%) > 37436768 (0%) > 107 222 d15_25_017 23437934592 1415 2957537280 (13%) > 37353984 (0%) > 108 222 d15_26_018 23437934592 1415 2957539328 (13%) > 37335872 (0%) > 125 222 d11_44_028 23437934592 1011 13297235968 (57%) > 20505568 (0%) > 126 222 d12_44_028 23437934592 1213 13296712704 (57%) > 20632768 (0%) > 127 222 d12_45_029 23437934592 1213 13297404928 (57%) > 20557408 (0%) > > GPFS consistently allocated the same number of blocks to disks with ~4x > the free space than it did the other disks in the pool. > > Suffice it to say-- I'm *very* confused :) > > -Aaron > > On 1/13/18 8:18 AM, Daniel Kidger wrote: >> Aaron, >> ? >> Also are your new NSDs the same size as your existing ones? >> i.e. the NSDs that are at a?higher %age full might have more free blocks >> than the other NSDs? >> Daniel >> >> ? >> IBM Storage Professional Badge >> >> ? >> ? >> *Dr Daniel Kidger* >> IBM?Technical Sales Specialist >> Software Defined Solution Sales >> >> +44-(0)7818 522 266 >> daniel.kidger at uk.ibm.com >> >> ? >> ? >> ? >> >> ----- Original message ----- >> From: Jan-Frode Myklebust >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug main discussion list >> Cc: >> Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> Date: Sat, Jan 13, 2018 9:25 AM >> ? >> Don?t have documentation/whitepaper, but as I recall, it will first >> allocate round-robin over failureGroup, then round-robin over >> nsdServers, and then round-robin over volumes. So if these new NSDs >> are defined as different failureGroup from the old disks, that might >> explain it.. >> >> >> -jf >> l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >> >: >> >> Apologies if this has been covered elsewhere (I couldn't find it >> if it >> has). I'm curious how GPFS decides where to allocate new blocks. >> >> We've got a filesystem that we added some NSDs to a while back >> and it >> hurt there for a little while because it appeared as though GPFS was >> choosing to allocate new blocks much more frequently on the >> ~100% free >> LUNs than the existing LUNs (I can't recall how free they were >> at the >> time). Looking at it now, though, it seems GPFS is doing the >> opposite. >> There's now a ~10% difference between the LUNs added and the >> existing >> LUNs (20% free vs 30% free) and GPFS is choosing to allocate new >> writes >> at a ratio of about 3:1 on the disks with *fewer* free blocks >> than on >> the disks with more free blocks. That's completely inconsistent with >> what we saw when we initially added the disks which makes me >> wonder how >> GPFS is choosing to allocate new blocks (other than the obvious bits >> about failure group, and replication factor). Could someone >> explain (or >> point me at a whitepaper) what factors GPFS uses when allocating >> blocks, >> particularly as it pertains to choosing one NSD over another >> within the >> same failure group. >> >> Thanks! >> >> -Aaron >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= >> >> ? >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From peserocka at gmail.com Sat Jan 13 17:00:49 2018 From: peserocka at gmail.com (Peter Serocka) Date: Sat, 13 Jan 2018 18:00:49 +0100 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> Message-ID: <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Within reasonable capacity limits it would also expect to direct incoming data to disks that are best ?available? from a current performance perspective ? like doing least IOPS, having lowest latency and shortest filled queue length. You new NSDs, filled only with recent data, might quickly have become the most busy units before reaching capacity balance, simply because recent data tends to be more active than older stuff. Makes sense? ? Peter > On 2018 Jan 13 Sat, at 17:18, Aaron Knister wrote: > > Thanks Everyone! I whipped up a script to dump the block layout of a > file and then join that with mmdf information. As part of my exploration > I wrote one 2GB file to each of this particular filesystem's 4 data > pools last night (using "touch $file; mmchattr $file -P $pool; dd > of=$file") and have attached a dump of the layout/nsd information for > each file/pool. The fields for the output are: > > diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, > freePct, freeKbFragments, freeKbFragmentsPct > > > Here's the highlight from pool1: > > 36 264 d13_06_006 23437934592 1213 4548935680 (19%) > 83304320 (0%) > 59 74 d10_41_025 23437934592 1011 6993759232 (30%) > 58642816 (0%) > > For this file (And anecdotally what I've seen looking at NSD I/O data > for other files written to this pool) the pattern of more blocks being > allocated to the NSDs that are ~20% free vs the NSDs that are 30% free > seems to be fairly consistent. > > > Looking at a snippet of pool2: > 101 238 d15_15_011 23437934592 1415 2008394752 (9%) > 181699328 (1%) > 102 235 d15_16_012 23437934592 1415 2009153536 (9%) > 182165312 (1%) > 116 248 d11_42_026 23437934592 1011 4146111488 (18%) > 134941504 (1%) > 117 249 d13_42_026 23437934592 1213 4147710976 (18%) > 135203776 (1%) > > there are slightly more blocks allocated in general on the NSDs with > twice the amount of free space, but it doesn't seem to be a significant > amount relative to the delta in free space. The pattern from pool1 > certainly doesn't hold true here. > > Pool4 isn't very interesting because all of the NSDs are well balanced > in terms of free space (all 16% free). > > Pool3, however, *is* particularly interesting. Here's a snippet: > > 106 222 d15_24_016 23437934592 1415 2957561856 (13%) > 37436768 (0%) > 107 222 d15_25_017 23437934592 1415 2957537280 (13%) > 37353984 (0%) > 108 222 d15_26_018 23437934592 1415 2957539328 (13%) > 37335872 (0%) > 125 222 d11_44_028 23437934592 1011 13297235968 (57%) > 20505568 (0%) > 126 222 d12_44_028 23437934592 1213 13296712704 (57%) > 20632768 (0%) > 127 222 d12_45_029 23437934592 1213 13297404928 (57%) > 20557408 (0%) > > GPFS consistently allocated the same number of blocks to disks with ~4x > the free space than it did the other disks in the pool. > > Suffice it to say-- I'm *very* confused :) > > -Aaron > > On 1/13/18 8:18 AM, Daniel Kidger wrote: >> Aaron, >> >> Also are your new NSDs the same size as your existing ones? >> i.e. the NSDs that are at a higher %age full might have more free blocks >> than the other NSDs? >> Daniel >> >> >> IBM Storage Professional Badge >> >> >> >> *Dr Daniel Kidger* >> IBM Technical Sales Specialist >> Software Defined Solution Sales >> >> +44-(0)7818 522 266 >> daniel.kidger at uk.ibm.com >> >> >> >> >> >> ----- Original message ----- >> From: Jan-Frode Myklebust >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug main discussion list >> Cc: >> Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> Date: Sat, Jan 13, 2018 9:25 AM >> >> Don?t have documentation/whitepaper, but as I recall, it will first >> allocate round-robin over failureGroup, then round-robin over >> nsdServers, and then round-robin over volumes. So if these new NSDs >> are defined as different failureGroup from the old disks, that might >> explain it.. >> >> >> -jf >> l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >> >: >> >> Apologies if this has been covered elsewhere (I couldn't find it >> if it >> has). I'm curious how GPFS decides where to allocate new blocks. >> >> We've got a filesystem that we added some NSDs to a while back >> and it >> hurt there for a little while because it appeared as though GPFS was >> choosing to allocate new blocks much more frequently on the >> ~100% free >> LUNs than the existing LUNs (I can't recall how free they were >> at the >> time). Looking at it now, though, it seems GPFS is doing the >> opposite. >> There's now a ~10% difference between the LUNs added and the >> existing >> LUNs (20% free vs 30% free) and GPFS is choosing to allocate new >> writes >> at a ratio of about 3:1 on the disks with *fewer* free blocks >> than on >> the disks with more free blocks. That's completely inconsistent with >> what we saw when we initially added the disks which makes me >> wonder how >> GPFS is choosing to allocate new blocks (other than the obvious bits >> about failure group, and replication factor). Could someone >> explain (or >> point me at a whitepaper) what factors GPFS uses when allocating >> blocks, >> particularly as it pertains to choosing one NSD over another >> within the >> same failure group. >> >> Thanks! >> >> -Aaron >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= >> >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 -------------- next part -------------- 1 221 d12_01_001 23437934592 1213 4550865920 (19%) 83734976 (0%) 2 266 d12_02_002 23437934592 1213 4550282240 (19%) 83712000 (0%) 3 265 d12_03_003 23437934592 1213 4550419456 (19%) 83986432 (0%) 4 265 d12_04_004 23437934592 1213 4550207488 (19%) 83382624 (0%) 5 266 d12_05_005 23437934592 1213 4551135232 (19%) 83820832 (0%) 6 264 d12_06_006 23437934592 1213 4549501952 (19%) 83537984 (0%) 31 256 d13_01_001 23437934592 1213 4550247424 (19%) 83557184 (0%) 32 266 d13_02_002 23437934592 1213 4548240384 (19%) 83289888 (0%) 33 264 d13_03_003 23437934592 1213 4548679680 (19%) 83851360 (0%) 34 266 d13_04_004 23437934592 1213 4551052288 (19%) 83524320 (0%) 35 265 d13_05_005 23437934592 1213 4550079488 (19%) 83578624 (0%) 36 264 d13_06_006 23437934592 1213 4548935680 (19%) 83304320 (0%) 59 74 d10_41_025 23437934592 1011 6993759232 (30%) 58642816 (0%) 61 216 d14_01_001 23437934592 1415 4575688704 (20%) 83600032 (0%) 62 266 d14_02_002 23437934592 1415 4574487552 (20%) 83376960 (0%) 63 265 d14_03_003 23437934592 1415 4574819328 (20%) 83378240 (0%) 64 266 d14_04_004 23437934592 1415 4575392768 (20%) 83162080 (0%) 65 264 d14_05_005 23437934592 1415 4576001024 (20%) 83447968 (0%) 66 265 d14_06_006 23437934592 1415 4575699968 (20%) 83043040 (0%) 85 72 d11_41_025 23437934592 1011 6991225856 (30%) 58576768 (0%) 86 66 d12_41_025 23437934592 1213 6992339968 (30%) 58402688 (0%) 87 71 d12_42_026 23437934592 1213 6992146432 (30%) 58284032 (0%) 88 67 d13_41_025 23437934592 1213 6993167360 (30%) 58134624 (0%) 89 75 d14_41_025 23437934592 1415 6992531456 (30%) 58169600 (0%) 90 71 d14_42_026 23437934592 1415 6993435648 (30%) 58234720 (0%) 91 265 d15_01_001 23437934592 1415 4575676416 (20%) 83394144 (0%) 92 264 d15_02_002 23437934592 1415 4576221184 (20%) 82999168 (0%) 93 264 d15_03_003 23437934592 1415 4576231424 (20%) 83179680 (0%) 94 266 d15_04_004 23437934592 1415 4577104896 (20%) 83445088 (0%) 95 265 d15_05_005 23437934592 1415 4576627712 (20%) 82928064 (0%) 96 259 d15_06_006 23437934592 1415 4576740352 (20%) 83355776 (0%) 115 73 d15_41_025 23437934592 1415 6990783488 (30%) 58595296 (0%) 145 266 d10_01_001 23437934592 1011 4588489728 (20%) 83100448 (0%) 146 266 d10_02_002 23437934592 1011 4587276288 (20%) 83239488 (0%) 147 265 d10_03_003 23437934592 1011 4586872832 (20%) 83024000 (0%) 148 264 d10_04_004 23437934592 1011 4585822208 (20%) 82897920 (0%) 149 264 d10_05_005 23437934592 1011 4586415104 (20%) 83277056 (0%) 150 266 d10_06_006 23437934592 1011 4587424768 (20%) 83076544 (0%) 151 265 d11_01_001 23437934592 1011 4586209280 (20%) 83178368 (0%) 152 265 d11_02_002 23437934592 1011 4587372544 (20%) 83091232 (0%) 153 264 d11_03_003 23437934592 1011 4587196416 (20%) 83147040 (0%) 154 264 d11_04_004 23437934592 1011 4586633216 (20%) 83049024 (0%) 155 265 d11_05_005 23437934592 1011 4586576896 (20%) 83188768 (0%) 156 264 d11_06_006 23437934592 1011 4587261952 (20%) 83393952 (0%) -------------- next part -------------- 7 238 d12_11_007 23437934592 1213 2005351424 (9%) 184092352 (1%) 8 237 d12_12_008 23437934592 1213 2005532672 (9%) 183893472 (1%) 9 243 d12_13_009 23437934592 1213 2004583424 (9%) 183000704 (1%) 10 239 d12_14_010 23437934592 1213 2004461568 (9%) 182958048 (1%) 11 228 d12_15_011 23437934592 1213 2004750336 (9%) 183160064 (1%) 12 240 d12_16_012 23437934592 1213 2008772608 (9%) 183263936 (1%) 37 228 d13_11_007 23437934592 1213 2006091776 (9%) 183041024 (1%) 38 230 d13_12_008 23437934592 1213 2003761152 (9%) 182625024 (1%) 39 229 d13_13_009 23437934592 1213 2004123648 (9%) 182907552 (1%) 40 208 d13_14_010 23437934592 1213 2006372352 (9%) 182797376 (1%) 41 233 d13_15_011 23437934592 1213 2003358720 (9%) 182322880 (1%) 42 232 d13_16_012 23437934592 1213 2004774912 (9%) 182729792 (1%) 67 234 d14_11_007 23437934592 1415 2006553600 (9%) 182736800 (1%) 68 249 d14_12_008 23437934592 1415 2010278912 (9%) 182304832 (1%) 69 226 d14_13_009 23437934592 1415 2008749056 (9%) 182854464 (1%) 70 237 d14_14_010 23437934592 1415 2009348096 (9%) 182257024 (1%) 71 249 d14_15_011 23437934592 1415 2008017920 (9%) 182217152 (1%) 72 233 d14_16_012 23437934592 1415 2008158208 (9%) 182239680 (1%) 97 237 d15_11_007 23437934592 1415 2007257088 (9%) 181898304 (1%) 98 233 d15_12_008 23437934592 1415 2008309760 (9%) 182000928 (1%) 99 229 d15_13_009 23437934592 1415 2008755200 (9%) 182380544 (1%) 100 224 d15_14_010 23437934592 1415 2008460288 (9%) 181691616 (1%) 101 238 d15_15_011 23437934592 1415 2008394752 (9%) 181699328 (1%) 102 235 d15_16_012 23437934592 1415 2009153536 (9%) 182165312 (1%) 116 248 d11_42_026 23437934592 1011 4146111488 (18%) 134941504 (1%) 117 249 d13_42_026 23437934592 1213 4147710976 (18%) 135203776 (1%) 118 248 d15_42_026 23437934592 1415 4147094528 (18%) 134748320 (1%) 119 249 d10_43_027 23437934592 1011 4148652032 (18%) 135124288 (1%) 120 247 d11_43_027 23437934592 1011 4147335168 (18%) 134808256 (1%) 121 250 d12_43_027 23437934592 1213 4147177472 (18%) 134812096 (1%) 122 248 d13_43_027 23437934592 1213 4147270656 (18%) 134516192 (1%) 123 248 d14_43_027 23437934592 1415 4148153344 (18%) 134928896 (1%) 124 248 d15_43_027 23437934592 1415 4149500928 (18%) 134818880 (1%) 157 247 d10_11_007 23437934592 1011 2008516608 (9%) 182482592 (1%) 159 248 d10_13_009 23437934592 1011 2009745408 (9%) 182393184 (1%) 161 233 d10_15_011 23437934592 1011 2007539712 (9%) 182169856 (1%) 163 234 d11_11_007 23437934592 1011 2009283584 (9%) 182173824 (1%) 164 244 d11_12_008 23437934592 1011 2009091072 (9%) 182099200 (1%) 165 248 d11_13_009 23437934592 1011 2008827904 (9%) 182029632 (1%) 166 231 d11_14_010 23437934592 1011 2010264576 (9%) 181675424 (1%) 167 235 d11_15_011 23437934592 1011 2010696704 (9%) 181854304 (1%) 168 236 d11_16_012 23437934592 1011 2008183808 (9%) 182236800 (1%) -------------- next part -------------- 13 222 d12_21_013 23437934592 1213 2957768704 (13%) 37317824 (0%) 14 223 d12_22_014 23437934592 1213 2957680640 (13%) 37608576 (0%) 15 223 d12_23_015 23437934592 1213 2958575616 (13%) 37380672 (0%) 16 223 d12_24_016 23437934592 1213 2957406208 (13%) 37403648 (0%) 17 223 d12_25_017 23437934592 1213 2957883392 (13%) 37558016 (0%) 18 222 d12_26_018 23437934592 1213 2957815808 (13%) 37556416 (0%) 43 222 d13_21_013 23437934592 1213 2957149184 (13%) 37745280 (0%) 44 222 d13_22_014 23437934592 1213 2957812736 (13%) 37520736 (0%) 45 222 d13_23_015 23437934592 1213 2957990912 (13%) 37341952 (0%) 46 222 d13_24_016 23437934592 1213 2958204928 (13%) 37477536 (0%) 47 222 d13_25_017 23437934592 1213 2958315520 (13%) 37507072 (0%) 48 222 d13_26_018 23437934592 1213 2958086144 (13%) 37407648 (0%) 73 222 d14_21_013 23437934592 1415 2957812736 (13%) 37631488 (0%) 74 223 d14_22_014 23437934592 1415 2959673344 (13%) 37427680 (0%) 75 223 d14_23_015 23437934592 1415 2957627392 (13%) 37400992 (0%) 76 223 d14_24_016 23437934592 1415 2957408256 (13%) 37235776 (0%) 77 222 d14_25_017 23437934592 1415 2957407232 (13%) 37499200 (0%) 78 222 d14_26_018 23437934592 1415 2956661760 (13%) 37633696 (0%) 103 222 d15_21_013 23437934592 1415 2957427712 (13%) 37629664 (0%) 104 222 d15_22_014 23437934592 1415 2957506560 (13%) 37571968 (0%) 105 222 d15_23_015 23437934592 1415 2958114816 (13%) 37422400 (0%) 106 222 d15_24_016 23437934592 1415 2957561856 (13%) 37436768 (0%) 107 222 d15_25_017 23437934592 1415 2957537280 (13%) 37353984 (0%) 108 222 d15_26_018 23437934592 1415 2957539328 (13%) 37335872 (0%) 125 222 d11_44_028 23437934592 1011 13297235968 (57%) 20505568 (0%) 126 222 d12_44_028 23437934592 1213 13296712704 (57%) 20632768 (0%) 127 222 d12_45_029 23437934592 1213 13297404928 (57%) 20557408 (0%) 128 222 d13_44_028 23437934592 1213 13297271808 (57%) 20502304 (0%) 129 222 d14_44_028 23437934592 1415 13294722048 (57%) 20569824 (0%) 130 222 d14_45_029 23437934592 1415 13296071680 (57%) 20667104 (0%) 131 222 d15_44_028 23437934592 1415 13297007616 (57%) 20429536 (0%) 132 223 d10_44_028 23437934592 1011 13309036544 (57%) 20299552 (0%) 133 223 d10_45_029 23437934592 1011 13309213696 (57%) 20464576 (0%) 169 223 d10_21_013 23437934592 1011 2797943808 (12%) 36881600 (0%) 170 222 d10_22_014 23437934592 1011 2798030848 (12%) 37152256 (0%) 171 222 d10_23_015 23437934592 1011 2798152704 (12%) 36959840 (0%) 172 222 d10_24_016 23437934592 1011 2798002176 (12%) 36954976 (0%) 173 222 d10_25_017 23437934592 1011 2798524416 (12%) 36737120 (0%) 174 222 d10_26_018 23437934592 1011 2797768704 (12%) 36888096 (0%) 175 222 d11_21_013 23437934592 1011 2958944256 (13%) 37504288 (0%) 176 222 d11_22_014 23437934592 1011 2957225984 (13%) 37575648 (0%) 177 222 d11_23_015 23437934592 1011 2958100480 (13%) 37546336 (0%) 178 222 d11_24_016 23437934592 1011 2958453760 (13%) 37578784 (0%) 179 222 d11_25_017 23437934592 1011 2958053376 (13%) 37331808 (0%) 180 222 d11_26_018 23437934592 1011 2958290944 (13%) 37445824 (0%) -------------- next part -------------- 19 223 d12_31_019 23437934592 1213 3803873280 (16%) 117281408 (1%) 20 222 d12_32_020 23437934592 1213 3779237888 (16%) 116652576 (0%) 21 222 d12_33_021 23437934592 1213 3779206144 (16%) 116820288 (0%) 22 222 d12_34_022 23437934592 1213 3776828416 (16%) 116108032 (0%) 23 222 d12_35_023 23437934592 1213 3776950272 (16%) 116238752 (0%) 24 222 d12_36_024 23437934592 1213 3776524288 (16%) 116599744 (0%) 49 222 d13_31_019 23437934592 1213 3778293760 (16%) 116365600 (0%) 50 222 d13_32_020 23437934592 1213 3778381824 (16%) 115960448 (0%) 51 222 d13_33_021 23437934592 1213 3777104896 (16%) 116225984 (0%) 52 222 d13_34_022 23437934592 1213 3775632384 (16%) 116496576 (0%) 53 222 d13_35_023 23437934592 1213 3776899072 (16%) 116397120 (0%) 54 222 d13_36_024 23437934592 1213 3776140288 (16%) 116400000 (0%) 79 222 d14_31_019 23437934592 1415 3844638720 (16%) 113531904 (0%) 80 222 d14_32_020 23437934592 1415 3817139200 (16%) 113600928 (0%) 81 222 d14_33_021 23437934592 1415 3815447552 (16%) 113358272 (0%) 82 222 d14_34_022 23437934592 1415 3814561792 (16%) 113348160 (0%) 83 222 d14_35_023 23437934592 1415 3815387136 (16%) 113582432 (0%) 84 222 d14_36_024 23437934592 1415 3815303168 (16%) 112914816 (0%) 109 222 d15_31_019 23437934592 1415 3814614016 (16%) 113298240 (0%) 110 222 d15_32_020 23437934592 1415 3814936576 (16%) 113053664 (0%) 111 222 d15_33_021 23437934592 1415 3815122944 (16%) 113438656 (0%) 112 222 d15_34_022 23437934592 1415 3813474304 (16%) 113481216 (0%) 113 222 d15_35_023 23437934592 1415 3812912128 (16%) 113618624 (0%) 114 222 d15_36_024 23437934592 1415 3813067776 (16%) 113282848 (0%) 134 223 d11_45_029 23437934592 1011 3683404800 (16%) 100860000 (0%) 135 222 d13_45_029 23437934592 1213 3682211840 (16%) 101101824 (0%) 136 223 d15_45_029 23437934592 1415 3682046976 (16%) 101375328 (0%) 137 222 d10_46_030 23437934592 1011 3684398080 (16%) 100946912 (0%) 138 222 d11_46_030 23437934592 1011 3683755008 (16%) 101295200 (0%) 139 223 d12_46_030 23437934592 1213 3682649088 (16%) 100540832 (0%) 140 223 d13_46_030 23437934592 1213 3684602880 (16%) 100489376 (0%) 141 223 d14_46_030 23437934592 1415 3681666048 (16%) 100854112 (0%) 142 223 d15_46_030 23437934592 1415 3685487616 (16%) 100715744 (0%) 181 222 d10_31_019 23437934592 1011 3816681472 (16%) 113389344 (0%) 182 222 d10_32_020 23437934592 1011 3816535040 (16%) 113652672 (0%) 183 222 d10_33_021 23437934592 1011 3817772032 (16%) 113124224 (0%) 184 222 d10_34_022 23437934592 1011 3817069568 (16%) 113225504 (0%) 185 222 d10_35_023 23437934592 1011 3816517632 (16%) 113163488 (0%) 186 222 d10_36_024 23437934592 1011 3815605248 (16%) 113475648 (0%) 187 222 d11_31_019 23437934592 1011 3815664640 (16%) 113100992 (0%) 188 222 d11_32_020 23437934592 1011 3815985152 (16%) 113325504 (0%) 189 222 d11_33_021 23437934592 1011 3815148544 (16%) 113328480 (0%) 190 223 d11_34_022 23437934592 1011 3814780928 (16%) 113457440 (0%) 191 223 d11_35_023 23437934592 1011 3815219200 (16%) 113065504 (0%) 192 223 d11_36_024 23437934592 1011 3815383040 (16%) 113170592 (0%) -------------- next part -------------- > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From aaron.s.knister at nasa.gov Sat Jan 13 17:26:51 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 13 Jan 2018 12:26:51 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Message-ID: Thanks, Peter. That definitely makes sense and I was actually wondering if performance was a factor. Do you know where to look to see what GPFS' perception of "performance" is for a given NSD? -Aaron On 1/13/18 12:00 PM, Peter Serocka wrote: > Within reasonable capacity limits it would also expect > to direct incoming data to disks that are best ?available? > from a current performance perspective ? like doing least > IOPS, having lowest latency and shortest filled queue length. > > You new NSDs, filled only with recent data, might quickly have > become the most busy units before reaching capacity balance, > simply because recent data tends to be more active than older stuff. > > Makes sense? > > ? Peter > >> On 2018 Jan 13 Sat, at 17:18, Aaron Knister wrote: >> >> Thanks Everyone! I whipped up a script to dump the block layout of a >> file and then join that with mmdf information. As part of my exploration >> I wrote one 2GB file to each of this particular filesystem's 4 data >> pools last night (using "touch $file; mmchattr $file -P $pool; dd >> of=$file") and have attached a dump of the layout/nsd information for >> each file/pool. The fields for the output are: >> >> diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, >> freePct, freeKbFragments, freeKbFragmentsPct >> >> >> Here's the highlight from pool1: >> >> 36? 264? d13_06_006??? 23437934592? 1213??? 4548935680? (19%) >> 83304320?? (0%) >> 59?? 74? d10_41_025??? 23437934592? 1011??? 6993759232? (30%) >> 58642816?? (0%) >> >> For this file (And anecdotally what I've seen looking at NSD I/O data >> for other files written to this pool) the pattern of more blocks being >> allocated to the NSDs that are ~20% free vs the NSDs that are 30% free >> seems to be fairly consistent. >> >> >> Looking at a snippet of pool2: >> 101? 238? d15_15_011??? 23437934592? 1415??? 2008394752?? (9%) >> 181699328?? (1%) >> 102? 235? d15_16_012??? 23437934592? 1415??? 2009153536?? (9%) >> 182165312?? (1%) >> 116? 248? d11_42_026??? 23437934592? 1011??? 4146111488? (18%) >> 134941504?? (1%) >> 117? 249? d13_42_026??? 23437934592? 1213??? 4147710976? (18%) >> 135203776?? (1%) >> >> there are slightly more blocks allocated in general on the NSDs with >> twice the amount of free space, but it doesn't seem to be a significant >> amount relative to the delta in free space. The pattern from pool1 >> certainly doesn't hold true here. >> >> Pool4 isn't very interesting because all of the NSDs are well balanced >> in terms of free space (all 16% free). >> >> Pool3, however, *is* particularly interesting. Here's a snippet: >> >> 106? 222? d15_24_016??? 23437934592? 1415??? 2957561856? (13%) >> 37436768?? (0%) >> 107? 222? d15_25_017??? 23437934592? 1415??? 2957537280? (13%) >> 37353984?? (0%) >> 108? 222? d15_26_018??? 23437934592? 1415??? 2957539328? (13%) >> 37335872?? (0%) >> 125? 222? d11_44_028??? 23437934592? 1011?? 13297235968? (57%) >> 20505568?? (0%) >> 126? 222? d12_44_028??? 23437934592? 1213?? 13296712704? (57%) >> 20632768?? (0%) >> 127? 222? d12_45_029??? 23437934592? 1213?? 13297404928? (57%) >> 20557408?? (0%) >> >> GPFS consistently allocated the same number of blocks to disks with ~4x >> the free space than it did the other disks in the pool. >> >> Suffice it to say-- I'm *very* confused :) >> >> -Aaron >> >> On 1/13/18 8:18 AM, Daniel Kidger wrote: >>> Aaron, >>>? >>> Also are your new NSDs the same size as your existing ones? >>> i.e. the NSDs that are at a higher %age full might have more free blocks >>> than the other NSDs? >>> Daniel >>> >>>? >>> IBM Storage Professional Badge >>> >>>? >>>???????????????? >>> *Dr Daniel Kidger* >>> IBM Technical Sales Specialist >>> Software Defined Solution Sales >>> >>> +44-(0)7818 522 266 >>> daniel.kidger at uk.ibm.com >>> >>>? >>>? >>>? >>> >>>??? ----- Original message ----- >>>??? From: Jan-Frode Myklebust >>>??? Sent by: gpfsug-discuss-bounces at spectrumscale.org >>>??? To: gpfsug main discussion list >>>??? Cc: >>>??? Subject: Re: [gpfsug-discuss] pool block allocation algorithm >>>??? Date: Sat, Jan 13, 2018 9:25 AM >>>???? >>>??? Don?t have documentation/whitepaper, but as I recall, it will first >>>??? allocate round-robin over failureGroup, then round-robin over >>>??? nsdServers, and then round-robin over volumes. So if these new NSDs >>>??? are defined as different failureGroup from the old disks, that might >>>??? explain it.. >>> >>> >>>??? -jf >>>??? l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >>>??? >: >>> >>>??????? Apologies if this has been covered elsewhere (I couldn't find it >>>??????? if it >>>??????? has). I'm curious how GPFS decides where to allocate new blocks. >>> >>>??????? We've got a filesystem that we added some NSDs to a while back >>>??????? and it >>>??????? hurt there for a little while because it appeared as though GPFS was >>>??????? choosing to allocate new blocks much more frequently on the >>>??????? ~100% free >>>??????? LUNs than the existing LUNs (I can't recall how free they were >>>??????? at the >>>??????? time). Looking at it now, though, it seems GPFS is doing the >>>??????? opposite. >>>??????? There's now a ~10% difference between the LUNs added and the >>>??????? existing >>>??????? LUNs (20% free vs 30% free) and GPFS is choosing to allocate new >>>??????? writes >>>??????? at a ratio of about 3:1 on the disks with *fewer* free blocks >>>??????? than on >>>??????? the disks with more free blocks. That's completely inconsistent with >>>??????? what we saw when we initially added the disks which makes me >>>??????? wonder how >>>??????? GPFS is choosing to allocate new blocks (other than the obvious bits >>>??????? about failure group, and replication factor). Could someone >>>??????? explain (or >>>??????? point me at a whitepaper) what factors GPFS uses when allocating >>>??????? blocks, >>>??????? particularly as it pertains to choosing one NSD over another >>>??????? within the >>>??????? same failure group. >>> >>>??????? Thanks! >>> >>>??????? -Aaron >>> >>>??????? -- >>>??????? Aaron Knister >>>??????? NASA Center for Climate Simulation (Code 606.2) >>>??????? Goddard Space Flight Center >>>??????? (301) 286-2776 >>>??????? _______________________________________________ >>>??????? gpfsug-discuss mailing list >>>??????? gpfsug-discuss at spectrumscale.org >>>??????? >>>??????? http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>??????? >>> >>>??? _______________________________________________ >>>??? gpfsug-discuss mailing list >>>??? gpfsug-discuss at spectrumscale.org >>>??? https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= >>> >>>? >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From wsawdon at us.ibm.com Sat Jan 13 19:43:22 2018 From: wsawdon at us.ibm.com (Wayne Sawdon) Date: Sat, 13 Jan 2018 11:43:22 -0800 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov><48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Message-ID: Originally, GPFS used a strict round robin, first over failure groups, then over volumes within each failure group. That had performance issues when one or more volumes was low on space. Then for a while there were a variety of weighted stripe methods including by free space and by capacity. The file system had an option allowing the user to change the stripe method. That option was removed when we switched to a "best effort" round robin, which does a round robin over the failure groups then volumes based on the allocation regions that a node owns. When the stripe width at a node drops below half of the failure groups or half of the volumes that node will acquire new allocation regions. Basically we vary the stripe width to avoid searching for free space on specific volumes. It will eventually even itself out or you could restripe the file system to even it out immediately. -Wayne ps. And of course, allocation in FPO is completely different. gpfsug-discuss-bounces at spectrumscale.org wrote on 01/13/2018 09:26:51 AM: > From: Aaron Knister > To: gpfsug main discussion list > Date: 01/13/2018 09:27 AM > Subject: Re: [gpfsug-discuss] pool block allocation algorithm > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Thanks, Peter. That definitely makes sense and I was actually wondering > if performance was a factor. Do you know where to look to see what GPFS' > perception of "performance" is for a given NSD? > > -Aaron > > On 1/13/18 12:00 PM, Peter Serocka wrote: > > Within reasonable capacity limits it would also expect > > to direct incoming data to disks that are best ?available? > > from a current performance perspective ? like doing least > > IOPS, having lowest latency and shortest filled queue length. > > > > You new NSDs, filled only with recent data, might quickly have > > become the most busy units before reaching capacity balance, > > simply because recent data tends to be more active than older stuff. > > > > Makes sense? > > > > ? Peter > > > >> On 2018 Jan 13 Sat, at 17:18, Aaron Knister > wrote: > >> > >> Thanks Everyone! I whipped up a script to dump the block layout of a > >> file and then join that with mmdf information. As part of my exploration > >> I wrote one 2GB file to each of this particular filesystem's 4 data > >> pools last night (using "touch $file; mmchattr $file -P $pool; dd > >> of=$file") and have attached a dump of the layout/nsd information for > >> each file/pool. The fields for the output are: > >> > >> diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, > >> freePct, freeKbFragments, freeKbFragmentsPct > >> > >> > >> Here's the highlight from pool1: > >> > >> 36? 264? d13_06_006??? 23437934592? 1213??? 4548935680? (19%) > >> 83304320?? (0%) > >> 59?? 74? d10_41_025??? 23437934592? 1011??? 6993759232? (30%) > >> 58642816?? (0%) > >> > >> For this file (And anecdotally what I've seen looking at NSD I/O data > >> for other files written to this pool) the pattern of more blocks being > >> allocated to the NSDs that are ~20% free vs the NSDs that are 30% free > >> seems to be fairly consistent. > >> > >> > >> Looking at a snippet of pool2: > >> 101? 238? d15_15_011??? 23437934592? 1415??? 2008394752?? (9%) > >> 181699328?? (1%) > >> 102? 235? d15_16_012??? 23437934592? 1415??? 2009153536?? (9%) > >> 182165312?? (1%) > >> 116? 248? d11_42_026??? 23437934592? 1011??? 4146111488? (18%) > >> 134941504?? (1%) > >> 117? 249? d13_42_026??? 23437934592? 1213??? 4147710976? (18%) > >> 135203776?? (1%) > >> > >> there are slightly more blocks allocated in general on the NSDs with > >> twice the amount of free space, but it doesn't seem to be a significant > >> amount relative to the delta in free space. The pattern from pool1 > >> certainly doesn't hold true here. > >> > >> Pool4 isn't very interesting because all of the NSDs are well balanced > >> in terms of free space (all 16% free). > >> > >> Pool3, however, *is* particularly interesting. Here's a snippet: > >> > >> 106? 222? d15_24_016??? 23437934592? 1415??? 2957561856? (13%) > >> 37436768?? (0%) > >> 107? 222? d15_25_017??? 23437934592? 1415??? 2957537280? (13%) > >> 37353984?? (0%) > >> 108? 222? d15_26_018??? 23437934592? 1415??? 2957539328? (13%) > >> 37335872?? (0%) > >> 125? 222? d11_44_028??? 23437934592? 1011?? 13297235968? (57%) > >> 20505568?? (0%) > >> 126? 222? d12_44_028??? 23437934592? 1213?? 13296712704? (57%) > >> 20632768?? (0%) > >> 127? 222? d12_45_029??? 23437934592? 1213?? 13297404928? (57%) > >> 20557408?? (0%) > >> > >> GPFS consistently allocated the same number of blocks to disks with ~4x > >> the free space than it did the other disks in the pool. > >> > >> Suffice it to say-- I'm *very* confused :) > >> > >> -Aaron > >> > >> On 1/13/18 8:18 AM, Daniel Kidger wrote: > >>> Aaron, > >>> > >>> Also are your new NSDs the same size as your existing ones? > >>> i.e. the NSDs that are at a higher %age full might have more free blocks > >>> than the other NSDs? > >>> Daniel > >>> > >>> > >>> IBM Storage Professional Badge > >>> u=https-3A__www.youracclaim.com_user_danel-2Dkidger&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=hu8pcNGJmsITfq8y9fzxDf9WoXD1Kr5ptVLEEpbwcjU&e=> > >>> > >>> > >>> *Dr Daniel Kidger* > >>> IBM Technical Sales Specialist > >>> Software Defined Solution Sales > >>> > >>> +44-(0)7818 522 266 > >>> daniel.kidger at uk.ibm.com > >>> > >>> > >>> > >>> > >>> > >>>??? ----- Original message ----- > >>>??? From: Jan-Frode Myklebust > >>>??? Sent by: gpfsug-discuss-bounces at spectrumscale.org > >>>??? To: gpfsug main discussion list > >>>??? Cc: > >>>??? Subject: Re: [gpfsug-discuss] pool block allocation algorithm > >>>??? Date: Sat, Jan 13, 2018 9:25 AM > >>> > >>>??? Don?t have documentation/whitepaper, but as I recall, it will first > >>>??? allocate round-robin over failureGroup, then round-robin over > >>>??? nsdServers, and then round-robin over volumes. So if these new NSDs > >>>??? are defined as different failureGroup from the old disks, that might > >>>??? explain it.. > >>> > >>> > >>>??? -jf > >>>??? l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister > >>>??? >: > >>> > >>>??????? Apologies if this has been covered elsewhere (I couldn't find it > >>>??????? if it > >>>??????? has). I'm curious how GPFS decides where to allocate new blocks. > >>> > >>>??????? We've got a filesystem that we added some NSDs to a while back > >>>??????? and it > >>>??????? hurt there for a little while because it appeared as > though GPFS was > >>>??????? choosing to allocate new blocks much more frequently on the > >>>??????? ~100% free > >>>??????? LUNs than the existing LUNs (I can't recall how free they were > >>>??????? at the > >>>??????? time). Looking at it now, though, it seems GPFS is doing the > >>>??????? opposite. > >>>??????? There's now a ~10% difference between the LUNs added and the > >>>??????? existing > >>>??????? LUNs (20% free vs 30% free) and GPFS is choosing to allocate new > >>>??????? writes > >>>??????? at a ratio of about 3:1 on the disks with *fewer* free blocks > >>>??????? than on > >>>??????? the disks with more free blocks. That's completely > inconsistent with > >>>??????? what we saw when we initially added the disks which makes me > >>>??????? wonder how > >>>??????? GPFS is choosing to allocate new blocks (other than the > obvious bits > >>>??????? about failure group, and replication factor). Could someone > >>>??????? explain (or > >>>??????? point me at a whitepaper) what factors GPFS uses when allocating > >>>??????? blocks, > >>>??????? particularly as it pertains to choosing one NSD over another > >>>??????? within the > >>>??????? same failure group. > >>> > >>>??????? Thanks! > >>> > >>>??????? -Aaron > >>> > >>>??????? -- > >>>??????? Aaron Knister > >>>??????? NASA Center for Climate Simulation (Code 606.2) > >>>??????? Goddard Space Flight Center > >>>??????? (301) 286-2776 > >>>??????? _______________________________________________ > >>>??????? gpfsug-discuss mailing list > >>>??????? gpfsug-discuss at spectrumscale.org > >>>??????? u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=QYsXVDOdNRcII7FPtAbCXEyYJzNSd_UXq8bmreALKxs&e= > > > >>>??????? https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > >>>??????? u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- > GTf8EwJ6AkZQiTsRQZ73UH20&e=> > >>> > >>>??? _______________________________________________ > >>>??? gpfsug-discuss mailing list > >>>??? gpfsug-discuss at spectrumscale.org > >>>??? https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- > GTf8EwJ6AkZQiTsRQZ73UH20&e= > >>> > >>> > >>> Unless stated otherwise above: > >>> IBM United Kingdom Limited - Registered in England and Wales with number > >>> 741598. > >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >>> > >>> > >>> > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > >>> > >> > >> -- > >> Aaron Knister > >> NASA Center for Climate Simulation (Code 606.2) > >> Goddard Space Flight Center > >> (301) 286-2776 > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sun Jan 14 22:22:15 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sun, 14 Jan 2018 17:22:15 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Message-ID: <80dccad3-f531-3a8e-ab28-3ffa83dbd990@nasa.gov> Thanks, Wayne. What you said makes sense although I'm not sure I completely grok it. Can you comment on whether or not historic LUN performance factors into allocation decisions? -Aaron On 1/13/18 2:43 PM, Wayne Sawdon wrote: > Originally, GPFS used a strict round robin, first over failure groups, > then over volumes within each > failure group. That had performance issues when one or more volumes was > low on space. Then > for a while there were a variety of weighted stripe methods including by > free space and by capacity. > The file system had an option allowing the user to change the stripe > method. That option was > removed when we switched to a "best effort" round robin, which does a > round robin over the > failure groups then volumes based on the allocation regions that a node > owns. When the stripe width > at a node drops below half of the failure groups or half of the volumes > that node will acquire new > allocation regions. > > Basically we vary the stripe width to avoid searching for free space on > specific volumes. It will > eventually even itself out or you could restripe the file system to even > it out immediately. > > -Wayne > > ps. And of course, allocation in FPO is completely different. > > > gpfsug-discuss-bounces at spectrumscale.org wrote on 01/13/2018 09:26:51 AM: > >> From: Aaron Knister >> To: gpfsug main discussion list >> Date: 01/13/2018 09:27 AM >> Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> >> Thanks, Peter. That definitely makes sense and I was actually wondering >> if performance was a factor. Do you know where to look to see what GPFS' >> perception of "performance" is for a given NSD? >> >> -Aaron >> >> On 1/13/18 12:00 PM, Peter Serocka wrote: >> > Within reasonable capacity limits it would also expect >> > to direct incoming data to disks that are best ?available? >> > from a current performance perspective ? like doing least >> > IOPS, having lowest latency and shortest filled queue length. >> > >> > You new NSDs, filled only with recent data, might quickly have >> > become the most busy units before reaching capacity balance, >> > simply because recent data tends to be more active than older stuff. >> > >> > Makes sense? >> > >> > ? Peter >> > >> >> On 2018 Jan 13 Sat, at 17:18, Aaron Knister >> wrote: >> >> >> >> Thanks Everyone! I whipped up a script to dump the block layout of a >> >> file and then join that with mmdf information. As part of my > exploration >> >> I wrote one 2GB file to each of this particular filesystem's 4 data >> >> pools last night (using "touch $file; mmchattr $file -P $pool; dd >> >> of=$file") and have attached a dump of the layout/nsd information for >> >> each file/pool. The fields for the output are: >> >> >> >> diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, >> >> freePct, freeKbFragments, freeKbFragmentsPct >> >> >> >> >> >> Here's the highlight from pool1: >> >> >> >> 36? 264? d13_06_006??? 23437934592? 1213??? 4548935680? (19%) >> >> 83304320?? (0%) >> >> 59?? 74? d10_41_025??? 23437934592? 1011??? 6993759232? (30%) >> >> 58642816?? (0%) >> >> >> >> For this file (And anecdotally what I've seen looking at NSD I/O data >> >> for other files written to this pool) the pattern of more blocks being >> >> allocated to the NSDs that are ~20% free vs the NSDs that are 30% free >> >> seems to be fairly consistent. >> >> >> >> >> >> Looking at a snippet of pool2: >> >> 101? 238? d15_15_011??? 23437934592? 1415??? 2008394752?? (9%) >> >> 181699328?? (1%) >> >> 102? 235? d15_16_012??? 23437934592? 1415??? 2009153536?? (9%) >> >> 182165312?? (1%) >> >> 116? 248? d11_42_026??? 23437934592? 1011??? 4146111488? (18%) >> >> 134941504?? (1%) >> >> 117? 249? d13_42_026??? 23437934592? 1213??? 4147710976? (18%) >> >> 135203776?? (1%) >> >> >> >> there are slightly more blocks allocated in general on the NSDs with >> >> twice the amount of free space, but it doesn't seem to be a significant >> >> amount relative to the delta in free space. The pattern from pool1 >> >> certainly doesn't hold true here. >> >> >> >> Pool4 isn't very interesting because all of the NSDs are well balanced >> >> in terms of free space (all 16% free). >> >> >> >> Pool3, however, *is* particularly interesting. Here's a snippet: >> >> >> >> 106? 222? d15_24_016??? 23437934592? 1415??? 2957561856? (13%) >> >> 37436768?? (0%) >> >> 107? 222? d15_25_017??? 23437934592? 1415??? 2957537280? (13%) >> >> 37353984?? (0%) >> >> 108? 222? d15_26_018??? 23437934592? 1415??? 2957539328? (13%) >> >> 37335872?? (0%) >> >> 125? 222? d11_44_028??? 23437934592? 1011?? 13297235968? (57%) >> >> 20505568?? (0%) >> >> 126? 222? d12_44_028??? 23437934592? 1213?? 13296712704? (57%) >> >> 20632768?? (0%) >> >> 127? 222? d12_45_029??? 23437934592? 1213?? 13297404928? (57%) >> >> 20557408?? (0%) >> >> >> >> GPFS consistently allocated the same number of blocks to disks with ~4x >> >> the free space than it did the other disks in the pool. >> >> >> >> Suffice it to say-- I'm *very* confused :) >> >> >> >> -Aaron >> >> >> >> On 1/13/18 8:18 AM, Daniel Kidger wrote: >> >>> Aaron, >> >>>? >> >>> Also are your new NSDs the same size as your existing ones? >> >>> i.e. the NSDs that are at a higher %age full might have more free > blocks >> >>> than the other NSDs? >> >>> Daniel >> >>> >> >>>? >> >>> IBM Storage Professional Badge >> >>> > > u=https-3A__www.youracclaim.com_user_danel-2Dkidger&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=hu8pcNGJmsITfq8y9fzxDf9WoXD1Kr5ptVLEEpbwcjU&e=> >> >>>? >> >>>???????????????? >> >>> *Dr Daniel Kidger* >> >>> IBM Technical Sales Specialist >> >>> Software Defined Solution Sales >> >>> >> >>> +44-(0)7818 522 266 >> >>> daniel.kidger at uk.ibm.com >> >>> >> >>>? >> >>>? >> >>>? >> >>> >> >>>??? ----- Original message ----- >> >>>??? From: Jan-Frode Myklebust >> >>>??? Sent by: gpfsug-discuss-bounces at spectrumscale.org >> >>>??? To: gpfsug main discussion list >> >>>??? Cc: >> >>>??? Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> >>>??? Date: Sat, Jan 13, 2018 9:25 AM >> >>>???? >> >>>??? Don?t have documentation/whitepaper, but as I recall, it will first >> >>>??? allocate round-robin over failureGroup, then round-robin over >> >>>??? nsdServers, and then round-robin over volumes. So if these new NSDs >> >>>??? are defined as different failureGroup from the old disks, that > might >> >>>??? explain it.. >> >>> >> >>> >> >>>??? -jf >> >>>??? l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >> >>>??? >: >> >>> >> >>>??????? Apologies if this has been covered elsewhere (I couldn't > find it >> >>>??????? if it >> >>>??????? has). I'm curious how GPFS decides where to allocate new > blocks. >> >>> >> >>>??????? We've got a filesystem that we added some NSDs to a while back >> >>>??????? and it >> >>>??????? hurt there for a little while because it appeared as >> though GPFS was >> >>>??????? choosing to allocate new blocks much more frequently on the >> >>>??????? ~100% free >> >>>??????? LUNs than the existing LUNs (I can't recall how free they were >> >>>??????? at the >> >>>??????? time). Looking at it now, though, it seems GPFS is doing the >> >>>??????? opposite. >> >>>??????? There's now a ~10% difference between the LUNs added and the >> >>>??????? existing >> >>>??????? LUNs (20% free vs 30% free) and GPFS is choosing to > allocate new >> >>>??????? writes >> >>>??????? at a ratio of about 3:1 on the disks with *fewer* free blocks >> >>>??????? than on >> >>>??????? the disks with more free blocks. That's completely >> inconsistent with >> >>>??????? what we saw when we initially added the disks which makes me >> >>>??????? wonder how >> >>>??????? GPFS is choosing to allocate new blocks (other than the >> obvious bits >> >>>??????? about failure group, and replication factor). Could someone >> >>>??????? explain (or >> >>>??????? point me at a whitepaper) what factors GPFS uses when > allocating >> >>>??????? blocks, >> >>>??????? particularly as it pertains to choosing one NSD over another >> >>>??????? within the >> >>>??????? same failure group. >> >>> >> >>>??????? Thanks! >> >>> >> >>>??????? -Aaron >> >>> >> >>>??????? -- >> >>>??????? Aaron Knister >> >>>??????? NASA Center for Climate Simulation (Code 606.2) >> >>>??????? Goddard Space Flight Center >> >>>??????? (301) 286-2776 >> >>>??????? _______________________________________________ >> >>>??????? gpfsug-discuss mailing list >> >>>??????? gpfsug-discuss at spectrumscale.org >> >>>??????? > u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=QYsXVDOdNRcII7FPtAbCXEyYJzNSd_UXq8bmreALKxs&e= >> > >> >>>??????? https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> >>>??????? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- >> GTf8EwJ6AkZQiTsRQZ73UH20&e=> >> >>> >> >>>??? _______________________________________________ >> >>>??? gpfsug-discuss mailing list >> >>>??? gpfsug-discuss at spectrumscale.org >> >>>??? https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- >> GTf8EwJ6AkZQiTsRQZ73UH20&e= >> >>> >> >>>? >> >>> Unless stated otherwise above: >> >>> IBM United Kingdom Limited - Registered in England and Wales with > number >> >>> 741598. >> >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > PO6 3AU >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> gpfsug-discuss mailing list >> >>> gpfsug-discuss at spectrumscale.org >> >>> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> >>> >> >> >> >> -- >> >> Aaron Knister >> >> NASA Center for Climate Simulation (Code 606.2) >> >> Goddard Space Flight Center >> >> (301) 286-2776 >> >> _______________________________________________ >> >> gpfsug-discuss mailing list >> >> gpfsug-discuss at spectrumscale.org >> >> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From kaustubh.katruwar at in.ibm.com Mon Jan 15 09:50:37 2018 From: kaustubh.katruwar at in.ibm.com (Kaustubh I Katruwar) Date: Mon, 15 Jan 2018 15:20:37 +0530 Subject: [gpfsug-discuss] SMB with LDAP In-Reply-To: References: Message-ID: Hi, Can you also describe your problem? The link talks about updating the LDAP server entries to help integrate with Spectrum Scale to serve SMB connections. Thanks and Regards, Kaustubh I. Katruwar From: Jeno Cram To: "gpfsug-discuss at spectrumscale.org" Date: 09/01/2018 09:06 PM Subject: [gpfsug-discuss] SMB with LDAP Sent by: gpfsug-discuss-bounces at spectrumscale.org Has anyone had any luck implementing CES with SMB using LDAP? This link doesn?t seem to have all of the information required to getting it working properly. https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_updateldapsmb.htm I?m assuming smbpasswd -a is still required for all users? What keeps the LDAP/SMB passwords in sync? What access does the bind user need in LDAP? What special requirements are required for Windows 10 clients to connect? Jeno Cram | Lead Application Support Engineer jcram at ddn.com _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=gTocoNEzBZA5cNptWoAoyoucYhc_7VA6hXTPOXtubUc&m=NEFCJtsgntiVgznfKbVs0BqwQJOubKG8dhgNlMXvQzs&s=0U9-rvVW4HxBCh0tdCYaElQer5JBxJJ9KFFWkkutUQU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From a.khiredine at meteo.dz Mon Jan 15 10:41:12 2018 From: a.khiredine at meteo.dz (atmane khiredine) Date: Mon, 15 Jan 2018 10:41:12 +0000 Subject: [gpfsug-discuss] temperature disk gss Message-ID: <4B32CB5C696F2849BDEF7DF9EACE884B85A15F93@SDEB-EXC01.meteo.dz> Dear All, is there a way to display the temperature of the drives in GSS Thanks Everyone! Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz From coetzee.ray at gmail.com Mon Jan 15 12:26:43 2018 From: coetzee.ray at gmail.com (Ray Coetzee) Date: Mon, 15 Jan 2018 12:26:43 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: Hi Sven yes, it's primarily reads. Kind regards Ray Coetzee Mob: +44 759 704 7060 Skype: ray.coetzee Email: coetzee.ray at gmail.com On Fri, Jan 12, 2018 at 8:57 PM, Sven Oehme wrote: > is this primary read or write ? > > > On Fri, Jan 12, 2018, 12:51 PM Ray Coetzee wrote: > >> Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. >> (Without the meltdown patch) >> >> Hey Bryan, I remember that quote from Yuri, that's why I hoped some >> "magic" may have been done since then. >> >> Other attempts to improve performance I've tried include: >> >> - Using LROC to have a larger chance of a cache hit (Unfortunately >> the entire dataset is multiple TB) >> - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for >> this purpose (Job runs halved in duration but nowhere near what NFS can >> give) >> - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and >> some others with little improvement. >> >> For those interested, as a performance comparison. The same job when run >> on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe >> scratch filesystem and over 60min on spindle based filesystem. >> >> Kind regards >> >> Ray Coetzee >> Email: coetzee.ray at gmail.com >> >> >> On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister < >> bbanister at jumptrading.com> wrote: >> >>> You could put all of your data onto SSDs in a RAID1 configuration so >>> that you don?t have insane read-modify-write penalties on writes (RAID1) >>> and avoid horrible seek thrashing that spinning rust requires (SSD random >>> access medium) for your 4K I/O operations. >>> >>> >>> >>> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best >>> not to touch it?. He gave many reasons why mmap operations on a >>> distributed file system is incredibly hard and not recommended. >>> >>> -Bryan >>> >>> >>> >>> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- >>> bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >>> *Sent:* Friday, January 12, 2018 8:45 AM >>> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >>> gpfsug-discuss at spectrumscale.org> >>> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >>> >>> >>> >>> *Note: External Email* >>> ------------------------------ >>> >>> what version of Scale are you using right now ? >>> >>> >>> >>> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >>> wrote: >>> >>> I'd like to ask the group of their experiences in improving the >>> performance of applications that use mmap calls against files on Spectrum >>> Scale. >>> >>> >>> >>> Besides using an NFS export from CES instead of a native GPFS mount, or >>> precaching the dataset into the pagepool, what other approaches are there >>> to offset the performance hit of the 4K IO size? >>> >>> >>> >>> Kind regards >>> >>> Ray Coetzee >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> ------------------------------ >>> >>> Note: This email is for the confidential use of the named addressee(s) >>> only and may contain proprietary, confidential or privileged information. >>> If you are not the intended recipient, you are hereby notified that any >>> review, dissemination or copying of this email is strictly prohibited, and >>> to please notify the sender immediately and destroy this email and any >>> attachments. Email transmission cannot be guaranteed to be secure or >>> error-free. The Company, therefore, does not make any guarantees as to the >>> completeness or accuracy of this email or any attachments. This email is >>> for informational purposes only and does not constitute a recommendation, >>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >>> or perform any type of transaction of a financial product. >>> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckerner at illinois.edu Mon Jan 15 14:03:26 2018 From: ckerner at illinois.edu (Chad Kerner) Date: Mon, 15 Jan 2018 08:03:26 -0600 Subject: [gpfsug-discuss] temperature disk gss In-Reply-To: References: <4300efd8e3134cf1ba50454cac76bd09@CHIHT3.ad.uillinois.edu> Message-ID: You can get it from the Smart statistics with smartctl -A . smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.10.0-229.el7.x86_64] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === Current Drive Temperature: 30 C Drive Trip Temperature: 65 C Manufactured in week 39 of year 2016 Specified cycle count over device lifetime: 50000 Accumulated start-stop cycles: 6 Specified load-unload count over device lifetime: 600000 Accumulated load-unload cycles: 35 Elements in grown defect list: 0 Vendor (Seagate) cache information Blocks sent to initiator = 418499985932288 Chad -- Chad Kerner, Senior Storage Engineer National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Jan 15, 2018 4:48 AM, "atmane khiredine" wrote: Dear All, is there a way to display the temperature of the drives in GSS Thanks Everyone! Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From alvise.dorigo at psi.ch Mon Jan 15 13:58:12 2018 From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI)) Date: Mon, 15 Jan 2018 13:58:12 +0000 Subject: [gpfsug-discuss] temperature disk gss In-Reply-To: <4B32CB5C696F2849BDEF7DF9EACE884B85A15F93@SDEB-EXC01.meteo.dz> References: <4B32CB5C696F2849BDEF7DF9EACE884B85A15F93@SDEB-EXC01.meteo.dz> Message-ID: <83A6EEB0EC738F459A39439733AE80451BBC421D@MBX114.d.ethz.ch> Hi Atmane, in my Gl2 system I can do: # mmlsenclosure all -L|grep -B2 tempSensor component type serial number component id failed value unit properties -------------- ------------- ------------ ------ ----- ---- ---------- tempSensor XXXXXXXXX DCM_0A no 41 C tempSensor XXXXXXXXX DCM_0B no 40 C tempSensor XXXXXXXXX DCM_1A no 49 C tempSensor XXXXXXXXX DCM_1B no 42 C tempSensor XXXXXXXXX DCM_2A no 50 C tempSensor XXXXXXXXX DCM_2B no 41 C tempSensor XXXXXXXXX DCM_3A no 48 C tempSensor XXXXXXXXX DCM_3B no 42 C tempSensor XXXXXXXXX DCM_4A no 47 C tempSensor XXXXXXXXX DCM_4B no 42 C tempSensor XXXXXXXXX ESM_A no 43 C tempSensor XXXXXXXXX ESM_B no 44 C tempSensor XXXXXXXXX POWERSUPPLY_BOT no 41 C tempSensor XXXXXXXXX POWERSUPPLY_TOP no 39 C tempSensor XXXXXXXXX DCM_0A no 50 C tempSensor XXXXXXXXX DCM_0B no 41 C tempSensor XXXXXXXXX DCM_1A no 48 C tempSensor XXXXXXXXX DCM_1B no 41 C tempSensor XXXXXXXXX DCM_2A no 47 C tempSensor XXXXXXXXX DCM_2B no 42 C tempSensor XXXXXXXXX DCM_3A no 47 C tempSensor XXXXXXXXX DCM_3B no 41 C tempSensor XXXXXXXXX DCM_4A no 47 C tempSensor XXXXXXXXX DCM_4B no 43 C tempSensor XXXXXXXXX ESM_A no 43 C tempSensor XXXXXXXXX ESM_B no 44 C tempSensor XXXXXXXXX POWERSUPPLY_BOT no 43 C tempSensor SV55261173 POWERSUPPLY_TOP no 39 C ( serials, 2nd column, are intentionally hidden by me). Check your admin guide to map the sensor names (3rd columns) to the particular locations inside your enclosures (which include locations very close to the disk drives) Alvise ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of atmane khiredine [a.khiredine at meteo.dz] Sent: Monday, January 15, 2018 11:41 AM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] temperature disk gss Dear All, is there a way to display the temperature of the drives in GSS Thanks Everyone! Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From oehmes at gmail.com Mon Jan 15 15:00:17 2018 From: oehmes at gmail.com (Sven Oehme) Date: Mon, 15 Jan 2018 15:00:17 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: do you have a repeatable test that runs on just a small number of nodes ? we where working (still are) on some mmap enhancements lest year, but didn't get all of them ready for the DCUT of Scale 5.0, so we integrated only a subset that was ready into Scale 5.0 and left the enhancement turned off by default (you can toggle it on/off via mmchconfig). if you are interested in testing this send me a direct mail and i will provide instructions on how to turn this on. it does help mmap read workloads significant (think factors) in testing with synthetic benchmarks in my lab, but i would be very interested in real application results and also interested to get a trace with the new code to see where we need to make further improvements. sven On Mon, Jan 15, 2018 at 4:26 AM Ray Coetzee wrote: > Hi Sven > yes, it's primarily reads. > > Kind regards > > Ray Coetzee > Mob: +44 759 704 7060 <+44%207597%20047060> > > Skype: ray.coetzee > > Email: coetzee.ray at gmail.com > > > On Fri, Jan 12, 2018 at 8:57 PM, Sven Oehme wrote: > >> is this primary read or write ? >> >> >> On Fri, Jan 12, 2018, 12:51 PM Ray Coetzee wrote: >> >>> Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. >>> (Without the meltdown patch) >>> >>> Hey Bryan, I remember that quote from Yuri, that's why I hoped some >>> "magic" may have been done since then. >>> >>> Other attempts to improve performance I've tried include: >>> >>> - Using LROC to have a larger chance of a cache hit (Unfortunately >>> the entire dataset is multiple TB) >>> - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for >>> this purpose (Job runs halved in duration but nowhere near what NFS can >>> give) >>> - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, >>> and some others with little improvement. >>> >>> For those interested, as a performance comparison. The same job when run >>> on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe >>> scratch filesystem and over 60min on spindle based filesystem. >>> >>> Kind regards >>> >>> Ray Coetzee >>> Email: coetzee.ray at gmail.com >>> >>> >>> On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister < >>> bbanister at jumptrading.com> wrote: >>> >>>> You could put all of your data onto SSDs in a RAID1 configuration so >>>> that you don?t have insane read-modify-write penalties on writes (RAID1) >>>> and avoid horrible seek thrashing that spinning rust requires (SSD random >>>> access medium) for your 4K I/O operations. >>>> >>>> >>>> >>>> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best >>>> not to touch it?. He gave many reasons why mmap operations on a >>>> distributed file system is incredibly hard and not recommended. >>>> >>>> -Bryan >>>> >>>> >>>> >>>> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: >>>> gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >>>> *Sent:* Friday, January 12, 2018 8:45 AM >>>> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >>>> gpfsug-discuss at spectrumscale.org> >>>> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >>>> >>>> >>>> >>>> *Note: External Email* >>>> ------------------------------ >>>> >>>> what version of Scale are you using right now ? >>>> >>>> >>>> >>>> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >>>> wrote: >>>> >>>> I'd like to ask the group of their experiences in improving the >>>> performance of applications that use mmap calls against files on Spectrum >>>> Scale. >>>> >>>> >>>> >>>> Besides using an NFS export from CES instead of a native GPFS mount, or >>>> precaching the dataset into the pagepool, what other approaches are there >>>> to offset the performance hit of the 4K IO size? >>>> >>>> >>>> >>>> Kind regards >>>> >>>> Ray Coetzee >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>>> >>>> ------------------------------ >>>> >>>> Note: This email is for the confidential use of the named addressee(s) >>>> only and may contain proprietary, confidential or privileged information. >>>> If you are not the intended recipient, you are hereby notified that any >>>> review, dissemination or copying of this email is strictly prohibited, and >>>> to please notify the sender immediately and destroy this email and any >>>> attachments. Email transmission cannot be guaranteed to be secure or >>>> error-free. The Company, therefore, does not make any guarantees as to the >>>> completeness or accuracy of this email or any attachments. This email is >>>> for informational purposes only and does not constitute a recommendation, >>>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >>>> or perform any type of transaction of a financial product. >>>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Christian.Fey at sva.de Mon Jan 15 16:16:05 2018 From: Christian.Fey at sva.de (Fey, Christian) Date: Mon, 15 Jan 2018 16:16:05 +0000 Subject: [gpfsug-discuss] switching on perfileset-quota Message-ID: <05334d4771da49b4bb5f4cb43f1c3558@sva.de> Hi all, does anyone here have experience about a possible impact of switching on "--perfileset-quota" for an existing gpfs filesystem in order to achieve quotas on per-user basis? During my very small tests I did not discover anything, but I would like to be sure. The existing filesystem already has quotas in place that hopefully do not get messed up. Is there anything I need to consider? Mit freundlichen Gr??en / Best Regards Christian Fey SVA System Vertrieb Alexander GmbH Borsigstra?e 14 65205 Wiesbaden Tel.: +49 6122 536-0 Fax: +49 6122 536-399 E-Mail: christian.fey at sva.de http://www.sva.de Gesch?ftsf?hrung: Philipp Alexander, Sven Eichelbaum Sitz der Gesellschaft: Wiesbaden Registergericht: Amtsgericht Wiesbaden, HRB 10315 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5415 bytes Desc: not available URL: From pierre-marie.brunet at cnes.fr Tue Jan 16 10:40:46 2018 From: pierre-marie.brunet at cnes.fr (Brunet Pierre-Marie) Date: Tue, 16 Jan 2018 10:40:46 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint Message-ID: Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I'm talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Tue Jan 16 12:41:28 2018 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 16 Jan 2018 12:41:28 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint Message-ID: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com> Hi PMB This is one of the areas where even the most experienced admins struggle. There is no single answer here. Much of it depends on your particular use case and the file system layout, storage choices, block sizes all go together. IBM has started to provide some templates for this (the one out is for Genomics) but much is left to do. If you could share some details on the overall user environment and data myself and others could offer some ideas. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Brunet Pierre-Marie Reply-To: gpfsug main discussion list Date: Tuesday, January 16, 2018 at 4:51 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] GPFS best practises : end user standpoint Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I?m talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jan 16 13:03:06 2018 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Tue, 16 Jan 2018 13:03:06 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com> References: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com> Message-ID: Aaron Knister liked your message with Boxer. On January 16, 2018 at 07:41:40 EST, Oesterlin, Robert wrote: Hi PMB This is one of the areas where even the most experienced admins struggle. There is no single answer here. Much of it depends on your particular use case and the file system layout, storage choices, block sizes all go together. IBM has started to provide some templates for this (the one out is for Genomics) but much is left to do. If you could share some details on the overall user environment and data myself and others could offer some ideas. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Brunet Pierre-Marie Reply-To: gpfsug main discussion list Date: Tuesday, January 16, 2018 at 4:51 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] GPFS best practises : end user standpoint Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I?m talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jan 16 13:47:19 2018 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Tue, 16 Jan 2018 13:47:19 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com>, Message-ID: <86593829-3D51-4203-98E0-E22B0A727C87@nasa.gov> Apologies for that. My mobile exchange email client has a like button you can tap in the email action menu. I just discovered it this morning accidentally and thought ?wonder what this does. Better push it to find out.? Nothing happened or so I thought. Apparently all it does is make you look like a moron on public mailing lists. Doh! On January 16, 2018 at 08:03:06 EST, Aaron Knister wrote: Aaron Knister liked your message with Boxer. On January 16, 2018 at 07:41:40 EST, Oesterlin, Robert wrote: Hi PMB This is one of the areas where even the most experienced admins struggle. There is no single answer here. Much of it depends on your particular use case and the file system layout, storage choices, block sizes all go together. IBM has started to provide some templates for this (the one out is for Genomics) but much is left to do. If you could share some details on the overall user environment and data myself and others could offer some ideas. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Brunet Pierre-Marie Reply-To: gpfsug main discussion list Date: Tuesday, January 16, 2018 at 4:51 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] GPFS best practises : end user standpoint Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I?m talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Tue Jan 16 15:47:54 2018 From: carlz at us.ibm.com (Carl Zetie) Date: Tue, 16 Jan 2018 15:47:54 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Tue Jan 16 16:08:15 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 16 Jan 2018 16:08:15 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: Message-ID: <1516118895.3326.25.camel@strath.ac.uk> On Tue, 2018-01-16 at 15:47 +0000, Carl Zetie wrote: > Maybe this would make for a good session at a future user group > meeting -- perhaps as an interactive session? IBM could potentially > provide a facilitator from our Design practice. > ? Most of it in my view is standard best practice regardless of the file system in use. So in our mandatory training for the HPC, we tell our users don't use whacked out characters in your file names and directories. Specifically no backticks, no asterik's, no question marks, no newlines (yes really), no slashes (either forward or backward) and for Mac users don't start the name with a space (forces sorting to the top). We recommend sticking to plain ASCII so no accented characters either (harder if your native language is not English I guess but we are UK based so...). We don't enforce that but if it causes the user problems then they are on their own. We also strongly recommend using ISO 8601 date formats in file names to get date sorting from a directory listing too. Surprisingly not widely known about, but a great "life hack". Then it boils down to don't create zillions of files. I would love to be able to somehow do per directory file number quota's where one could say set a default of a few thousand. Users would then have to justify needing a larger quota. Sure you can set a file number quota but that does not stop them putting them all in one directory. If users really need to have zillions of files then charge them more so you can afford to beef up your metadata disks to SSD. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 16 16:35:05 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 16 Jan 2018 16:35:05 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516118895.3326.25.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> Message-ID: Hi Jonathan, Comments / questions inline. Thanks! Kevin > On Jan 16, 2018, at 10:08 AM, Jonathan Buzzard wrote: > > On Tue, 2018-01-16 at 15:47 +0000, Carl Zetie wrote: >> Maybe this would make for a good session at a future user group >> meeting -- perhaps as an interactive session? IBM could potentially >> provide a facilitator from our Design practice. >> > > Most of it in my view is standard best practice regardless of the file > system in use. > > So in our mandatory training for the HPC, we tell our users don't use > whacked out characters in your file names and directories. Specifically > no backticks, no asterik's, no question marks, no newlines (yes > really), no slashes (either forward or backward) and for Mac users > don't start the name with a space (forces sorting to the top). We > recommend sticking to plain ASCII so no accented characters either > (harder if your native language is not English I guess but we are UK > based so...). We don't enforce that but if it causes the user problems > then they are on their own. We?re in Tennessee, so not only do we not speak English, we barely speak American ? y?all will just have to understand, bless your hearts! ;-). But seriously, like most Universities, we have a ton of users for whom English is not their ?primary? language, so dealing with ?interesting? filenames is pretty hard to avoid. And users? problems are our problems whether or not they?re our problem. > > We also strongly recommend using ISO 8601 date formats in file names to > get date sorting from a directory listing too. Surprisingly not widely > known about, but a great "life hack". > > Then it boils down to don't create zillions of files. I would love to > be able to somehow do per directory file number quota's where one could > say set a default of a few thousand. Users would then have to justify > needing a larger quota. Sure you can set a file number quota but that > does not stop them putting them all in one directory. If you?ve got (bio)medical users using your cluster I don?t see how you avoid this ? they?re using commercial apps that do this kind of stupid stuff (10?s of thousands of files in a directory and the full path to each file is longer than the contents of the files themselves!). This reminds me of way back in 2005 when we moved from an NFS server to GPFS ? I was moving users over by tarring up their home directories on the NFS server, copying the tarball over to GPFS and untarring it there ? worked great for 699 out of 700 users. But there was one user for whom the untar would fail every time I tried ? turned out that back in early versions of GPFS 2.3 IBM hadn?t considered that someone would put 6 million files in one directory! :-O > > If users really need to have zillions of files then charge them more so > you can afford to beef up your metadata disks to SSD. OK, so here?s my main question ? you?re right that SSD?s are the answer ? but how do you charge them more? SSDs are move expensive than hard disks, and enterprise SSDs are stupid expensive ? and users barely want to pay hard drive prices for their storage. If you?ve got the magic answer to how to charge them enough to pay for SSDs I?m sure I?m not the only one who?d love to hear how you do it?!?! > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cdd3310fd309b4986f95c08d55cfb5d10%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636517157039256068&sdata=jZZV718gaMie92MW43qaxlDl6EQcULdk6FONrXpsP7c%3D&reserved=0 From heinz.ernst at id.ethz.ch Tue Jan 16 16:56:52 2018 From: heinz.ernst at id.ethz.ch (Ernst Heinz (ID SD)) Date: Tue, 16 Jan 2018 16:56:52 +0000 Subject: [gpfsug-discuss] GPFS GA 5.0.0.0: mmces commands with inconsistent output Message-ID: <0CE1E9F220AF564DB0114D8539010F9B5229A68C@MBX116.d.ethz.ch> Hello to all peers and gurus Since more or less two weeks we have gpfs GA 5.0.0.0 running on our testenvironment Today I've seen following behavior on our SpectrumScale-testcluster which slighdly surprised me Following: Checking status of the cluster on different ways [root at testnas13ces01 idsd_erh_t1]# mmces state cluster CLUSTER AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13.ethz.ch FAILED DISABLED HEALTHY DISABLED DEPEND DISABLED DEPEND FAILED [root at testnas13ces01 idsd_erh_t1]# mmces state show -a NODE AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13ces01-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY testnas13ces02-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY does anyone of you guys has an explanation therefore? Is there someone else who has seen a behavior like this? By the way we have a similar view on one of our clusters on gpfs 4.2.3.4 (open PMR: 30218.112.848) Any kind of response would be very grateful Kind regards Heinz =============================================================== Heinz Ernst ID-Systemdienste WEC C 16 Weinbergstrasse 11 CH-8092 Zurich heinz.ernst at id.ethz.ch Phone: +41 44 633 84 48 Mobile: +41 79 216 15 50 =============================================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Tue Jan 16 17:25:47 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 16 Jan 2018 17:25:47 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: <1516118895.3326.25.camel@strath.ac.uk> Message-ID: <1516123547.3326.29.camel@strath.ac.uk> On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: [SNIP] > > We?re in Tennessee, so not only do we not speak English, we barely > speak American ? y?all will just have to understand, bless your > hearts!??;-).? > > But seriously, like most Universities, we have a ton of users for > whom English is not their ?primary? language, so dealing with > ?interesting? filenames is pretty hard to avoid.??And users? problems > are our problems whether or not they?re our problem. > User comes with problem, you investigate find problem is due to "wacky" characters point them to the mandatory training documentation, tell them they need to rename their files to something sane and take no further action. Sure English is not their primary language but *they* have chosen to study in an English speaking country so best to actually embrace that. I do get it, many of our users are not native English speakers as well. Yes it's a tough policy but on the other hand pandering to them does them no favours either. [SNIP] > If you?ve got (bio)medical users using your cluster I don?t see how > you avoid this ? they?re using commercial apps that do this kind of > stupid stuff (10?s of thousands of files in a directory and the full > path to each file is longer than the contents of the files > themselves!). Well then they have justified the use; aka it's not their fault so you up the quota for them. Though they could use different less brain dead software. The idea is to force a bump in the road so the users are aware that what they are doing is considered bad practice. Most users have no idea that putting a million files in a directory is not sensible and worse that trying to access them using a GUI file manager is positively brain dead. [SNIP] > OK, so here?s my main question ? you?re right that SSD?s are the > answer ? but how do you charge them more???SSDs are move expensive > than hard disks, and enterprise SSDs are stupid expensive ? and users > barely want to pay hard drive prices for their storage.??If you?ve > got the magic answer to how to charge them enough to pay for SSDs I?m > sure I?m not the only one who?d love to hear how you do it?!?! > Give every user a one million file number quota. Need to store more than one million files, then you are going to have to pay $X per extra million files. Either they cough up the money to continue using their brain dead software or they switch to less stupid software. If they complain you just say that enterprise SSD's are stupidly expensive and you are using that space up at an above average rate and so have to pay the costs. I am quite sure someone storing 1PB has to pay more than someone storing 1TB, so why should someone storing 20 million files not have to pay more than someone storing 100k files? The only difference is people are used to paying more to store extra bytes and not used to paying more for more files, but that is because most sane people don't store millions and millions of files necessitating the purchase of large amounts of expensive enterprise SSD's. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From valdis.kletnieks at vt.edu Wed Jan 17 11:35:58 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 17 Jan 2018 06:35:58 -0500 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516123547.3326.29.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> Message-ID: <72644.1516188958@turing-police.cc.vt.edu> On Tue, 16 Jan 2018 17:25:47 +0000, Jonathan Buzzard said: > User comes with problem, you investigate find problem is due to "wacky" > characters point them to the mandatory training documentation, tell > them they need to rename their files to something sane and take no > further action. Sure English is not their primary language but *they* > have chosen to study in an English speaking country so best to actually > embrace that. > I do get it, many of our users are not native English speakers as well. > Yes it's a tough policy but on the other hand pandering to them does > them no favours either. Go ahead and try to make that policy stick in Tokyo or Cairo or Berlin or any other city or country where English isn't the primary language. Seriously - this is something that IBM has to make work well across all languages if they want to make this fly globally and not just in places that are English-predominant. Though to be honest, most of the problems I've had have been with embedded blanks in filenames, newline characters in filenames (fortunately only a half dozen or so out of almost 500M files), and esc-[-random from people banging on F8 instead of 8, etc. Have the occasional backspace character creep in too. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From jonathan.buzzard at strath.ac.uk Wed Jan 17 13:48:14 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 17 Jan 2018 13:48:14 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <72644.1516188958@turing-police.cc.vt.edu> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> <72644.1516188958@turing-police.cc.vt.edu> Message-ID: <1516196894.3326.37.camel@strath.ac.uk> On Wed, 2018-01-17 at 06:35 -0500, valdis.kletnieks at vt.edu wrote: > On Tue, 16 Jan 2018 17:25:47 +0000, Jonathan Buzzard said: > > User comes with problem, you investigate find problem is due to > > "wacky" > > characters point them to the mandatory training documentation, tell > > them they need to rename their files to something sane and take no > > further action. Sure English is not their primary language but > > *they* > > have chosen to study in an English speaking country so best to > > actually > > embrace that. > > I do get it, many of our users are not native English speakers as > > well. > > Yes it's a tough policy but on the other hand pandering to them > > does > > them no favours either. > > Go ahead and try to make that policy stick in Tokyo or Cairo or > Berlin or any other city or country where English isn't the primary > language. Sure hard, but not *my* problem :-) Though to be fair missing accent's off is not that bad. Obviously if your language does not use the Roman alphabet then things are somewhat more serious. Note I am in the UK and you just don't use a ? in your file name (if your sensible) so I really do get it. > > Seriously - this is something that IBM has to make work well across > all languages if they want to make this fly globally and not just in > places that are English-predominant. Oh GPFS works fine; well unless it is ancient versions of mmbackup in which case they just cause your backup to be aborted without doing anything! It's all the other end user programs that, especially end user scripts that cause problems. So either we advise users that special characters might cause them problems and best to not use them in the first place, or wait till they have a problem and then tell them they need to rename some thousands of files. In sort it's not IBM's problem other than they are one of the major culprits who helped create this mess back in the day :-) > Though to be honest, most of the problems I've had have been with > embedded blanks in filenames, newline characters in filenames > (fortunately only a half dozen or so out of almost 500M files), and > esc-[-random from people banging on F8 instead of 8, etc.? Have the > occasional backspace character creep in too. The mind boggles how you manage to get a backspace character into a file name. A new line is bad enough, but a backspace!!! To be honest its more this sort of nonsense than none ASCII characters that cause the problems. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From valdis.kletnieks at vt.edu Wed Jan 17 14:43:05 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 17 Jan 2018 09:43:05 -0500 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516196894.3326.37.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> <72644.1516188958@turing-police.cc.vt.edu> <1516196894.3326.37.camel@strath.ac.uk> Message-ID: <8491.1516200185@turing-police.cc.vt.edu> On Wed, 17 Jan 2018 13:48:14 +0000, Jonathan Buzzard said: > The mind boggles how you manage to get a backspace character into a > file name. You can get yourself into all sorts of trouble with stty - in particular, if you're ssh'ed from one system to another, and they disagree on whether the "make the last character go away" key is Delete or Backspace. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jan 17 16:56:42 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 17 Jan 2018 16:56:42 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516123547.3326.29.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> Message-ID: <38F290EB-D2DD-40E8-910E-C0C9C2812E1E@vanderbilt.edu> Inline? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 On Jan 16, 2018, at 11:25 AM, Jonathan Buzzard > wrote: On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: [SNIP] I am quite sure someone storing 1PB has to pay more than someone storing 1TB, so why should someone storing 20 million files not have to pay more than someone storing 100k files? Because they won?t ? they?ll do something more brain dead like put a WD MyBook they bought at Costco on their desk and expect their job to copy data back and forth from it to /tmp on the compute node. We have to offer a service that users are willing to pay for ? we can?t dictate to them the way things WILL be. There?s a big difference between the way things should be and the way things actually are ? trust me, those of us in the United States know that better than most people around the world after the past year! Bigly! Buh-leave ME! :-O JAB. -------------- next part -------------- An HTML attachment was scrubbed... URL: From MDIETZ at de.ibm.com Wed Jan 17 17:08:02 2018 From: MDIETZ at de.ibm.com (Mathias Dietz) Date: Wed, 17 Jan 2018 18:08:02 +0100 Subject: [gpfsug-discuss] GPFS GA 5.0.0.0: mmces commands with inconsistent output In-Reply-To: <0CE1E9F220AF564DB0114D8539010F9B5229A68C@MBX116.d.ethz.ch> References: <0CE1E9F220AF564DB0114D8539010F9B5229A68C@MBX116.d.ethz.ch> Message-ID: Hi, let me start with a recommendation first before I explain how the cluster state is build. Starting with 4.2.1 please use the mmhealth command instead of using the mmces state/events command. The mmces state/event command will be deprecated in future releases. mmhealth node show -> show the node state for all components (incl. CES) mmhealth node show CES -> shows the CES components only. mmhealth cluster show -> show the cluster state Now to your problem: The Spectrum Scale health monitoring is done by a daemon which runs on each cluster node. This daemon is monitoring the state of all Spectrum Scale components on the local system and based on the resulting monitoring events it compiles a local system state (shown by mmhealth node show). By having a decentralized monitoring we reduce the monitoring overhead and increase resiliency against network glitches. In order to show a cluster wide state view we have to consolidate the events from all cluster nodes on a single node. The health monitoring daemon running on the cluster manager is taking the role (CSM) to receive events from all nodes through RPC calls and to compile the cluster state (shown by mmhealth cluster show) There can be cases where the (async) event forwarding to the CSM is delayed or dropped because of network delays, high system load, cluster manager failover or split brain cases. Those cases should resolve automatically after some time when event is resend. Summary: the cluster state might be temporary out of sync (eventually consistent), for getting a current state you should refer to mmhealth node show. If the problem does not resolve automatically, restarting the monitoring daemon will force a re-sync. Please open a PMR for the 5.0 issue too if the problem persist. Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale Development - Release Lead Architect (4.2.x) Spectrum Scale RAS Architect --------------------------------------------------------------------------- IBM Deutschland Am Weiher 24 65451 Kelsterbach Phone: +49 70342744105 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ----------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk WittkoppSitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Ernst Heinz (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Date: 01/16/2018 06:09 PM Subject: [gpfsug-discuss] GPFS GA 5.0.0.0: mmces commands with inconsistent output Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello to all peers and gurus Since more or less two weeks we have gpfs GA 5.0.0.0 running on our testenvironment Today I?ve seen following behavior on our SpectrumScale-testcluster which slighdly surprised me Following: Checking status of the cluster on different ways [root at testnas13ces01 idsd_erh_t1]# mmces state cluster CLUSTER AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13.ethz.ch FAILED DISABLED HEALTHY DISABLED DEPEND DISABLED DEPEND FAILED [root at testnas13ces01 idsd_erh_t1]# mmces state show -a NODE AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13ces01-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY testnas13ces02-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY does anyone of you guys has an explanation therefore? Is there someone else who has seen a behavior like this? By the way we have a similar view on one of our clusters on gpfs 4.2.3.4 (open PMR: 30218.112.848) Any kind of response would be very grateful Kind regards Heinz =============================================================== Heinz Ernst ID-Systemdienste WEC C 16 Weinbergstrasse 11 CH-8092 Zurich heinz.ernst at id.ethz.ch Phone: +41 44 633 84 48 Mobile: +41 79 216 15 50 =============================================================== _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Wed Jan 17 17:31:39 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 17 Jan 2018 17:31:39 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <38F290EB-D2DD-40E8-910E-C0C9C2812E1E@vanderbilt.edu> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> <38F290EB-D2DD-40E8-910E-C0C9C2812E1E@vanderbilt.edu> Message-ID: <1516210299.3326.43.camel@strath.ac.uk> On Wed, 2018-01-17 at 16:56 +0000, Buterbaugh, Kevin L wrote: > Inline? > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu?- (615)875-9633 > > > On Jan 16, 2018, at 11:25 AM, Jonathan Buzzard > rath.ac.uk> wrote: > > > > On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: > > > > [SNIP] > > > > I am quite sure someone storing 1PB has to pay more than someone > > storing 1TB, so why should someone storing 20 million files not > > have to pay more than someone storing 100k files?? > > > ? > Because they won?t ? they?ll do something more brain dead like put a > WD MyBook they bought at Costco on their desk and expect their job to > copy data back and forth from it to /tmp on the compute node. ?We > have to offer a service that users are willing to pay for ? we can?t > dictate to them the way things WILL be. However users need to be willing to pay for something that works. As they say cheap, fast and resilient pick any two. Getting back to your scenario, their performance will suck assuming it would work in the first place and secondly it's not your problem any more :-) > There?s a big difference between the way things should be and the way > things actually are ? trust me, those of us in the United States know > that better than most people around the world after the past year! > ?Bigly! ?Buh-leave ME! ?:-O > In my experience part of the problem is system administrators *not* pushing back a bit to users when they want something unreasonable. If you don't all you do is raise a generation of spoilt children. Secondly in my experience when you do push back most users are reasonable and understanding when it's explained to them in a manner they understand that what they are trying to do is not sensible. Oh and for the record I have been right their and had to put a file quota on a user that effectively stopped them dead in their tracks because half the files on the file system where from that one user, and everyone else was complaining that performance was going down the tube. I relevant example is a research group in my university where too cheap to by a proper file server so brought a large NAS box instead from a well known vendor. Just before Christmas the NAS box popped, it's still not replaced yet. Said research group have put their hands in their pockets for a server from a tier one vendor with a 4hr response maintenance plan. Funny that :-) JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From makaplan at us.ibm.com Wed Jan 17 20:10:45 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 17 Jan 2018 17:10:45 -0300 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: <1516118895.3326.25.camel@strath.ac.uk> Message-ID: Yes, "special" characters in pathnames can lead to trouble... But just for the record... GPFS supports the same liberal file name policy as standard POSIX. Namely any bytestring is valid, except: / delimits directory names The \0 (zero or Null Character) byte value marks the end of the pathname. There are limits on the length of an individual file or directory name. AND there is an OS imposed limit on the total length of a pathname you can pass through the file system APIs. From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/16/2018 01:58 PM Subject: Re: [gpfsug-discuss] GPFS best practises : end user standpoint Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Jonathan, Comments / questions inline. Thanks! Kevin > On Jan 16, 2018, at 10:08 AM, Jonathan Buzzard wrote: > > On Tue, 2018-01-16 at 15:47 +0000, Carl Zetie wrote: >> Maybe this would make for a good session at a future user group >> meeting -- perhaps as an interactive session? IBM could potentially >> provide a facilitator from our Design practice. >> > > Most of it in my view is standard best practice regardless of the file > system in use. > > So in our mandatory training for the HPC, we tell our users don't use > whacked out characters in your file names and directories. Specifically > no backticks, no asterik's, no question marks, no newlines (yes > really), no slashes (either forward or backward) and for Mac users > don't start the name with a space (forces sorting to the top). We > recommend sticking to plain ASCII so no accented characters either > (harder if your native language is not English I guess but we are UK > based so...). We don't enforce that but if it causes the user problems > then they are on their own. We?re in Tennessee, so not only do we not speak English, we barely speak American ? y?all will just have to understand, bless your hearts! ;-). But seriously, like most Universities, we have a ton of users for whom English is not their ?primary? language, so dealing with ?interesting? filenames is pretty hard to avoid. And users? problems are our problems whether or not they?re our problem. > > We also strongly recommend using ISO 8601 date formats in file names to > get date sorting from a directory listing too. Surprisingly not widely > known about, but a great "life hack". > > Then it boils down to don't create zillions of files. I would love to > be able to somehow do per directory file number quota's where one could > say set a default of a few thousand. Users would then have to justify > needing a larger quota. Sure you can set a file number quota but that > does not stop them putting them all in one directory. If you?ve got (bio)medical users using your cluster I don?t see how you avoid this ? they?re using commercial apps that do this kind of stupid stuff (10?s of thousands of files in a directory and the full path to each file is longer than the contents of the files themselves!). This reminds me of way back in 2005 when we moved from an NFS server to GPFS ? I was moving users over by tarring up their home directories on the NFS server, copying the tarball over to GPFS and untarring it there ? worked great for 699 out of 700 users. But there was one user for whom the untar would fail every time I tried ? turned out that back in early versions of GPFS 2.3 IBM hadn?t considered that someone would put 6 million files in one directory! :-O > > If users really need to have zillions of files then charge them more so > you can afford to beef up your metadata disks to SSD. OK, so here?s my main question ? you?re right that SSD?s are the answer ? but how do you charge them more? SSDs are move expensive than hard disks, and enterprise SSDs are stupid expensive ? and users barely want to pay hard drive prices for their storage. If you?ve got the magic answer to how to charge them enough to pay for SSDs I?m sure I?m not the only one who?d love to hear how you do it?!?! > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fgpfsug.org-252Fmailman-252Flistinfo-252Fgpfsug-2Ddiscuss-26data-3D02-257C01-257CKevin.Buterbaugh-2540vanderbilt.edu-257Cdd3310fd309b4986f95c08d55cfb5d10-257Cba5a7f39e3be4ab3b45067fa80faecad-257C0-257C1-257C636517157039256068-26sdata-3DjZZV718gaMie92MW43qaxlDl6EQcULdk6FONrXpsP7c-253D-26reserved-3D0&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=7n72wm4bwHfrK-yGlxSHLkVEIq0FDXA7XrPI_pyQq1M&s=1cYF3dt9odnG5zCHjcxMGl9_LbVAVNFrFu1iuv5585U&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=7n72wm4bwHfrK-yGlxSHLkVEIq0FDXA7XrPI_pyQq1M&s=_puCWAGyopu4m3M7evNjbILg3LkDCmiI9vJN2IG1iBE&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Wed Jan 17 22:12:01 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 17 Jan 2018 22:12:01 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Message-ID: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn't say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to 'generic' not 'dmm'. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with 'generic' vs. 'dmm' vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Wed Jan 17 08:39:35 2018 From: john.hearns at asml.com (John Hearns) Date: Wed, 17 Jan 2018 08:39:35 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516123547.3326.29.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> Message-ID: My own thoughts on this one are that I believe when software is being developed, the developers use 'small' use cases. At the company which develops the software, the developers will probably have a desktop machine with a modest amount of RAM and disk space. Then the company might have a small to medium sized HPC cluster. I know I am stretching things a bit, but I would imagine a lot of effort goes into verifying the correct operation of software on given data sets. When the software is released to customers, either commercially or internally within a company, it is suddenly applied to larger datasets, and is applied repetitively. Hence the creation of directories filled with thousands of small files. My own example from this here is in a former job, wind tunnel data was captured as thousands of PNG files which were frames from a camera. The data was shipped back to me on a hard drive, and I was asked to store it on an HSM system with tape as the lowest tier. I knew that the PNG files had all bene processed anyway, and significant data had been extracted. The engineers wanted to keep the data 'just in case'. I knew that keeping thousands of files is bad for filesystem performance and also on a tape based system you can have the fiels stored on many tapes, so if you ever do trigger a recall you have a festival of robot tape loading. So what I did was zip up all the directories. -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jonathan Buzzard Sent: Tuesday, January 16, 2018 6:26 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPFS best practises : end user standpoint On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: [SNIP] > > We?re in Tennessee, so not only do we not speak English, we barely > speak American ? y?all will just have to understand, bless your > hearts! ;-). > > But seriously, like most Universities, we have a ton of users for whom > English is not their ?primary? language, so dealing with ?interesting? > filenames is pretty hard to avoid. And users? problems are our > problems whether or not they?re our problem. > User comes with problem, you investigate find problem is due to "wacky" characters point them to the mandatory training documentation, tell them they need to rename their files to something sane and take no further action. Sure English is not their primary language but *they* have chosen to study in an English speaking country so best to actually embrace that. I do get it, many of our users are not native English speakers as well. Yes it's a tough policy but on the other hand pandering to them does them no favours either. [SNIP] > If you?ve got (bio)medical users using your cluster I don?t see how > you avoid this ? they?re using commercial apps that do this kind of > stupid stuff (10?s of thousands of files in a directory and the full > path to each file is longer than the contents of the files > themselves!). Well then they have justified the use; aka it's not their fault so you up the quota for them. Though they could use different less brain dead software. The idea is to force a bump in the road so the users are aware that what they are doing is considered bad practice. Most users have no idea that putting a million files in a directory is not sensible and worse that trying to access them using a GUI file manager is positively brain dead. [SNIP] > OK, so here?s my main question ? you?re right that SSD?s are the > answer ? but how do you charge them more? SSDs are move expensive > than hard disks, and enterprise SSDs are stupid expensive ? and users > barely want to pay hard drive prices for their storage. If you?ve got > the magic answer to how to charge them enough to pay for SSDs I?m sure > I?m not the only one who?d love to hear how you do it?!?! > Give every user a one million file number quota. Need to store more than one million files, then you are going to have to pay $X per extra million files. Either they cough up the money to continue using their brain dead software or they switch to less stupid software. If they complain you just say that enterprise SSD's are stupidly expensive and you are using that space up at an above average rate and so have to pay the costs. I am quite sure someone storing 1PB has to pay more than someone storing 1TB, so why should someone storing 20 million files not have to pay more than someone storing 100k files? The only difference is people are used to paying more to store extra bytes and not used to paying more for more files, but that is because most sane people don't store millions and millions of files necessitating the purchase of large amounts of expensive enterprise SSD's. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7Cf9b43f106c124ced6a4108d55d063196%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=c5B3JAJZDp3YiCN2uOzTmf%2BlsLMVRw6BsIzacQuORN8%3D&reserved=0 -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. From jjdoherty at yahoo.com Thu Jan 18 00:28:06 2018 From: jjdoherty at yahoo.com (Jim Doherty) Date: Thu, 18 Jan 2018 00:28:06 +0000 (UTC) Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> Message-ID: <1743984571.54681.1516235286972@mail.yahoo.com> Run a mmlsnsd -X ? I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN,?? not the /dev/mapper/ path.?? In our case the device is setup as dmm? [root at service5 ~]# mmlsnsd -X ?Disk name??? NSD volume ID????? Device???????? Devtype? Node name??????????????? Remarks???????? ? --------------------------------------------------------------------------------------------------- ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service5.pok.stglabs.ibm.com server node ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service6.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-4????? dmm????? service5.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-3????? dmm????? service6.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-1????? dmm????? service5.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-2????? dmm????? service6.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-3????? dmm????? service5.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-4????? dmm????? service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s? it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic ? 0972B6CD587CD8E0 /dev/sdk generic ? 0972B6CD587CD8E0 /dev/sdu generic ? 0972B6CD587CD8E0 /dev/sdah generic ? 0972B6CD587CD8E0 /dev/dm-0 dmm ? [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister wrote: Hi all, ? We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. ? The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: ? # Known disk types currently are: # #?? powerdisk? - EMC power path disk #?? vpath????? - IBM virtual path disk #?? dmm??????? - Device-Mapper Multipath (DMM) #?? dlmfdrv??? - Hitachi dlm #?? hdisk????? - AIX hard disk #?? lv???????? - AIX logical volume.? Historical usage only. #??????????????? Not allowed as a new device to mmcrnsd. #?? gpt??????? - GPFS partition on Windows disk #?? generic??? - Device having no unique failover or multipathing #??????????????? characteristic (predominantly Linux devices). #?? dasd?????? - DASD device (for Linux on z Systems) ? We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?.? This is configured in the /var/mmfs/etc/nsddevices file: ? if [[ $osName = Linux ]] then ? : # Add function to discover disks in the Linux environment. ? ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ? ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 "generic"}' ? ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi ? Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? ? Thanks in advance! -Bryan Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Thu Jan 18 15:09:22 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 18 Jan 2018 15:09:22 +0000 Subject: [gpfsug-discuss] Kerberos NFS Message-ID: Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Jan 18 15:49:15 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 15:49:15 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <1743984571.54681.1516235286972@mail.yahoo.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: <38f69158067c4b7495d525b9aba161ff@jumptrading.com> Hi Jim, Our NSDs are reporting the /dev/mapper/ devices in the `mmlsnsd -X` output, but due to the nsddevices file configuration they are all configured as ?generic? Devtype. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjdoherty at yahoo.com Thu Jan 18 16:32:58 2018 From: jjdoherty at yahoo.com (Jim Doherty) Date: Thu, 18 Jan 2018 16:32:58 +0000 (UTC) Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <38f69158067c4b7495d525b9aba161ff@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <38f69158067c4b7495d525b9aba161ff@jumptrading.com> Message-ID: <244901478.616245.1516293178108@mail.yahoo.com> What does the /var/mmfs/gen/mmsdrfs file show? [root at service5 ~]# grep SG_DISK /var/mmfs/gen/mmsdrfs | cut -f 15 -d ':' dmm dmm dmm dmm On Thursday, January 18, 2018, 10:49:32 AM EST, Bryan Banister wrote: #yiv4648160129 #yiv4648160129 -- _filtered #yiv4648160129 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv4648160129 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv4648160129 {font-family:Verdana;panose-1:2 11 6 4 3 5 4 4 2 4;} _filtered #yiv4648160129 {}#yiv4648160129 #yiv4648160129 p.yiv4648160129MsoNormal, #yiv4648160129 li.yiv4648160129MsoNormal, #yiv4648160129 div.yiv4648160129MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv4648160129 a:link, #yiv4648160129 span.yiv4648160129MsoHyperlink {color:blue;text-decoration:underline;}#yiv4648160129 a:visited, #yiv4648160129 span.yiv4648160129MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv4648160129 p.yiv4648160129msonormal, #yiv4648160129 li.yiv4648160129msonormal, #yiv4648160129 div.yiv4648160129msonormal {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 p.yiv4648160129msochpdefault, #yiv4648160129 li.yiv4648160129msochpdefault, #yiv4648160129 div.yiv4648160129msochpdefault {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 span.yiv4648160129msohyperlink {}#yiv4648160129 span.yiv4648160129msohyperlinkfollowed {}#yiv4648160129 span.yiv4648160129emailstyle17 {}#yiv4648160129 p.yiv4648160129msonormal1, #yiv4648160129 li.yiv4648160129msonormal1, #yiv4648160129 div.yiv4648160129msonormal1 {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;}#yiv4648160129 span.yiv4648160129msohyperlink1 {color:#0563C1;text-decoration:underline;}#yiv4648160129 span.yiv4648160129msohyperlinkfollowed1 {color:#954F72;text-decoration:underline;}#yiv4648160129 span.yiv4648160129emailstyle171 {color:windowtext;}#yiv4648160129 p.yiv4648160129msochpdefault1, #yiv4648160129 li.yiv4648160129msochpdefault1, #yiv4648160129 div.yiv4648160129msochpdefault1 {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 span.yiv4648160129EmailStyle28 {color:#1F497D;}#yiv4648160129 .yiv4648160129MsoChpDefault {font-size:10.0pt;} _filtered #yiv4648160129 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv4648160129 div.yiv4648160129WordSection1 {}#yiv4648160129 Hi Jim, ? Our NSDs are reporting the /dev/mapper/ devices in the `mmlsnsd -X` output, but due to the nsddevices file configuration they are all configured as ?generic? Devtype. -Bryan ? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org]On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file ? Note: External Email ? Run a mmlsnsd -X ? I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN,?? not the /dev/mapper/ path.?? In our case the device is setup as dmm? ? [root at service5 ~]# mmlsnsd -X ?Disk name??? NSD volume ID????? Device???????? Devtype? Node name??????????????? Remarks???????? ? --------------------------------------------------------------------------------------------------- ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service5.pok.stglabs.ibm.com server node ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service6.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-4????? dmm????? service5.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-3????? dmm????? service6.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-1????? dmm????? service5.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-2????? dmm????? service6.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-3????? dmm????? service5.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-4????? dmm????? service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# ? If you run an tspreparedisk -s? it will show you all of the paths. ? [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic ? 0972B6CD587CD8E0 /dev/sdk generic ? 0972B6CD587CD8E0 /dev/sdu generic ? 0972B6CD587CD8E0 /dev/sdah generic ? 0972B6CD587CD8E0 /dev/dm-0 dmm ? [root at service5 ~]# ? Jim ? ? ? Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister wrote: ? ? Hi all, ? We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. ? The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: ? # Known disk types currently are: # #?? powerdisk? - EMC power path disk #?? vpath????? - IBM virtual path disk #?? dmm??????? - Device-Mapper Multipath (DMM) #?? dlmfdrv??? - Hitachi dlm #?? hdisk????? - AIX hard disk #?? lv???????? - AIX logical volume.? Historical usage only. #??????????????? Not allowed as a new device to mmcrnsd. #?? gpt??????? - GPFS partition on Windows disk #?? generic??? - Device having no unique failover or multipathing #??????????????? characteristic (predominantly Linux devices). #?? dasd?????? - DASD device (for Linux on z Systems) ? We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?.? This is configured in the /var/mmfs/etc/nsddevices file: ? if [[ $osName = Linux ]] then ? : # Add function to discover disks in the Linux environment. ? ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ? ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 "generic"}' ? ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi ? Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? ? Thanks in advance! -Bryan ? Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Jan 18 16:35:23 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 16:35:23 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <244901478.616245.1516293178108@mail.yahoo.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <38f69158067c4b7495d525b9aba161ff@jumptrading.com> <244901478.616245.1516293178108@mail.yahoo.com> Message-ID: <9715f09cbdc74465ac2ba7cd883c9a8e@jumptrading.com> Generic: # grep SG_DISK /var/mmfs/gen/mmsdrfs | cut -f 15 -d ':' generic generic generic generic generic Cheers, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Thursday, January 18, 2018 10:33 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ What does the /var/mmfs/gen/mmsdrfs file show? [root at service5 ~]# grep SG_DISK /var/mmfs/gen/mmsdrfs | cut -f 15 -d ':' dmm dmm dmm dmm On Thursday, January 18, 2018, 10:49:32 AM EST, Bryan Banister > wrote: Hi Jim, Our NSDs are reporting the /dev/mapper/ devices in the `mmlsnsd -X` output, but due to the nsddevices file configuration they are all configured as ?generic? Devtype. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From JRLang at uwyo.edu Thu Jan 18 15:20:18 2018 From: JRLang at uwyo.edu (Jeffrey R. Lang) Date: Thu, 18 Jan 2018 15:20:18 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <1743984571.54681.1516235286972@mail.yahoo.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: So is there a way to change it if it?s set incorrectly? jeff From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Jan 18 16:59:15 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 16:59:15 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the Devtype to the ?correct? setting, for example: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi My question is what is the correct setting? And what does GPFS do differently with each setting? Thanks, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. Lang Sent: Thursday, January 18, 2018 9:20 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ So is there a way to change it if it?s set incorrectly? jeff From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From er.a.ross at gmail.com Thu Jan 18 20:01:16 2018 From: er.a.ross at gmail.com (Eric Ross) Date: Thu, 18 Jan 2018 14:01:16 -0600 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: Bryan, While waiting on the "official" word as to what each setting does differently, I remembered there was a brief explanation of the differences (at least of dmm vs generic) in the /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit (https://github.com/finley/GPFS-Goodies) I use to use when I was at IBM. //snip dmm vs. generic is used by GPFS to prioritize internal order of searching through available disks, then later GPFS discards other disk device names that it finds that match as the same NSD device by a different path. For this reason, dmm vs. generic is an important distinction if you are not explicitly producing the entire and exclusive set of disks that GPFS should use, as output from this script (nsddevices) _and_ exiting this script with a "return 0". -Brian Finley //end snip If I read that correctly, it would indicate GPFS uses those additional labels (at least dmm vs generic) as a mechanism to determine which device files to prefer when scanning a system and finding the same NSD available via different devices (i.e. /dev/mapper/foo vs /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed device, I guess it would just ignore the /dev/sd${foo} devices when it scans them. Also, the difference only seems to matter if you're not explicitly creating a list a Brian F. indicates. If you simply generate the list and exit (via return 0), GPFS wouldn't continue scanning, and then find the associated /dev/sd${foo} devices to begin with, and therefore wouldn't need a label like dmm to prioritize it over say a generic device. - Eric On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister wrote: > Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > Devtype to the ?correct? setting, for example: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > My question is what is the correct setting? > > > > And what does GPFS do differently with each setting? > > > > Thanks, > > -Bryan > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. > Lang > Sent: Thursday, January 18, 2018 9:20 AM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ________________________________ > > So is there a way to change it if it?s set incorrectly? > > > > jeff > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty > Sent: Wednesday, January 17, 2018 6:28 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > > > Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our > case the device is setup as dmm > > > > [root at service5 ~]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > --------------------------------------------------------------------------------------------------- > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service5.pok.stglabs.ibm.com server node > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service6.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > service5.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > service6.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > service5.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > service6.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-3 dmm > service5.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-4 dmm > service6.pok.stglabs.ibm.com server node > > [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > [root at service5 ~]# > > > > If you run an tspreparedisk -s it will show you all of the paths. > > > > [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > 0972B6CD587CD8E0 /dev/sda generic > 0972B6CD587CD8E0 /dev/sdk generic > 0972B6CD587CD8E0 /dev/sdu generic > 0972B6CD587CD8E0 /dev/sdah generic > 0972B6CD587CD8E0 /dev/dm-0 dmm > [root at service5 ~]# > > > > Jim > > > > > > > > Jim > > On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: > > > > > > Hi all, > > > > We are reviewing some of our configurations and were not sure what to make > of the NSD Device Types that GPFS uses and what, if anything, do they change > about how GPFS accesses/recovers/manages/etc the underlying storage based on > this setting. > > > > The documentation doesn?t say much about it other than to consult the > /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > section: > > > > # Known disk types currently are: > > # > > # powerdisk - EMC power path disk > > # vpath - IBM virtual path disk > > # dmm - Device-Mapper Multipath (DMM) > > # dlmfdrv - Hitachi dlm > > # hdisk - AIX hard disk > > # lv - AIX logical volume. Historical usage only. > > # Not allowed as a new device to mmcrnsd. > > # gpt - GPFS partition on Windows disk > > # generic - Device having no unique failover or multipathing > > # characteristic (predominantly Linux devices). > > # dasd - DASD device (for Linux on z Systems) > > > > We have our storage under Linux Device-Mapper Multipath control (two device > paths to all storage, active/passive) and are accessible under /dev/mapper, > but the NSD types are current set to ?generic? not ?dmm?. This is > configured in the /var/mmfs/etc/nsddevices file: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > Can somebody from IBM explain what the correct setting should be and what > differences GPFS does with ?generic? vs. ?dmm? vs. others? > > > > Thanks in advance! > > -Bryan > > > > ________________________________ > > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From bbanister at jumptrading.com Thu Jan 18 20:09:09 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 20:09:09 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: <149af11d97db41b0a343932cd936eafc@jumptrading.com> Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross Sent: Thursday, January 18, 2018 2:01 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ------------------------------------------------- Bryan, While waiting on the "official" word as to what each setting does differently, I remembered there was a brief explanation of the differences (at least of dmm vs generic) in the /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit (https://github.com/finley/GPFS-Goodies) I use to use when I was at IBM. //snip dmm vs. generic is used by GPFS to prioritize internal order of searching through available disks, then later GPFS discards other disk device names that it finds that match as the same NSD device by a different path. For this reason, dmm vs. generic is an important distinction if you are not explicitly producing the entire and exclusive set of disks that GPFS should use, as output from this script (nsddevices) _and_ exiting this script with a "return 0". -Brian Finley //end snip If I read that correctly, it would indicate GPFS uses those additional labels (at least dmm vs generic) as a mechanism to determine which device files to prefer when scanning a system and finding the same NSD available via different devices (i.e. /dev/mapper/foo vs /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed device, I guess it would just ignore the /dev/sd${foo} devices when it scans them. Also, the difference only seems to matter if you're not explicitly creating a list a Brian F. indicates. If you simply generate the list and exit (via return 0), GPFS wouldn't continue scanning, and then find the associated /dev/sd${foo} devices to begin with, and therefore wouldn't need a label like dmm to prioritize it over say a generic device. - Eric On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister wrote: > Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > Devtype to the ?correct? setting, for example: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > My question is what is the correct setting? > > > > And what does GPFS do differently with each setting? > > > > Thanks, > > -Bryan > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. > Lang > Sent: Thursday, January 18, 2018 9:20 AM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ________________________________ > > So is there a way to change it if it?s set incorrectly? > > > > jeff > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty > Sent: Wednesday, January 17, 2018 6:28 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > > > Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our > case the device is setup as dmm > > > > [root at service5 ~]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > --------------------------------------------------------------------------------------------------- > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service5.pok.stglabs.ibm.com server node > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service6.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > service5.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > service6.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > service5.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > service6.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-3 dmm > service5.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-4 dmm > service6.pok.stglabs.ibm.com server node > > [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > [root at service5 ~]# > > > > If you run an tspreparedisk -s it will show you all of the paths. > > > > [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > 0972B6CD587CD8E0 /dev/sda generic > 0972B6CD587CD8E0 /dev/sdk generic > 0972B6CD587CD8E0 /dev/sdu generic > 0972B6CD587CD8E0 /dev/sdah generic > 0972B6CD587CD8E0 /dev/dm-0 dmm > [root at service5 ~]# > > > > Jim > > > > > > > > Jim > > On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: > > > > > > Hi all, > > > > We are reviewing some of our configurations and were not sure what to make > of the NSD Device Types that GPFS uses and what, if anything, do they change > about how GPFS accesses/recovers/manages/etc the underlying storage based on > this setting. > > > > The documentation doesn?t say much about it other than to consult the > /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > section: > > > > # Known disk types currently are: > > # > > # powerdisk - EMC power path disk > > # vpath - IBM virtual path disk > > # dmm - Device-Mapper Multipath (DMM) > > # dlmfdrv - Hitachi dlm > > # hdisk - AIX hard disk > > # lv - AIX logical volume. Historical usage only. > > # Not allowed as a new device to mmcrnsd. > > # gpt - GPFS partition on Windows disk > > # generic - Device having no unique failover or multipathing > > # characteristic (predominantly Linux devices). > > # dasd - DASD device (for Linux on z Systems) > > > > We have our storage under Linux Device-Mapper Multipath control (two device > paths to all storage, active/passive) and are accessible under /dev/mapper, > but the NSD types are current set to ?generic? not ?dmm?. This is > configured in the /var/mmfs/etc/nsddevices file: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > Can somebody from IBM explain what the correct setting should be and what > differences GPFS does with ?generic? vs. ?dmm? vs. others? > > > > Thanks in advance! > > -Bryan > > > > ________________________________ > > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From bbanister at jumptrading.com Thu Jan 18 20:12:15 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 20:12:15 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <149af11d97db41b0a343932cd936eafc@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: I should have also stated that the iostat results also report data going through the device-mapper-multipath devices as desired, -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Bryan Banister Sent: Thursday, January 18, 2018 2:09 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ------------------------------------------------- Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross Sent: Thursday, January 18, 2018 2:01 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ------------------------------------------------- Bryan, While waiting on the "official" word as to what each setting does differently, I remembered there was a brief explanation of the differences (at least of dmm vs generic) in the /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit (https://github.com/finley/GPFS-Goodies) I use to use when I was at IBM. //snip dmm vs. generic is used by GPFS to prioritize internal order of searching through available disks, then later GPFS discards other disk device names that it finds that match as the same NSD device by a different path. For this reason, dmm vs. generic is an important distinction if you are not explicitly producing the entire and exclusive set of disks that GPFS should use, as output from this script (nsddevices) _and_ exiting this script with a "return 0". -Brian Finley //end snip If I read that correctly, it would indicate GPFS uses those additional labels (at least dmm vs generic) as a mechanism to determine which device files to prefer when scanning a system and finding the same NSD available via different devices (i.e. /dev/mapper/foo vs /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed device, I guess it would just ignore the /dev/sd${foo} devices when it scans them. Also, the difference only seems to matter if you're not explicitly creating a list a Brian F. indicates. If you simply generate the list and exit (via return 0), GPFS wouldn't continue scanning, and then find the associated /dev/sd${foo} devices to begin with, and therefore wouldn't need a label like dmm to prioritize it over say a generic device. - Eric On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister wrote: > Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > Devtype to the ?correct? setting, for example: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > My question is what is the correct setting? > > > > And what does GPFS do differently with each setting? > > > > Thanks, > > -Bryan > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. > Lang > Sent: Thursday, January 18, 2018 9:20 AM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ________________________________ > > So is there a way to change it if it?s set incorrectly? > > > > jeff > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty > Sent: Wednesday, January 17, 2018 6:28 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > > > Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our > case the device is setup as dmm > > > > [root at service5 ~]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > --------------------------------------------------------------------------------------------------- > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service5.pok.stglabs.ibm.com server node > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service6.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > service5.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > service6.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > service5.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > service6.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-3 dmm > service5.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-4 dmm > service6.pok.stglabs.ibm.com server node > > [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > [root at service5 ~]# > > > > If you run an tspreparedisk -s it will show you all of the paths. > > > > [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > 0972B6CD587CD8E0 /dev/sda generic > 0972B6CD587CD8E0 /dev/sdk generic > 0972B6CD587CD8E0 /dev/sdu generic > 0972B6CD587CD8E0 /dev/sdah generic > 0972B6CD587CD8E0 /dev/dm-0 dmm > [root at service5 ~]# > > > > Jim > > > > > > > > Jim > > On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: > > > > > > Hi all, > > > > We are reviewing some of our configurations and were not sure what to make > of the NSD Device Types that GPFS uses and what, if anything, do they change > about how GPFS accesses/recovers/manages/etc the underlying storage based on > this setting. > > > > The documentation doesn?t say much about it other than to consult the > /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > section: > > > > # Known disk types currently are: > > # > > # powerdisk - EMC power path disk > > # vpath - IBM virtual path disk > > # dmm - Device-Mapper Multipath (DMM) > > # dlmfdrv - Hitachi dlm > > # hdisk - AIX hard disk > > # lv - AIX logical volume. Historical usage only. > > # Not allowed as a new device to mmcrnsd. > > # gpt - GPFS partition on Windows disk > > # generic - Device having no unique failover or multipathing > > # characteristic (predominantly Linux devices). > > # dasd - DASD device (for Linux on z Systems) > > > > We have our storage under Linux Device-Mapper Multipath control (two device > paths to all storage, active/passive) and are accessible under /dev/mapper, > but the NSD types are current set to ?generic? not ?dmm?. This is > configured in the /var/mmfs/etc/nsddevices file: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > Can somebody from IBM explain what the correct setting should be and what > differences GPFS does with ?generic? vs. ?dmm? vs. others? > > > > Thanks in advance! > > -Bryan > > > > ________________________________ > > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From er.a.ross at gmail.com Thu Jan 18 20:22:09 2018 From: er.a.ross at gmail.com (Eric Ross) Date: Thu, 18 Jan 2018 14:22:09 -0600 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <149af11d97db41b0a343932cd936eafc@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: I would say that's what it *appears* to be doing. The sys admin in me would still want some "official" word from IBM, as I wouldn't my assumption to be the cause of a meltdown at 16.50 on a Friday when everyone is heading out for the weekend ;) Cheers, -Eric On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister wrote: > Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross > Sent: Thursday, January 18, 2018 2:01 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file > > Note: External Email > ------------------------------------------------- > > Bryan, > > While waiting on the "official" word as to what each setting does > differently, I remembered there was a brief explanation of the > differences (at least of dmm vs generic) in the > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit > (https://github.com/finley/GPFS-Goodies) I use to use when I was at > IBM. > > //snip > > dmm vs. generic is used by GPFS to prioritize internal order of > searching through available disks, then later GPFS discards other > disk device names that it finds that match as the same NSD device > by a different path. For this reason, dmm vs. generic is an > important distinction if you are not explicitly producing the > entire and exclusive set of disks that GPFS should use, as output > from this script (nsddevices) _and_ exiting this script with a > "return 0". -Brian Finley > > //end snip > > If I read that correctly, it would indicate GPFS uses those additional > labels (at least dmm vs generic) as a mechanism to determine which > device files to prefer when scanning a system and finding the same NSD > available via different devices (i.e. /dev/mapper/foo vs > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed > device, I guess it would just ignore the /dev/sd${foo} devices when it > scans them. Also, the difference only seems to matter if you're not > explicitly creating a list a Brian F. indicates. If you simply > generate the list and exit (via return 0), GPFS wouldn't continue > scanning, and then find the associated /dev/sd${foo} devices to begin > with, and therefore wouldn't need a label like dmm to prioritize it > over say a generic device. > > > - Eric > > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister > wrote: >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the >> Devtype to the ?correct? setting, for example: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> My question is what is the correct setting? >> >> >> >> And what does GPFS do differently with each setting? >> >> >> >> Thanks, >> >> -Bryan >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. >> Lang >> Sent: Thursday, January 18, 2018 9:20 AM >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> Note: External Email >> >> ________________________________ >> >> So is there a way to change it if it?s set incorrectly? >> >> >> >> jeff >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty >> Sent: Wednesday, January 17, 2018 6:28 PM >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> >> >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our >> case the device is setup as dmm >> >> >> >> [root at service5 ~]# mmlsnsd -X >> >> Disk name NSD volume ID Device Devtype Node name >> Remarks >> --------------------------------------------------------------------------------------------------- >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service5.pok.stglabs.ibm.com server node >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service6.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm >> service5.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm >> service6.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm >> service5.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm >> service6.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-3 dmm >> service5.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-4 dmm >> service6.pok.stglabs.ibm.com server node >> >> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: >> [root at service5 ~]# >> >> >> >> If you run an tspreparedisk -s it will show you all of the paths. >> >> >> >> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 >> 0972B6CD587CD8E0 /dev/sda generic >> 0972B6CD587CD8E0 /dev/sdk generic >> 0972B6CD587CD8E0 /dev/sdu generic >> 0972B6CD587CD8E0 /dev/sdah generic >> 0972B6CD587CD8E0 /dev/dm-0 dmm >> [root at service5 ~]# >> >> >> >> Jim >> >> >> >> >> >> >> >> Jim >> >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister >> wrote: >> >> >> >> >> >> Hi all, >> >> >> >> We are reviewing some of our configurations and were not sure what to make >> of the NSD Device Types that GPFS uses and what, if anything, do they change >> about how GPFS accesses/recovers/manages/etc the underlying storage based on >> this setting. >> >> >> >> The documentation doesn?t say much about it other than to consult the >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this >> section: >> >> >> >> # Known disk types currently are: >> >> # >> >> # powerdisk - EMC power path disk >> >> # vpath - IBM virtual path disk >> >> # dmm - Device-Mapper Multipath (DMM) >> >> # dlmfdrv - Hitachi dlm >> >> # hdisk - AIX hard disk >> >> # lv - AIX logical volume. Historical usage only. >> >> # Not allowed as a new device to mmcrnsd. >> >> # gpt - GPFS partition on Windows disk >> >> # generic - Device having no unique failover or multipathing >> >> # characteristic (predominantly Linux devices). >> >> # dasd - DASD device (for Linux on z Systems) >> >> >> >> We have our storage under Linux Device-Mapper Multipath control (two device >> paths to all storage, active/passive) and are accessible under /dev/mapper, >> but the NSD types are current set to ?generic? not ?dmm?. This is >> configured in the /var/mmfs/etc/nsddevices file: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> Can somebody from IBM explain what the correct setting should be and what >> differences GPFS does with ?generic? vs. ?dmm? vs. others? >> >> >> >> Thanks in advance! >> >> -Bryan >> >> >> >> ________________________________ >> >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ________________________________ >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From yguvvala at cambridgecomputer.com Thu Jan 18 22:10:35 2018 From: yguvvala at cambridgecomputer.com (Yugendra Guvvala) Date: Thu, 18 Jan 2018 17:10:35 -0500 Subject: [gpfsug-discuss] Excluding Filesets on snapshots Message-ID: Hi, We have 5 dependent Filesets linked to root fileset (mount point) and we are creating a new independent Fileset which will be linked to root fileset (mount point). We want to take snapshots of the dependent filesets and independent fileset on different periodic intervals. Is there a way to exclude independent fileset and take snapshot of all dependent filesets ? -- Thanks, Yugi -------------- next part -------------- An HTML attachment was scrubbed... URL: From rohwedder at de.ibm.com Fri Jan 19 08:07:17 2018 From: rohwedder at de.ibm.com (Markus Rohwedder) Date: Fri, 19 Jan 2018 09:07:17 +0100 Subject: [gpfsug-discuss] Excluding Filesets on snapshots In-Reply-To: References: Message-ID: Hello Yugendra, Snapshots are taken per Inode space. Both the root fileset and the independent fileset you created own separate inode spaces, even if the independent fileset is nested in the root fileset. Taking a snapshot of the root fileset should exclude the independent fileset. e.g: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.03.52' -j 'root' Taking a filesystem snapshot however would include all inode spaces of the file system and would therefore include the independent fileset. e.g.: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.04.47' Mit freundlichen Gr??en / Kind regards Dr. Markus Rohwedder Spectrum Scale GUI Development Phone: +49 7034 6430190 IBM Deutschland E-Mail: rohwedder at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina K?deritz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Yugendra Guvvala To: gpfsug main discussion list Date: 01/18/2018 11:11 PM Subject: [gpfsug-discuss] Excluding Filesets on snapshots Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, We have 5 dependent Filesets linked to root fileset (mount point) and we are creating a new independent Fileset which will be linked to root fileset (mount point). We want to take snapshots of the dependent filesets and independent fileset on different periodic intervals. Is there a way to exclude independent fileset and take snapshot of all dependent filesets ? -- Thanks, Yugi _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=vZOWTPN66Z-dgh-Xs1VV4BI9dhIVi3BUFi7Zt1gPE2E&s=3HAnqhprHHsQe1cvwJIYK4cFJwG7BcNK6hKxXaDQ7sA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14162451.gif Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From jonbernard at gmail.com Fri Jan 19 15:57:11 2018 From: jonbernard at gmail.com (Jon Bernard) Date: Fri, 19 Jan 2018 10:57:11 -0500 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: A few years ago, Yuri noted that "dmm" has a special meaning to GPFS code. SCSI3 PR code keys off this disk type. https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014924464#77777777-0000-0000-0000-000014924464 https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#scsi3 Jon On Thu, Jan 18, 2018 at 3:22 PM, Eric Ross wrote: > I would say that's what it *appears* to be doing. The sys admin in me > would still want some "official" word from IBM, as I wouldn't my > assumption to be the cause of a meltdown at 16.50 on a Friday when > everyone is heading out for the weekend ;) > > Cheers, > -Eric > > On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister > wrote: > > Great! So this is just for selecting devices and according to my > `mmlsnsd -X` output it's using the correct ones, so I can probably return > to ignoring this parameter! > > -Bryan > > > > -----Original Message----- > > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org] On Behalf Of Eric Ross > > Sent: Thursday, January 18, 2018 2:01 PM > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ------------------------------------------------- > > > > Bryan, > > > > While waiting on the "official" word as to what each setting does > > differently, I remembered there was a brief explanation of the > > differences (at least of dmm vs generic) in the > > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit > > (https://github.com/finley/GPFS-Goodies) I use to use when I was at > > IBM. > > > > //snip > > > > dmm vs. generic is used by GPFS to prioritize internal order of > > searching through available disks, then later GPFS discards other > > disk device names that it finds that match as the same NSD device > > by a different path. For this reason, dmm vs. generic is an > > important distinction if you are not explicitly producing the > > entire and exclusive set of disks that GPFS should use, as output > > from this script (nsddevices) _and_ exiting this script with a > > "return 0". -Brian Finley > > > > //end snip > > > > If I read that correctly, it would indicate GPFS uses those additional > > labels (at least dmm vs generic) as a mechanism to determine which > > device files to prefer when scanning a system and finding the same NSD > > available via different devices (i.e. /dev/mapper/foo vs > > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed > > device, I guess it would just ignore the /dev/sd${foo} devices when it > > scans them. Also, the difference only seems to matter if you're not > > explicitly creating a list a Brian F. indicates. If you simply > > generate the list and exit (via return 0), GPFS wouldn't continue > > scanning, and then find the associated /dev/sd${foo} devices to begin > > with, and therefore wouldn't need a label like dmm to prioritize it > > over say a generic device. > > > > > > - Eric > > > > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister > > wrote: > >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > >> Devtype to the ?correct? setting, for example: > >> > >> > >> > >> if [[ $osName = Linux ]] > >> > >> then > >> > >> : # Add function to discover disks in the Linux environment. > >> > >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > >> > >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > >> > >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > >> > >> fi > >> > >> > >> > >> My question is what is the correct setting? > >> > >> > >> > >> And what does GPFS do differently with each setting? > >> > >> > >> > >> Thanks, > >> > >> -Bryan > >> > >> > >> > >> From: gpfsug-discuss-bounces at spectrumscale.org > >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey > R. > >> Lang > >> Sent: Thursday, January 18, 2018 9:20 AM > >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > >> nsddevices file > >> > >> > >> > >> Note: External Email > >> > >> ________________________________ > >> > >> So is there a way to change it if it?s set incorrectly? > >> > >> > >> > >> jeff > >> > >> > >> > >> From: gpfsug-discuss-bounces at spectrumscale.org > >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim > Doherty > >> Sent: Wednesday, January 17, 2018 6:28 PM > >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > >> nsddevices file > >> > >> > >> > >> > >> > >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In > our > >> case the device is setup as dmm > >> > >> > >> > >> [root at service5 ~]# mmlsnsd -X > >> > >> Disk name NSD volume ID Device Devtype Node name > >> Remarks > >> ------------------------------------------------------------ > --------------------------------------- > >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume4 0972B6CE587CF625 /dev/dm-3 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume4 0972B6CE587CF625 /dev/dm-4 dmm > >> service6.pok.stglabs.ibm.com server node > >> > >> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata: > 0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6. > pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system: > service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > >> [root at service5 ~]# > >> > >> > >> > >> If you run an tspreparedisk -s it will show you all of the paths. > >> > >> > >> > >> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > >> 0972B6CD587CD8E0 /dev/sda generic > >> 0972B6CD587CD8E0 /dev/sdk generic > >> 0972B6CD587CD8E0 /dev/sdu generic > >> 0972B6CD587CD8E0 /dev/sdah generic > >> 0972B6CD587CD8E0 /dev/dm-0 dmm > >> [root at service5 ~]# > >> > >> > >> > >> Jim > >> > >> > >> > >> > >> > >> > >> > >> Jim > >> > >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > >> wrote: > >> > >> > >> > >> > >> > >> Hi all, > >> > >> > >> > >> We are reviewing some of our configurations and were not sure what to > make > >> of the NSD Device Types that GPFS uses and what, if anything, do they > change > >> about how GPFS accesses/recovers/manages/etc the underlying storage > based on > >> this setting. > >> > >> > >> > >> The documentation doesn?t say much about it other than to consult the > >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > >> section: > >> > >> > >> > >> # Known disk types currently are: > >> > >> # > >> > >> # powerdisk - EMC power path disk > >> > >> # vpath - IBM virtual path disk > >> > >> # dmm - Device-Mapper Multipath (DMM) > >> > >> # dlmfdrv - Hitachi dlm > >> > >> # hdisk - AIX hard disk > >> > >> # lv - AIX logical volume. Historical usage only. > >> > >> # Not allowed as a new device to mmcrnsd. > >> > >> # gpt - GPFS partition on Windows disk > >> > >> # generic - Device having no unique failover or multipathing > >> > >> # characteristic (predominantly Linux devices). > >> > >> # dasd - DASD device (for Linux on z Systems) > >> > >> > >> > >> We have our storage under Linux Device-Mapper Multipath control (two > device > >> paths to all storage, active/passive) and are accessible under > /dev/mapper, > >> but the NSD types are current set to ?generic? not ?dmm?. This is > >> configured in the /var/mmfs/etc/nsddevices file: > >> > >> > >> > >> if [[ $osName = Linux ]] > >> > >> then > >> > >> : # Add function to discover disks in the Linux environment. > >> > >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > >> > >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > >> > >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > >> > >> fi > >> > >> > >> > >> Can somebody from IBM explain what the correct setting should be and > what > >> differences GPFS does with ?generic? vs. ?dmm? vs. others? > >> > >> > >> > >> Thanks in advance! > >> > >> -Bryan > >> > >> > >> > >> ________________________________ > >> > >> > >> Note: This email is for the confidential use of the named addressee(s) > only > >> and may contain proprietary, confidential or privileged information. If > you > >> are not the intended recipient, you are hereby notified that any review, > >> dissemination or copying of this email is strictly prohibited, and to > please > >> notify the sender immediately and destroy this email and any > attachments. > >> Email transmission cannot be guaranteed to be secure or error-free. The > >> Company, therefore, does not make any guarantees as to the completeness > or > >> accuracy of this email or any attachments. This email is for > informational > >> purposes only and does not constitute a recommendation, offer, request > or > >> solicitation of any kind to buy, sell, subscribe, redeem or perform any > type > >> of transaction of a financial product. > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> > >> ________________________________ > >> > >> Note: This email is for the confidential use of the named addressee(s) > only > >> and may contain proprietary, confidential or privileged information. If > you > >> are not the intended recipient, you are hereby notified that any review, > >> dissemination or copying of this email is strictly prohibited, and to > please > >> notify the sender immediately and destroy this email and any > attachments. > >> Email transmission cannot be guaranteed to be secure or error-free. The > >> Company, therefore, does not make any guarantees as to the completeness > or > >> accuracy of this email or any attachments. This email is for > informational > >> purposes only and does not constitute a recommendation, offer, request > or > >> solicitation of any kind to buy, sell, subscribe, redeem or perform any > type > >> of transaction of a financial product. > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ________________________________ > > > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Fri Jan 19 16:10:14 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 19 Jan 2018 16:10:14 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: <1b8e86ededfb41de966d23df47b3439f@jumptrading.com> I was thinking that (SCSI3 Persistent Reserve) might be used with dmm after I saw something related to IBM RDAC software in this posting by Scott Fadden: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+%28GPFS%29/page/Device+Naming+and+Discovery+in+GPFS Thanks Jon, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jon Bernard Sent: Friday, January 19, 2018 9:57 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ A few years ago, Yuri noted that "dmm" has a special meaning to GPFS code. SCSI3 PR code keys off this disk type. https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014924464#77777777-0000-0000-0000-000014924464 https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#scsi3 Jon On Thu, Jan 18, 2018 at 3:22 PM, Eric Ross > wrote: I would say that's what it *appears* to be doing. The sys admin in me would still want some "official" word from IBM, as I wouldn't my assumption to be the cause of a meltdown at 16.50 on a Friday when everyone is heading out for the weekend ;) Cheers, -Eric On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister > wrote: > Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross > Sent: Thursday, January 18, 2018 2:01 PM > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file > > Note: External Email > ------------------------------------------------- > > Bryan, > > While waiting on the "official" word as to what each setting does > differently, I remembered there was a brief explanation of the > differences (at least of dmm vs generic) in the > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit > (https://github.com/finley/GPFS-Goodies) I use to use when I was at > IBM. > > //snip > > dmm vs. generic is used by GPFS to prioritize internal order of > searching through available disks, then later GPFS discards other > disk device names that it finds that match as the same NSD device > by a different path. For this reason, dmm vs. generic is an > important distinction if you are not explicitly producing the > entire and exclusive set of disks that GPFS should use, as output > from this script (nsddevices) _and_ exiting this script with a > "return 0". -Brian Finley > > //end snip > > If I read that correctly, it would indicate GPFS uses those additional > labels (at least dmm vs generic) as a mechanism to determine which > device files to prefer when scanning a system and finding the same NSD > available via different devices (i.e. /dev/mapper/foo vs > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed > device, I guess it would just ignore the /dev/sd${foo} devices when it > scans them. Also, the difference only seems to matter if you're not > explicitly creating a list a Brian F. indicates. If you simply > generate the list and exit (via return 0), GPFS wouldn't continue > scanning, and then find the associated /dev/sd${foo} devices to begin > with, and therefore wouldn't need a label like dmm to prioritize it > over say a generic device. > > > - Eric > > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister > > wrote: >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the >> Devtype to the ?correct? setting, for example: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> My question is what is the correct setting? >> >> >> >> And what does GPFS do differently with each setting? >> >> >> >> Thanks, >> >> -Bryan >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. >> Lang >> Sent: Thursday, January 18, 2018 9:20 AM >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> Note: External Email >> >> ________________________________ >> >> So is there a way to change it if it?s set incorrectly? >> >> >> >> jeff >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty >> Sent: Wednesday, January 17, 2018 6:28 PM >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> >> >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our >> case the device is setup as dmm >> >> >> >> [root at service5 ~]# mmlsnsd -X >> >> Disk name NSD volume ID Device Devtype Node name >> Remarks >> --------------------------------------------------------------------------------------------------- >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service5.pok.stglabs.ibm.com server node >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service6.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm >> service5.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm >> service6.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm >> service5.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm >> service6.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-3 dmm >> service5.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-4 dmm >> service6.pok.stglabs.ibm.com server node >> >> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: >> [root at service5 ~]# >> >> >> >> If you run an tspreparedisk -s it will show you all of the paths. >> >> >> >> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 >> 0972B6CD587CD8E0 /dev/sda generic >> 0972B6CD587CD8E0 /dev/sdk generic >> 0972B6CD587CD8E0 /dev/sdu generic >> 0972B6CD587CD8E0 /dev/sdah generic >> 0972B6CD587CD8E0 /dev/dm-0 dmm >> [root at service5 ~]# >> >> >> >> Jim >> >> >> >> >> >> >> >> Jim >> >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister >> > wrote: >> >> >> >> >> >> Hi all, >> >> >> >> We are reviewing some of our configurations and were not sure what to make >> of the NSD Device Types that GPFS uses and what, if anything, do they change >> about how GPFS accesses/recovers/manages/etc the underlying storage based on >> this setting. >> >> >> >> The documentation doesn?t say much about it other than to consult the >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this >> section: >> >> >> >> # Known disk types currently are: >> >> # >> >> # powerdisk - EMC power path disk >> >> # vpath - IBM virtual path disk >> >> # dmm - Device-Mapper Multipath (DMM) >> >> # dlmfdrv - Hitachi dlm >> >> # hdisk - AIX hard disk >> >> # lv - AIX logical volume. Historical usage only. >> >> # Not allowed as a new device to mmcrnsd. >> >> # gpt - GPFS partition on Windows disk >> >> # generic - Device having no unique failover or multipathing >> >> # characteristic (predominantly Linux devices). >> >> # dasd - DASD device (for Linux on z Systems) >> >> >> >> We have our storage under Linux Device-Mapper Multipath control (two device >> paths to all storage, active/passive) and are accessible under /dev/mapper, >> but the NSD types are current set to ?generic? not ?dmm?. This is >> configured in the /var/mmfs/etc/nsddevices file: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> Can somebody from IBM explain what the correct setting should be and what >> differences GPFS does with ?generic? vs. ?dmm? vs. others? >> >> >> >> Thanks in advance! >> >> -Bryan >> >> >> >> ________________________________ >> >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ________________________________ >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Fri Jan 19 21:38:03 2018 From: ewahl at osc.edu (Edward Wahl) Date: Fri, 19 Jan 2018 16:38:03 -0500 Subject: [gpfsug-discuss] policy ilm features? Message-ID: <20180119163803.79fddbeb@osc.edu> This one has been on my list a long time so I figured I'd ask here first before I open an apar or request an enhancement (most likely). Is there a way using the policy engine to determine the following? -metadata replication total/current -unbalanced file Looking to catch things like this that stand out on my filesystem without having to run several hundred million 'mmlsattr's. metadata replication: 1 max 2 flags: unbalanced Ed -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From makaplan at us.ibm.com Sat Jan 20 16:06:58 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sat, 20 Jan 2018 13:06:58 -0300 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180119163803.79fddbeb@osc.edu> References: <20180119163803.79fddbeb@osc.edu> Message-ID: Hint. RTFineManual, particularly the Admin guide, and look for MISC_ATTRIBUTES, Regarding metadata replication, one first has to ask, which metadata is replicated when and what if anything the mmchattr -m does or changes... https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+(GPFS)/page/Configuring+GPFS+for+Reliability --marc K. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hmorales at optimizeit.co Sun Jan 21 21:19:29 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Sun, 21 Jan 2018 16:19:29 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem Message-ID: Hello, I have a GPFS cluster with two filesystems. The disks associated to one filesystem reside on an old storage and the other filesystem disks reside on a much more modern storage system. I have successfully moved data from one fs to the other but there are questions about data integrity that still need verification so the old filesystem needs somehow to be preserved. My question is: Can I remove the old filesystem LUNs association to the NSDs servers without removing spectrum scale filesystems, so that later on, if necessary, I could associate them back and the old filesystem would be operating as normal? If possible: what would be the general steps to achieve this? ----- Thank you, Harold. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Sun Jan 21 21:35:44 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sun, 21 Jan 2018 21:35:44 +0000 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: <1d625007-9b14-a39b-5b30-027cb03a383c@strath.ac.uk> On 21/01/18 21:19, Harold Morales wrote: > Hello, > > I have a GPFS cluster with two filesystems. The disks associated to one > filesystem reside on an old storage and the other filesystem disks > reside on a much more modern storage system. I have successfully moved > data from one fs to the other but there are questions about data > integrity that still need verification so the old filesystem needs > somehow to be preserved. > > My question is: Can I remove the old filesystem LUNs association to the > NSDs servers without removing spectrum scale filesystems, so that later > on, if necessary, I could associate them back and the old filesystem > would be operating as normal? If possible: what would be the general > steps to achieve this? > I would take a look at mmexportfs/mmimportfs myself to be on the safe side. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From hmorales at optimizeit.co Sun Jan 21 23:35:01 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Sun, 21 Jan 2018 18:35:01 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: <1d625007-9b14-a39b-5b30-027cb03a383c@strath.ac.uk> References: <1d625007-9b14-a39b-5b30-027cb03a383c@strath.ac.uk> Message-ID: Thank you. The filesystems belong both to the same cluster. My question is whether LUNs can be unmapped from the host even though filesystems have not been configured and when they are mapped back the filesystem is going to operate correctly. El 21/01/2018 5:35 p.m., "Jonathan Buzzard" escribi?: On 21/01/18 21:19, Harold Morales wrote: > Hello, > > I have a GPFS cluster with two filesystems. The disks associated to one > filesystem reside on an old storage and the other filesystem disks reside > on a much more modern storage system. I have successfully moved data from > one fs to the other but there are questions about data integrity that still > need verification so the old filesystem needs somehow to be preserved. > > My question is: Can I remove the old filesystem LUNs association to the > NSDs servers without removing spectrum scale filesystems, so that later on, > if necessary, I could associate them back and the old filesystem would be > operating as normal? If possible: what would be the general steps to > achieve this? > > I would take a look at mmexportfs/mmimportfs myself to be on the safe side. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Sun Jan 21 23:43:47 2018 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Sun, 21 Jan 2018 23:43:47 +0000 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Mon Jan 22 03:41:58 2018 From: ulmer at ulmer.org (Stephen Ulmer) Date: Sun, 21 Jan 2018 22:41:58 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: Harold, The way I read your question, no one has actually answered it fully: You want to put the old file system in cold storage for forensic purposes ? exactly as it is. You want the NSDs to go away until and unless you need them in the future. QUICK AND DIRTY - YOU SHOULD NOT DO THIS: Set the old file system to NOT mount automatically. Make sure it is unmounted everywhere(!!!!!). Make sure none of those NSDs are being used as quorum tie-breakers, etc. Print your resume. Unmap the LUNs. This will leave all of the information ABOUT the filesystem in the Spectrum Scale configuration, but it won?t be available. You?ll get a bunch of errors that the NSDs are gone. Lots of them. It will make starting up Spectrum Scale take a long time while it looks for and fails to find them. That will be mostly noise. I?m not sure how you could corrupt the file system when the LUNs are no longer accessible and there can?t be any writes pending. I have done this recently to keep an old file system around after a migration (the customer required an "A/B" switch when installing new storage). This was with a very old version of GPFS that could not be upgraded. I would not do this unless the time frame to keep this configuration is very short ? I only did it because I didn?t have a choice. PROBABLY BETTER - READ THIS ONE TWICE: Take a look at mmexportfs (this was already suggested). The point of this command to to be able to carry the Spectrum Scale "configuration" for a file system AND the LUNs full of data around and plug them into a cluster later. I think a misunderstanding here led to your objection about this method: It doesn?t have to be a different cluster ? you can use the same one it came from. The advantage to this approach is that it leaves you with a more hygienic cluster ? no "missing" storage errors. The ONLY disadvantage that I can think of at the moment is that the file system configuration MIGHT have some lint removed during the import/export process. I?m not sure that *any* checking is done during the export, but I?d guess that the import involves at least some validation of the imported file. I hope. So if you think the file system *configuration* is wonky, you should call support and see if they will look at your export, or get a snap while you?ve still got the file system in your cluster. If you think that the *structure* of the file system on disk (or maybe just the copy method you?re using) might be wonky, then learn how to use mmexportfs. Note that the learning can certainly include asking questions here. When you want to look at the old file system again, you WILL have to re-map the LUNs, and use mmimportfs (and the file you made with mmexportfs). Then the file system will be part of your cluster again. STANDARD DISCLAIMERS APPLY: Your mileage may vary. Following this advice could cause nausea, vomiting, sweating, weight gain, sleeplessness, unemployment, weight loss, temporary tooth loss, redundancy, olfactory hallucinations, oozing (generalized), redundancy, uneven tire wear, excessive body oder, scoliosis, halitosis, ring worm, and intermittent acute ignorance. Good luck! :) Liberty, -- Stephen > On Jan 21, 2018, at 6:43 PM, Andrew Beattie > wrote: > > Harold, > > How big is the old file system, Spectrum Scale is going to throw a bunch of errors if you remove the luns from the old file system while attempting to keep the file system "data" on the luns. its likely to cause you all sorts of errors and potential data corruption. Its not something that I would recommend. > > Can you do a backup of the old filesystem so you can do a restore of data if you need to? > > Regards, > Andrew Beattie > Software Defined Storage - IT Specialist > Phone: 614-2133-7927 > E-mail: abeattie at au1.ibm.com > > > ----- Original message ----- > From: Harold Morales > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem > Date: Mon, Jan 22, 2018 7:20 AM > > Hello, > > I have a GPFS cluster with two filesystems. The disks associated to one filesystem reside on an old storage and the other filesystem disks reside on a much more modern storage system. I have successfully moved data from one fs to the other but there are questions about data integrity that still need verification so the old filesystem needs somehow to be preserved. > > My question is: Can I remove the old filesystem LUNs association to the NSDs servers without removing spectrum scale filesystems, so that later on, if necessary, I could associate them back and the old filesystem would be operating as normal? If possible: what would be the general steps to achieve this? > > ----- > Thank you, > > Harold. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=deVdfZ4CCXfI09tUiJGGP1c17jRhhwjx2TcB12uunoc&s=y3EUXWX1ecxWLR1HG0Ohwn9xsPKHZ6Pdodxoz44HV7A&e= > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon Jan 22 09:57:04 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 22 Jan 2018 09:57:04 +0000 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: <73d242a4-ee85-a9af-22c0-5c4fe23e3367@strath.ac.uk> On 22/01/18 03:41, Stephen Ulmer wrote: > Harold, > > The way I read your question, no one has actually answered it fully: > > You want to put the old file system in cold storage for forensic > purposes ? exactly as it is. You want the NSDs to go away until and > unless you need them in the future. > Hum, I would have said I did answer it fully even if I didn't go into detail. The only in my view sensible approach is to do mmexportfs so that should you need access to the data then you can get to it by reimporting it using mmimportfs. In the interim you can unmap all the NSD's from the cluster without causing GPFS to go care in the slightest. Otherwise you are doing something that is in my view at the best fool hardy and more generally down right idiotic. Personally I would outright refuse to do it without someone else higher up signing a written disclaimer that I was not responsible should it all go pear shaped. Note that the mmexportfs takes a few seconds at most to complete, and likewise with the mmimport. I am however somewhat perplexed by the data integrity side of the issue. If you suspect the data is corrupt in the old file system then having it around for reference purposes is surely not going to help. What are you going to do mount it again to find the file is corrupt; how is that going to help? If you suspect that whatever method you used to move the files from the old file system to the new one may have introduced corruption, then I would suggest that the best way out of that is to do an rsync with the -c flag so that it takes an MD5 checksum of the files on both file systems. Once that is complete you can safely ditch the old file system completely knowing that you have recovered it as best as possible. You could probably kick a bunch of rsyncs of in parallel to speed this method up. In fact a "rsync -c" would be a gold standard of look it's absolutely the same on the old and new file systems and remove all doubts about the transfer introducing corruption. At that point if someone comes and says your transfer corrupted my files, you can with justification say they where corrupt in the old file system and I have done my best to transfer them over. Note if any user was deliberately creating MD5 collisions then they get everything they deserve in my view, and accidental collisions in MD5 are about as likely as ?. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From john.hearns at asml.com Mon Jan 22 08:29:42 2018 From: john.hearns at asml.com (John Hearns) Date: Mon, 22 Jan 2018 08:29:42 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180119163803.79fddbeb@osc.edu> References: <20180119163803.79fddbeb@osc.edu> Message-ID: Ed, This is not a perfect answer. You need to look at policies for this. I have been doing something similar recently. Something like: RULE 'list_file' EXTERNAL LIST 'all-files' EXEC '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files' SHOW( varchar(kb_allocated) || ' ' || varchar(file_size) || ' ' || varchar(misc_attributes) || ' ' || name || ' ' || fileset_name ) WHERE REGEX(misc_attributes,'[J]') So this policy shows the kbytes allocates, file size, the miscellaneous attributes, name and fileset name For all files with miscellaneous attributes of 'J' which means 'Some data blocks might be ill replicated' -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Edward Wahl Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] policy ilm features? This one has been on my list a long time so I figured I'd ask here first before I open an apar or request an enhancement (most likely). Is there a way using the policy engine to determine the following? -metadata replication total/current -unbalanced file Looking to catch things like this that stand out on my filesystem without having to run several hundred million 'mmlsattr's. metadata replication: 1 max 2 flags: unbalanced Ed -- Ed Wahl Ohio Supercomputer Center 614-292-9302 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D&reserved=0 -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. From yguvvala at cambridgecomputer.com Mon Jan 22 21:13:15 2018 From: yguvvala at cambridgecomputer.com (Yugendra Guvvala) Date: Mon, 22 Jan 2018 16:13:15 -0500 Subject: [gpfsug-discuss] Excluding Filesets on snapshots In-Reply-To: References: Message-ID: Thank you that works, we have to specify root fileset to get a snapshot. not specifiying root:snapshotname will include all all file sets. example : mmcrsnapshot device root:snapshot_name On Fri, Jan 19, 2018 at 3:07 AM, Markus Rohwedder wrote: > Hello Yugendra, > > Snapshots are taken per Inode space. Both the root fileset and the > independent fileset you created own separate inode spaces, > even if the independent fileset is nested in the root fileset. > > Taking a snapshot of the root fileset should exclude the independent > fileset. > e.g: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.03.52' -j 'root' > > Taking a filesystem snapshot however would include all inode spaces of the > file system and would therefore include the independent fileset. > e.g.: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.04.47' > > > Mit freundlichen Gr??en / Kind regards > > *Dr. Markus Rohwedder* > > Spectrum Scale GUI Development > ------------------------------ > Phone: +49 7034 6430190 <+49%207034%206430190> IBM Deutschland > E-Mail: rohwedder at de.ibm.com Am Weiher 24 > 65451 Kelsterbach > Germany > ------------------------------ > IBM Deutschland Research & Development GmbH / Vorsitzender des > Aufsichtsrats: Martina K?deritz > Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, > HRB 243294 > > [image: Inactive hide details for Yugendra Guvvala ---01/18/2018 11:11:55 > PM---Hi, We have 5 dependent Filesets linked to root fileset]Yugendra > Guvvala ---01/18/2018 11:11:55 PM---Hi, We have 5 dependent Filesets linked > to root fileset (mount point) and we > > From: Yugendra Guvvala > To: gpfsug main discussion list > Date: 01/18/2018 11:11 PM > Subject: [gpfsug-discuss] Excluding Filesets on snapshots > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi, > > We have 5 dependent Filesets linked to root fileset (mount point) and we > are creating a new independent Fileset which will be linked to root fileset > (mount point). We want to take snapshots of the dependent filesets and > independent fileset on different periodic intervals. Is there a way to > exclude independent fileset and take snapshot of all dependent filesets ? > > > -- > Thanks, > Yugi > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r= > IbxtjdkPAM2Sbon4Lbbi4w&m=vZOWTPN66Z-dgh-Xs1VV4BI9dhIVi3BUFi7Zt1gPE2E&s= > 3HAnqhprHHsQe1cvwJIYK4cFJwG7BcNK6hKxXaDQ7sA&e= > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Thanks, *Yugendra Guvvala | HPC Technologist ** |** Cambridge Computer ** |** "Artists in Data Storage" * Direct: 781-250-3273 | Cell: 806-773-4464 | yguvvala at cambridgecomputer.com | www.cambridgecomputer.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14162451.gif Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From hmorales at optimizeit.co Tue Jan 23 04:23:47 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Mon, 22 Jan 2018 23:23:47 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: <73d242a4-ee85-a9af-22c0-5c4fe23e3367@strath.ac.uk> References: <73d242a4-ee85-a9af-22c0-5c4fe23e3367@strath.ac.uk> Message-ID: Thanks Stephen and Jonathan for pointing this important things out. Now I begin to understand the importance of mmimportfs/mmexportfs for operations like the one I discuss here. Will let you know how it goes. 2018-01-22 4:57 GMT-05:00 Jonathan Buzzard : > On 22/01/18 03:41, Stephen Ulmer wrote: > >> Harold, >> >> The way I read your question, no one has actually answered it fully: >> >> You want to put the old file system in cold storage for forensic purposes >> ? exactly as it is. You want the NSDs to go away until and unless you need >> them in the future. >> >> > Hum, I would have said I did answer it fully even if I didn't go into > detail. The only in my view sensible approach is to do mmexportfs so that > should you need access to the data then you can get to it by reimporting it > using mmimportfs. In the interim you can unmap all the NSD's from the > cluster without causing GPFS to go care in the slightest. > > Otherwise you are doing something that is in my view at the best fool > hardy and more generally down right idiotic. Personally I would outright > refuse to do it without someone else higher up signing a written disclaimer > that I was not responsible should it all go pear shaped. Note that the > mmexportfs takes a few seconds at most to complete, and likewise with the > mmimport. > > I am however somewhat perplexed by the data integrity side of the issue. > If you suspect the data is corrupt in the old file system then having it > around for reference purposes is surely not going to help. What are you > going to do mount it again to find the file is corrupt; how is that going > to help? > > If you suspect that whatever method you used to move the files from the > old file system to the new one may have introduced corruption, then I would > suggest that the best way out of that is to do an rsync with the -c flag so > that it takes an MD5 checksum of the files on both file systems. Once that > is complete you can safely ditch the old file system completely knowing > that you have recovered it as best as possible. You could probably kick a > bunch of rsyncs of in parallel to speed this method up. > > In fact a "rsync -c" would be a gold standard of look it's absolutely the > same on the old and new file systems and remove all doubts about the > transfer introducing corruption. At that point if someone comes and says > your transfer corrupted my files, you can with justification say they where > corrupt in the old file system and I have done my best to transfer them > over. Note if any user was deliberately creating MD5 collisions then they > get everything they deserve in my view, and accidental collisions in MD5 > are about as likely as ?. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chetkulk at in.ibm.com Tue Jan 23 09:30:10 2018 From: chetkulk at in.ibm.com (Chetan R Kulkarni) Date: Tue, 23 Jan 2018 15:00:10 +0530 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: Message-ID: Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Tue Jan 23 11:10:57 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 23 Jan 2018 11:10:57 +0000 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: Message-ID: Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services to halt/timeout/fail/stop/go slow/cause an earthquake or is it completely transparent? My existing config is: FILE access configuration : AD PARAMETERS VALUES ------------------------------------------------- ENABLE_NFS_KERBEROS true SERVERS server.domain USER_NAME username NETBIOS_NAME store IDMAP_ROLE master IDMAP_RANGE 10000000-299999999 IDMAP_RANGE_SIZE 1000000 UNIXMAP_DOMAINS DOMAIN(500 - 2000000) LDAPMAP_DOMAINS none OBJECT access not configured PARAMETERS VALUES Enabled services: SMB SMB is running Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Chetan R Kulkarni Sent: 23 January 2018 09:30 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. [Inactive hide details for "Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a f]"Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nf From: "Sobey, Richard A" > To: "'gpfsug-discuss at spectrumscale.org'" > Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 23 17:16:36 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 23 Jan 2018 17:16:36 +0000 Subject: [gpfsug-discuss] Metadata only system pool Message-ID: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_johnson at brown.edu Tue Jan 23 17:23:59 2018 From: david_johnson at brown.edu (david_johnson at brown.edu) Date: Tue, 23 Jan 2018 12:23:59 -0500 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: If the new files need indirect blocks or extended attributes that don?t fit in the basic inode, additional metadata space would need to be allocated. There might be other reasons but these come to mind immediately. -- ddj Dave Johnson > On Jan 23, 2018, at 12:16 PM, Buterbaugh, Kevin L wrote: > > Hi All, > > I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: > > Inode Information > ----------------- > Number of used inodes: 218635454 > Number of free inodes: 131364674 > Number of allocated inodes: 350000128 > Maximum number of inodes: 350000128 > > I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. > > Now my system pool is almost ?full?: > > (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) > > But again, what - outside of me creating more inodes - would cause that to change?? > > Thanks? > > Kevin > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Jan 23 17:25:11 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 23 Jan 2018 17:25:11 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: <00afb095d1314045963e740f624882b8@jumptrading.com> Directory entries are also stored in the system pool and existing directories must grow in size (indirect blocks) to hold additional files/dir references as new files/dirs are created. That would be my initial guess anyways. The ?pool total? output below seems to indicate that you only have 34Mb of free space? that is definitely not good, -B From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L Sent: Tuesday, January 23, 2018 11:17 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] Metadata only system pool Note: External Email ________________________________ Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Tue Jan 23 17:25:48 2018 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 23 Jan 2018 12:25:48 -0500 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at calicolabs.com Tue Jan 23 17:27:57 2018 From: alex at calicolabs.com (Alex Chekholko) Date: Tue, 23 Jan 2018 09:27:57 -0800 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: 2.8TB seems quite high for only 350M inodes. Are you sure you only have metadata in there? On Tue, Jan 23, 2018 at 9:25 AM, Frederick Stock wrote: > One possibility is the creation/expansion of directories or allocation of > indirect blocks for large files. > > Not sure if this is the issue here but at one time inode allocation was > considered slow and so folks may have pre-allocated inodes to avoid that > overhead during file creation. To my understanding inode creation time is > not so slow that users need to pre-allocate inodes. Yes, there are likely > some applications where pre-allocating may be necessary but I expect they > would be the exception. I mention this because you have a lot of free > inodes and of course once they are allocated they cannot be de-allocated. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821> > stockf at us.ibm.com > > > > From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/23/2018 12:17 PM > Subject: [gpfsug-discuss] Metadata only system pool > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi All, > > I was under the (possibly false) impression that if you have a filesystem > where the system pool contains metadata only then the only thing that would > cause the amount of free space in that pool to change is the creation of > more inodes ? is that correct? In other words, given that I have a > filesystem with 130 million free (but allocated) inodes: > > Inode Information > ----------------- > Number of used inodes: 218635454 > Number of free inodes: 131364674 > Number of allocated inodes: 350000128 > Maximum number of inodes: 350000128 > > I would not expect that a user creating a few hundred or thousands of > files could cause a ?no space left on device? error (which I?ve got one > user getting). There?s plenty of free data space, BTW. > > Now my system pool is almost ?full?: > > (pool total) 2.878T 34M ( 0%) > 140.9M ( 0%) > > But again, what - outside of me creating more inodes - would cause that to > change?? > > Thanks? > > Kevin > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > *Kevin.Buterbaugh at vanderbilt.edu* - > (615)875-9633 <(615)%20875-9633> > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m= > gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_ > vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cphoffma at uoregon.edu Tue Jan 23 17:27:52 2018 From: cphoffma at uoregon.edu (Chris Hoffman) Date: Tue, 23 Jan 2018 17:27:52 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu>, Message-ID: <1516728472608.73429@uoregon.edu> If you are still running out of space when mmdf shows preallocated inodes left I'd check MaxInodes vs AllocInodes for that fileset. You can run: mmlsfileset -L Thanks, Chris? ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederick Stock Sent: Tuesday, January 23, 2018 9:25 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Metadata only system pool One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ... is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a "no space left on device" error (which I've got one user getting). There's plenty of free data space, BTW. Now my system pool is almost "full": (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks... Kevin - Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 23 17:37:51 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 23 Jan 2018 17:37:51 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: Hi All, I do have metadata replication set to two, so Alex, does that make more sense? And I had forgotten about indirect blocks for large files, which actually makes sense with the user in question ? my apologies for that ? due to a very gravely ill pet and a recovering at home from pneumonia family member I?m way more sleep deprived right now than I?d like. :-( Fred - I think you?ve already answered this ? but mmchfs can only create / allocate more inodes ? it cannot be used to shrink the number of inodes? That would make sense, and if that?s the case then I can allocate more NSDs to the system pool. Thanks? Kevin On Jan 23, 2018, at 11:27 AM, Alex Chekholko > wrote: 2.8TB seems quite high for only 350M inodes. Are you sure you only have metadata in there? On Tue, Jan 23, 2018 at 9:25 AM, Frederick Stock > wrote: One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list > Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C1607a3fe872e4241587b08d56286a746%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636523252830007825&sdata=rIFx3lzbAIH5SZtFxJsVqWMMSo%2F0LssNc4K4tZH3uQc%3D&reserved=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Tue Jan 23 18:18:43 2018 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 23 Jan 2018 13:18:43 -0500 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: You are correct about mmchfs, you can increase the inode maximum but once an inode is allocated it cannot be de-allocated in the sense that the space can be recovered. You can of course decreased the inode maximum to a value equal to the used and allocated inodes but that would not help you here. Providing more metadata space via additional NSDs seems your most expedient option to address the issue. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 01:10 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I do have metadata replication set to two, so Alex, does that make more sense? And I had forgotten about indirect blocks for large files, which actually makes sense with the user in question ? my apologies for that ? due to a very gravely ill pet and a recovering at home from pneumonia family member I?m way more sleep deprived right now than I?d like. :-( Fred - I think you?ve already answered this ? but mmchfs can only create / allocate more inodes ? it cannot be used to shrink the number of inodes? That would make sense, and if that?s the case then I can allocate more NSDs to the system pool. Thanks? Kevin On Jan 23, 2018, at 11:27 AM, Alex Chekholko wrote: 2.8TB seems quite high for only 350M inodes. Are you sure you only have metadata in there? On Tue, Jan 23, 2018 at 9:25 AM, Frederick Stock wrote: One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C1607a3fe872e4241587b08d56286a746%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636523252830007825&sdata=rIFx3lzbAIH5SZtFxJsVqWMMSo%2F0LssNc4K4tZH3uQc%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=LN14zBEOYVrP2YRk3of_f08Ok5f256m7SYf1xL0qDvU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From UWEFALKE at de.ibm.com Tue Jan 23 19:03:40 2018 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Tue, 23 Jan 2018 20:03:40 +0100 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Tue Jan 23 19:12:50 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 23 Jan 2018 16:12:50 -0300 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: If one were starting over, it might make sense to use a smaller inode size. I believe we still support 512, 1K, 2K. Tradeoff with the fact that inodes can store data and EAs. From: "Uwe Falke" To: gpfsug main discussion list Date: 01/23/2018 04:04 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 23 19:25:54 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 23 Jan 2018 19:25:54 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: <9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> Hi All, This is all making sense and I appreciate everyone?s responses ? and again I apologize for not thinking about the indirect blocks. Marc - we specifically chose 4K inodes when we created this filesystem a little over a year ago so that small files could fit in the inode and therefore be stored on the metadata SSDs. This is more of a curiosity question ? is it documented somewhere how a 4K inode is used? I understand that for very small files up to 3.5K of that can be for data, but what about for large files? I.e., how much of that 4K is used for block addresses (3.5K plus whatever portion was already allocated to block addresses??) ? or what I?m really asking is, given 4K inodes and a 1M block size how big does a file have to be before it will need to use indirect blocks? Thanks again? Kevin On Jan 23, 2018, at 1:12 PM, Marc A Kaplan > wrote: If one were starting over, it might make sense to use a smaller inode size. I believe we still support 512, 1K, 2K. Tradeoff with the fact that inodes can store data and EAs. From: "Uwe Falke" > To: gpfsug main discussion list > Date: 01/23/2018 04:04 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C77fefde14ec54e04b35708d5629550bd%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636523315820548341&sdata=rbazh5e%2BxgHGvgF65VHTs9Hf4kk9EtUizsb19l5rr7U%3D&reserved=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From UWEFALKE at de.ibm.com Tue Jan 23 20:35:56 2018 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Tue, 23 Jan 2018 21:35:56 +0100 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> <9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> Message-ID: Hi, Kevin, (reproducing mainly what I've learned from Yuri) a 512B inode can hold about 32 Disk Addresses (DAs) addressing 32 physical disk blocks (but mind: if max replication - not necessarily actual repl. -- is set to more than 1, that mans for example 16 logical blocks (R=2). A DA is 12 bytes. With a 4k inode you can expect the other 3.5kiB to contain also DAs throughout (about 298) So with R=2 (even if actual replication is r=1), a 4k inode can hold max about 330 DAs addressing 165 logical blocks The max file size w/out indirect addressing is thus 165 x blocksize (e.g. 1320MiB @8MiB) However, there could be other data structures in an inode (EAs) reducing the space available for DAs Hence, YMMV Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 08:26 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, This is all making sense and I appreciate everyone?s responses ? and again I apologize for not thinking about the indirect blocks. Marc - we specifically chose 4K inodes when we created this filesystem a little over a year ago so that small files could fit in the inode and therefore be stored on the metadata SSDs. This is more of a curiosity question ? is it documented somewhere how a 4K inode is used? I understand that for very small files up to 3.5K of that can be for data, but what about for large files? I.e., how much of that 4K is used for block addresses (3.5K plus whatever portion was already allocated to block addresses??) ? or what I?m really asking is, given 4K inodes and a 1M block size how big does a file have to be before it will need to use indirect blocks? Thanks again? Kevin On Jan 23, 2018, at 1:12 PM, Marc A Kaplan wrote: If one were starting over, it might make sense to use a smaller inode size. I believe we still support 512, 1K, 2K. Tradeoff with the fact that inodes can store data and EAs. From: "Uwe Falke" To: gpfsug main discussion list Date: 01/23/2018 04:04 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C77fefde14ec54e04b35708d5629550bd%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636523315820548341&sdata=rbazh5e%2BxgHGvgF65VHTs9Hf4kk9EtUizsb19l5rr7U%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Tue Jan 23 21:13:20 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 23 Jan 2018 18:13:20 -0300 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu><9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> Message-ID: One way to know for sure it to do some experiments and dump some inodes with the command tsdbfs filesystem_name inode inode_number Of course improper use of tsdbfs command can destroy or corrupt your filesystems. So I take no responsibility for that. To stay safe, only use it on test filesystems you can afford to lose. (If you are root `dd if=/dev/zero of=/dev/some-disk-you-should-not-wipe` is also something you should not do!) -------------- next part -------------- An HTML attachment was scrubbed... URL: From hmorales at optimizeit.co Wed Jan 24 01:39:52 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Tue, 23 Jan 2018 20:39:52 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale Message-ID: Hello, Is there any proven strategy to replicate disks from one local storage to one storage on another site and have this as a spectrum scale backup strategy. Let me explain: Context: - site A simple three node cluster configured (AIX nodes). two failure groups. two filesystems. No quota, no ILM. - Site B the same cluster config as in site A configured (AIX nodes) same hdisk device names used as in site A for each and every NSDs - NO TCPIP connection between those two sites. Same ip addresses available locally on each one. and same non-scale based filesystems. Same scale version installed. (esentially site B is a copy from site A). My question is, can I replicate the spectrum scale disks from site A to site B storage to another (HP 3PAR) and maintain consistency between scale sites? can this be considered a valid DR configuration? People here don't want a stretch cluster (without any apparent reason, which talks very bad about them). Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Jan 24 06:22:33 2018 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Wed, 24 Jan 2018 06:22:33 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: How are you proposing to replicate the data between the sites? You mention no IP connection, so were you planning some other form of data shipping? I would suggest it would be a very high risk strategy to try this. Stretched cluster or ASYNC-DR are more appropriate for this type of DR solution. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of hmorales at optimizeit.co [hmorales at optimizeit.co] Sent: 24 January 2018 01:39 To: gpfsug main discussion list Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale Hello, Is there any proven strategy to replicate disks from one local storage to one storage on another site and have this as a spectrum scale backup strategy. Let me explain: Context: - site A simple three node cluster configured (AIX nodes). two failure groups. two filesystems. No quota, no ILM. - Site B the same cluster config as in site A configured (AIX nodes) same hdisk device names used as in site A for each and every NSDs - NO TCPIP connection between those two sites. Same ip addresses available locally on each one. and same non-scale based filesystems. Same scale version installed. (esentially site B is a copy from site A). My question is, can I replicate the spectrum scale disks from site A to site B storage to another (HP 3PAR) and maintain consistency between scale sites? can this be considered a valid DR configuration? People here don't want a stretch cluster (without any apparent reason, which talks very bad about them). Thanks. From hmorales at optimizeit.co Wed Jan 24 06:33:10 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Wed, 24 Jan 2018 01:33:10 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Wed Jan 24 07:03:46 2018 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 24 Jan 2018 07:03:46 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: Message-ID: Hi They are not even open for a routed L3 network? How they talk between DCs today then? I would really go L3 and AFM here if the sole purpose here is to have a DR -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous > On 24 Jan 2018, at 8.33, Harold Morales wrote: > > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Wed Jan 24 07:07:46 2018 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 24 Jan 2018 07:07:46 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm ? Seems to cover what you?re looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales : > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > IP connection between storage boxes but not between servers on both sites, > there is a dark fiber connecting both sites. Here they dont want to explore > the idea of a scaled-based. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alevin at gmail.com Wed Jan 24 07:30:15 2018 From: alevin at gmail.com (Alex Levin) Date: Wed, 24 Jan 2018 02:30:15 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Hi, We are using a similar type of replication. I assume the site B is the cold site prepared for DR The storage layer is EMC VMAX and the LUNs are replicated with SRDF. All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication group to ensure consistency. The cluster name, IP addresses , hostnames of the cluster nodes are different on another site - it can be a pre-configured cluster without gpfs filesystems or with another filesystem. Same names and addresses shouldn't be a problem. Additionally to the replicated LUNs/NSDs you need to deliver copy of /var/mmfs/gen/mmsdrfs file from A to B site. There is no need to replicate it in real-time, only after the change of the cluster configuration. To activate site B - present replicated LUNs to the nodes in the DR cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" Tested with multiples LUNs and filesystems on various workloads - seems to be working --Alex On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales wrote: > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > IP connection between storage boxes but not between servers on both sites, > there is a dark fiber connecting both sites. Here they dont want to explore > the idea of a scaled-based. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chetkulk at in.ibm.com Wed Jan 24 09:37:01 2018 From: chetkulk at in.ibm.com (Chetan R Kulkarni) Date: Wed, 24 Jan 2018 15:07:01 +0530 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: Message-ID: Hi, One can't enable/disable smb/nfs service if file authentication is already configured. Hence, with your already existing config; you can't enable NFS service directly. You need to re-configure file authentication (i.e. remove file auth, enable nfs service; configure file auth). May be following sequence of commands will help you. mmuserauth service remove --data-access-method file mmces service enable nfs mmces service list -a # check nfs is enabled and running mmuserauth service create --data-access-method file --type ad ..... # please complete this command as per your set up details (the way you already configured) FYI, Enabling NFS service won't affect other services. Thanks, Chetan. From: "Sobey, Richard A" To: gpfsug main discussion list Date: 01/23/2018 04:42 PM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services to halt/timeout/fail/stop/go slow/cause an earthquake or is it completely transparent? My existing config is: FILE access configuration : AD PARAMETERS VALUES ------------------------------------------------- ENABLE_NFS_KERBEROS true SERVERS server.domain USER_NAME username NETBIOS_NAME store IDMAP_ROLE master IDMAP_RANGE 10000000-299999999 IDMAP_RANGE_SIZE 1000000 UNIXMAP_DOMAINS DOMAIN(500 - 2000000) LDAPMAP_DOMAINS none OBJECT access not configured PARAMETERS VALUES Enabled services: SMB SMB is running Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Chetan R Kulkarni Sent: 23 January 2018 09:30 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. Inactive hide details for "Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a f"Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nf From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=mn6WfAqcjuBRZT4DaqmRBJ7KKacO2ma_baqc0GPm0PU&s=7ItSqugWx9SDxGAB2Odvj6oWUoFCiCzOH7cHdvRszY4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Wed Jan 24 16:13:01 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 24 Jan 2018 16:13:01 +0000 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , Message-ID: Gosh.. Seriously? I need downtime to enable NFS? Can that change in future releases? I have major issues configuring Ad file auth and I would not be confident getting my SMB working again in a timely manner. Thanks for letting me know anyway. Get Outlook for Android ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Chetan R Kulkarni Sent: Wednesday, January 24, 2018 9:37:01 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, One can't enable/disable smb/nfs service if file authentication is already configured. Hence, with your already existing config; you can't enable NFS service directly. You need to re-configure file authentication (i.e. remove file auth, enable nfs service; configure file auth). May be following sequence of commands will help you. mmuserauth service remove --data-access-method file mmces service enable nfs mmces service list -a # check nfs is enabled and running mmuserauth service create --data-access-method file --type ad ..... # please complete this command as per your set up details (the way you already configured) FYI, Enabling NFS service won't affect other services. Thanks, Chetan. [Inactive hide details for "Sobey, Richard A" ---01/23/2018 04:42:33 PM---Many thanks Chetan. Next question, does enabling the N]"Sobey, Richard A" ---01/23/2018 04:42:33 PM---Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services From: "Sobey, Richard A" To: gpfsug main discussion list Date: 01/23/2018 04:42 PM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services to halt/timeout/fail/stop/go slow/cause an earthquake or is it completely transparent? My existing config is: FILE access configuration : AD PARAMETERS VALUES ------------------------------------------------- ENABLE_NFS_KERBEROS true SERVERS server.domain USER_NAME username NETBIOS_NAME store IDMAP_ROLE master IDMAP_RANGE 10000000-299999999 IDMAP_RANGE_SIZE 1000000 UNIXMAP_DOMAINS DOMAIN(500 - 2000000) LDAPMAP_DOMAINS none OBJECT access not configured PARAMETERS VALUES Enabled services: SMB SMB is running Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Chetan R Kulkarni Sent: 23 January 2018 09:30 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. [Inactive hide details for "Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a f]"Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nf From: "Sobey, Richard A" > To: "'gpfsug-discuss at spectrumscale.org'" > Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=mn6WfAqcjuBRZT4DaqmRBJ7KKacO2ma_baqc0GPm0PU&s=7ItSqugWx9SDxGAB2Odvj6oWUoFCiCzOH7cHdvRszY4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From valdis.kletnieks at vt.edu Wed Jan 24 16:28:21 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 24 Jan 2018 11:28:21 -0500 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , Message-ID: <17153.1516811301@turing-police.cc.vt.edu> On Wed, 24 Jan 2018 16:13:01 +0000, "Sobey, Richard A" said: > Gosh.. Seriously? I need downtime to enable NFS? Wait till you get to the part where 'mmnfs export change /foo/whatever --nfsadd (whatever)' bounces your nfs-ganesha services on all protocol nodes - at the same time. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From valdis.kletnieks at vt.edu Wed Jan 24 16:31:26 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 24 Jan 2018 11:31:26 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: <17310.1516811486@turing-police.cc.vt.edu> On Tue, 23 Jan 2018 20:39:52 -0500, Harold Morales said: > - site A simple three node cluster configured (AIX nodes). two failure > groups. two filesystems. No quota, no ILM. > - Site B the same cluster config as in site A configured (AIX nodes) same > hdisk device names used as in site A for each and every NSDs > - NO TCPIP connection between those two sites. Same ip addresses available > locally on each one. and same non-scale based filesystems. Same scale > version installed. (esentially site B is a copy from site A). You're going to have a *really* bad time with semantic stability at site B, as while it's reading the replica from site A, it will continually be reading metadata and data blocks that it didn't write there. (Thought experiment - what happens when site B thinks there's 3 files in a directory, and a replication block from A changes that to 2 without B's knowledge?) Be ready to be dealing with filesystem issues at B - it's *always* bad karma to do writes to a filesystem that the operating system doesn't know about.... If B is a *cold* backup site, this should work just fine and you can ignore all this. But if B has the filesystem mounted while A is making changes, it *will* go pear shaped on you. > People here don't want a stretch cluster (without any apparent reason, which > talks very bad about them). For what it's worth, we're running a stretch cluster with about 95 cable miles separating the two halves. Biggest problem we have is that a network hiccup can cause the cluster to partition and down the NSD's at the remote site (yes, it's designed that way - 5 nodes at each site, and on a partition the main site has enough quorum nodes to stay up, but the remote site doesn't. So it downs the NSDs' there. Just means a 'mmchdisk start -a' once the cluster recovers all the nodes at the far end (which is a lot less painful than the alternative, which is a split-brain situation). (And yeah, we 're working on getting reliable alternate network connections. Just a bit hard to find another 10G link to light up that isn't fate-sharing *and* plays nice with the rest of our BGP routing and can be fixed to have failover fast enough for GPFS to not notice the blink. We're in the part of Virginia that doesn't have dark fiber all over the place.) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From stockf at us.ibm.com Wed Jan 24 16:36:55 2018 From: stockf at us.ibm.com (Frederick Stock) Date: Wed, 24 Jan 2018 11:36:55 -0500 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: <17153.1516811301@turing-police.cc.vt.edu> References: , <17153.1516811301@turing-police.cc.vt.edu> Message-ID: Thankfully the issue of Ganesha being restarted across the cluster when an export is changed has been addressed in the 5.0 release of Spectrum Scale. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list Date: 01/24/2018 11:34 AM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org On Wed, 24 Jan 2018 16:13:01 +0000, "Sobey, Richard A" said: > Gosh.. Seriously? I need downtime to enable NFS? Wait till you get to the part where 'mmnfs export change /foo/whatever --nfsadd (whatever)' bounces your nfs-ganesha services on all protocol nodes - at the same time. [attachment "attxn3je.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Wed Jan 24 19:16:44 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 24 Jan 2018 19:16:44 +0000 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , <17153.1516811301@turing-police.cc.vt.edu>, Message-ID: I saw that yes. Maybe I will wait until I upgrade to v5 and sort out NFS then - one downtime window to rule them all. Richard Get Outlook for Android ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederick Stock Sent: Wednesday, January 24, 2018 4:36:55 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Thankfully the issue of Ganesha being restarted across the cluster when an export is changed has been addressed in the 5.0 release of Spectrum Scale. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list Date: 01/24/2018 11:34 AM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ On Wed, 24 Jan 2018 16:13:01 +0000, "Sobey, Richard A" said: > Gosh.. Seriously? I need downtime to enable NFS? Wait till you get to the part where 'mmnfs export change /foo/whatever --nfsadd (whatever)' bounces your nfs-ganesha services on all protocol nodes - at the same time. [attachment "attxn3je.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Wed Jan 24 19:31:41 2018 From: chair at spectrumscale.org (Simon Thompson (Spectrum Scale UG Chair)) Date: Wed, 24 Jan 2018 19:31:41 +0000 Subject: [gpfsug-discuss] UK April Meeting Agenda Message-ID: <7dd288e6-679e-4bd8-9803-b5f21607e92a@email.android.com> An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Wed Jan 24 20:32:00 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 24 Jan 2018 15:32:00 -0500 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , <17153.1516811301@turing-police.cc.vt.edu> Message-ID: <50389.1516825920@turing-police.cc.vt.edu> On Wed, 24 Jan 2018 11:36:55 -0500, "Frederick Stock" said: > Thankfully the issue of Ganesha being restarted across the cluster when an > export is changed has been addressed in the 5.0 release of Spectrum Scale. Thanks for the info. Now all I need is a version of LTFS/EE that supports 5.0. :) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From janfrode at tanso.net Thu Jan 25 08:27:37 2018 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 25 Jan 2018 09:27:37 +0100 Subject: [gpfsug-discuss] AFM and RHEL 7.4 Message-ID: The FAQ has a note stating: 1. AFM, Asynch Disaster Recovery with AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4. Could someone please clarify this sentence ? It can't be right that none of these features are supported with RHEL 7.4, or .. ? -jf -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Thu Jan 25 09:53:06 2018 From: john.hearns at asml.com (John Hearns) Date: Thu, 25 Jan 2018 09:53:06 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Jan Frode, thankyou for that link. I have a general question regarding remote GPFS filesystems. If we have two clusters, in separate locations on separate Infiniband fabrics, we can set up a remote relationship between filesystems. As Valdis discusses, what happens if the IP link between the clusters goes down or is unstable? Can nodes in one cluster vote out nodes in the other cluster? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: Wednesday, January 24, 2018 8:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm ? Seems to cover what you?re looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales >: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Thu Jan 25 10:08:38 2018 From: YARD at il.ibm.com (Yaron Daniel) Date: Thu, 25 Jan 2018 12:08:38 +0200 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Hi You can do remote mount between GPFS clusters. https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_admmcch.htm Where is you daemon communication network ? IP ir IP Over IB ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: John Hearns To: gpfsug main discussion list Date: 01/25/2018 11:53 AM Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Sent by: gpfsug-discuss-bounces at spectrumscale.org Jan Frode, thankyou for that link. I have a general question regarding remote GPFS filesystems. If we have two clusters, in separate locations on separate Infiniband fabrics, we can set up a remote relationship between filesystems. As Valdis discusses, what happens if the IP link between the clusters goes down or is unstable? Can nodes in one cluster vote out nodes in the other cluster? From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: Wednesday, January 24, 2018 8:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm ? Seems to cover what you?re looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales : Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=BMyrARi4hlfjG-EugDznaiyWSqErGF5FyFpQQ-o4ScU&s=R0w70yvIjZaXpcs4P2mGJNQYBSNlUp3aZcCNks-sveU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4376 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 5093 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4746 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4557 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 11294 bytes Desc: not available URL: From Achim.Rehor at de.ibm.com Thu Jan 25 10:20:41 2018 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Thu, 25 Jan 2018 11:20:41 +0100 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: From john.hearns at asml.com Thu Jan 25 10:44:28 2018 From: john.hearns at asml.com (John Hearns) Date: Thu, 25 Jan 2018 10:44:28 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Achim, Thankyou for your clear reply. Yes indeed we are using AFM at the moment. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Achim Rehor Sent: Thursday, January 25, 2018 11:21 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale John, yes, they definitely can! Nodes in a remote cluster are tob viewed just as local nodes in terms of taking part in the mechanisms of access to data. Token management will be done just as with local nodes. So if one node in cluster A recognizes a communication issue with a node in cluster B, it will let the clustermgr know, and that one then decides on whether to expel one or the other. Having a remote cluster connected relies on a stable and low latency network, just as a local cluster does. if your network is not reliable, you woudl go for AFM or other replication mechanisms (as the thread title implies;) ) Mit freundlichen Gr??en / Kind regards Achim Rehor ________________________________ Software Technical Support Specialist AIX/ Emea HPC Support [cid:image001.gif at 01D395D1.DA27C7E0] IBM Certified Advanced Technical Expert - Power Systems with AIX TSCC Software Service, Dept. 7922 Global Technology Services ________________________________ Phone: +49-7034-274-7862 IBM Deutschland E-Mail: Achim.Rehor at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany ________________________________ IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martin Hartmann (Vorsitzender), Norbert Janzen, Stefan Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 WEEE-Reg.-Nr. DE 99369940 From: John Hearns > To: gpfsug main discussion list > Date: 25/01/2018 10:53 Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Jan Frode, thankyou for that link. I have a general question regarding remote GPFS filesystems. If we have two clusters, in separate locations on separate Infiniband fabrics, we can set up a remote relationship between filesystems. As Valdis discusses, what happens if the IP link between the clusters goes down or is unstable? Can nodes in one cluster vote out nodes in the other cluster? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: Wednesday, January 24, 2018 8:08 AM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm? Seems to cover what you're looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales >: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 7182 bytes Desc: image001.gif URL: From olaf.weiser at de.ibm.com Thu Jan 25 13:17:52 2018 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 25 Jan 2018 14:17:52 +0100 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 14941 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: From hmorales at optimizeit.co Fri Jan 26 18:29:09 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Fri, 26 Jan 2018 13:29:09 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Hi Alex, This set up seems close to what I am trying to achieve. With regards to this kind of replication: any prereqs need to be met in the target environment for this to work? for example, should disk devices naming on AIX be the same as in the source environment? when importing the mmsdrfs file, how is Scale going to know which disks should it assign to the cluster? by their hdisk name alone? Thanks again, 2018-01-24 2:30 GMT-05:00 Alex Levin : > Hi, > > We are using a similar type of replication. > I assume the site B is the cold site prepared for DR > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > group to ensure consistency. > > The cluster name, IP addresses , hostnames of the cluster nodes are > different on another site - it can be a pre-configured cluster without > gpfs filesystems or with another filesystem. > Same names and addresses shouldn't be a problem. > > Additionally to the replicated LUNs/NSDs you need to deliver copy > of /var/mmfs/gen/mmsdrfs file from A to B site. > There is no need to replicate it in real-time, only after the change of > the cluster configuration. > > To activate site B - present replicated LUNs to the nodes in the DR > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > Tested with multiples LUNs and filesystems on various workloads - seems > to be working > > --Alex > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > wrote: > >> Thanks for answering. >> >> Essentially, the idea being explored is to replicate LUNs between >> identical storage hardware (HP 3PAR volumesrein) on both sites. There is an >> IP connection between storage boxes but not between servers on both sites, >> there is a dark fiber connecting both sites. Here they dont want to explore >> the idea of a scaled-based. >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gcorneau at us.ibm.com Fri Jan 26 20:21:15 2018 From: gcorneau at us.ibm.com (Glen Corneau) Date: Fri, 26 Jan 2018 14:21:15 -0600 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Scale will walk across all discovered disks upon start time and attempt to read the NSD identifiers from the disks. Once it finds them, it makes a local map file that correlates the NSD id and the hdiskX identifier. The names do not have to be the same as either the source cluster or even from node-to-node. The main thing to keep in mind is to keep the file system definitions in sync between the source and destination clusters. The "syncFSconfig" user exit is the best way to do it because it's automatic. You generally shouldn't be shuffling the mmsdrfs file between sites, that's what the "mmfsctl syncFSconfig" does for you, on a per-file system basis. GPFS+AIX customers have been using this kind of storage replication for over 10 years, it's business as usual. ------------------ Glen Corneau Power Systems Washington Systems Center gcorneau at us.ibm.com From: Harold Morales To: gpfsug main discussion list Date: 01/26/2018 12:30 PM Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Alex, This set up seems close to what I am trying to achieve. With regards to this kind of replication: any prereqs need to be met in the target environment for this to work? for example, should disk devices naming on AIX be the same as in the source environment? when importing the mmsdrfs file, how is Scale going to know which disks should it assign to the cluster? by their hdisk name alone? Thanks again, 2018-01-24 2:30 GMT-05:00 Alex Levin : Hi, We are using a similar type of replication. I assume the site B is the cold site prepared for DR The storage layer is EMC VMAX and the LUNs are replicated with SRDF. All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication group to ensure consistency. The cluster name, IP addresses , hostnames of the cluster nodes are different on another site - it can be a pre-configured cluster without gpfs filesystems or with another filesystem. Same names and addresses shouldn't be a problem. Additionally to the replicated LUNs/NSDs you need to deliver copy of /var/mmfs/gen/mmsdrfs file from A to B site. There is no need to replicate it in real-time, only after the change of the cluster configuration. To activate site B - present replicated LUNs to the nodes in the DR cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" Tested with multiples LUNs and filesystems on various workloads - seems to be working --Alex On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales wrote: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=d-vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m=VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh-GeVJiVS49opBwwCFs-FKQrQ7rc&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 26117 bytes Desc: not available URL: From hmorales at optimizeit.co Fri Jan 26 20:47:03 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Fri, 26 Jan 2018 15:47:03 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Thanks for participating in the discussion. Immediately after replication I am getting the error documented below, I have moved the mmsdrfs file (upon deleting the previous filesystem definitions, because I have managed to configure exactly equal my source and target clusters). Even then, I am getting the following error: GPFS: 6027-419 Failed to read a file system descriptor. There is an input or output error. mmlsfs: 6027-1639 Command failed. Examine previous That's the same error that upon first replicating without taking any other action. I think I am missing something really important but still dont know what it is. 2018-01-26 15:21 GMT-05:00 Glen Corneau : > Scale will walk across all discovered disks upon start time and attempt to > read the NSD identifiers from the disks. Once it finds them, it makes a > local map file that correlates the NSD id and the hdiskX identifier. The > names do not have to be the same as either the source cluster or even from > node-to-node. > > The main thing to keep in mind is to keep the file system definitions in > sync between the source and destination clusters. The "syncFSconfig" user > exit is the best way to do it because it's automatic. You generally > shouldn't be shuffling the mmsdrfs file between sites, that's what the > "mmfsctl syncFSconfig" does for you, on a per-file system basis. > > GPFS+AIX customers have been using this kind of storage replication for > over 10 years, it's business as usual. > > ------------------ > Glen Corneau > Power Systems > Washington Systems Center > gcorneau at us.ibm.com > > > > > > From: Harold Morales > To: gpfsug main discussion list > Date: 01/26/2018 12:30 PM > Subject: Re: [gpfsug-discuss] storage-based replication for > Spectrum Scale > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > With regards to this kind of replication: any prereqs need to be met in > the target environment for this to work? for example, should disk devices > naming on AIX be the same as in the source environment? when importing the > mmsdrfs file, how is Scale going to know which disks should it assign to > the cluster? by their hdisk name alone? > > Thanks again, > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin <*alevin at gmail.com* > >: > Hi, > > We are using a similar type of replication. > I assume the site B is the cold site prepared for DR > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > group to ensure consistency. > > The cluster name, IP addresses , hostnames of the cluster nodes are > different on another site - it can be a pre-configured cluster without > gpfs filesystems or with another filesystem. > Same names and addresses shouldn't be a problem. > > Additionally to the replicated LUNs/NSDs you need to deliver copy > of /var/mmfs/gen/mmsdrfs file from A to B site. > There is no need to replicate it in real-time, only after the change of > the cluster configuration. > > To activate site B - present replicated LUNs to the nodes in the DR > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > Tested with multiples LUNs and filesystems on various workloads - seems > to be working > > --Alex > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales <*hmorales at optimizeit.co* > > wrote: > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > IP connection between storage boxes but not between servers on both sites, > there is a dark fiber connecting both sites. Here they dont want to explore > the idea of a scaled-based. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at *spectrumscale.org* > > *http://gpfsug.org/mailman/listinfo/gpfsug-discuss* > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at *spectrumscale.org* > > *http://gpfsug.org/mailman/listinfo/gpfsug-discuss* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=d-vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m= > VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- > GeVJiVS49opBwwCFs-FKQrQ7rc&e= > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 26117 bytes Desc: not available URL: From sxiao at us.ibm.com Fri Jan 26 21:04:03 2018 From: sxiao at us.ibm.com (Steve Xiao) Date: Fri, 26 Jan 2018 16:04:03 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: When using this method of replication, you need to either issue "mmfsctl suspend or suspend-write" command before replication or setup a single consistency group for all LUNs. This is needed to ensure replica contain a consistent copy of GPFS data. Steve Y. Xiao gpfsug-discuss-bounces at spectrumscale.org wrote on 01/26/2018 03:21:23 PM: > From: gpfsug-discuss-request at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Date: 01/26/2018 03:21 PM > Subject: gpfsug-discuss Digest, Vol 72, Issue 69 > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: storage-based replication for Spectrum Scale (Harold Morales) > 2. Re: storage-based replication for Spectrum Scale (Glen Corneau) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 26 Jan 2018 13:29:09 -0500 > From: Harold Morales > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > Scale > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Hi Alex, This set up seems close to what I am trying to achieve. > > With regards to this kind of replication: any prereqs need to be met in the > target environment for this to work? for example, should disk devices > naming on AIX be the same as in the source environment? when importing the > mmsdrfs file, how is Scale going to know which disks should it assign to > the cluster? by their hdisk name alone? > > Thanks again, > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > > > Hi, > > > > We are using a similar type of replication. > > I assume the site B is the cold site prepared for DR > > > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > > group to ensure consistency. > > > > The cluster name, IP addresses , hostnames of the cluster nodes are > > different on another site - it can be a pre-configured cluster without > > gpfs filesystems or with another filesystem. > > Same names and addresses shouldn't be a problem. > > > > Additionally to the replicated LUNs/NSDs you need to deliver copy > > of /var/mmfs/gen/mmsdrfs file from A to B site. > > There is no need to replicate it in real-time, only after the change of > > the cluster configuration. > > > > To activate site B - present replicated LUNs to the nodes in the DR > > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > > > Tested with multiples LUNs and filesystems on various workloads - seems > > to be working > > > > --Alex > > > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > > wrote: > > > >> Thanks for answering. > >> > >> Essentially, the idea being explored is to replicate LUNs between > >> identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > >> IP connection between storage boxes but not between servers on both sites, > >> there is a dark fiber connecting both sites. Here they dont want to explore > >> the idea of a scaled-based. > >> > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > >> > >> > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180126_1503995b_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=SKdMmQae8uzHNWZq3vuRTp5UVwYFeeusLAxtbaposX0&e=> > > ------------------------------ > > Message: 2 > Date: Fri, 26 Jan 2018 14:21:15 -0600 > From: "Glen Corneau" > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > Scale > Message-ID: > 006FCEEE at notes.na.collabserv.com> > > Content-Type: text/plain; charset="us-ascii" > > Scale will walk across all discovered disks upon start time and attempt to > read the NSD identifiers from the disks. Once it finds them, it makes a > local map file that correlates the NSD id and the hdiskX identifier. The > names do not have to be the same as either the source cluster or even from > node-to-node. > > The main thing to keep in mind is to keep the file system definitions in > sync between the source and destination clusters. The "syncFSconfig" user > exit is the best way to do it because it's automatic. You generally > shouldn't be shuffling the mmsdrfs file between sites, that's what the > "mmfsctl syncFSconfig" does for you, on a per-file system basis. > > GPFS+AIX customers have been using this kind of storage replication for > over 10 years, it's business as usual. > > ------------------ > Glen Corneau > Power Systems > Washington Systems Center > gcorneau at us.ibm.com > > > > > > From: Harold Morales > To: gpfsug main discussion list > Date: 01/26/2018 12:30 PM > Subject: Re: [gpfsug-discuss] storage-based replication for > Spectrum Scale > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > With regards to this kind of replication: any prereqs need to be met in > the target environment for this to work? for example, should disk devices > naming on AIX be the same as in the source environment? when importing the > mmsdrfs file, how is Scale going to know which disks should it assign to > the cluster? by their hdisk name alone? > > Thanks again, > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > Hi, > > We are using a similar type of replication. > I assume the site B is the cold site prepared for DR > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > group to ensure consistency. > > The cluster name, IP addresses , hostnames of the cluster nodes are > different on another site - it can be a pre-configured cluster without > gpfs filesystems or with another filesystem. > Same names and addresses shouldn't be a problem. > > Additionally to the replicated LUNs/NSDs you need to deliver copy > of /var/mmfs/gen/mmsdrfs file from A to B site. > There is no need to replicate it in real-time, only after the change of > the cluster configuration. > > To activate site B - present replicated LUNs to the nodes in the DR > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > Tested with multiples LUNs and filesystems on various workloads - seems > to be working > > --Alex > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > wrote: > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is > an IP connection between storage boxes but not between servers on both > sites, there is a dark fiber connecting both sites. Here they dont want to > explore the idea of a scaled-based. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=d- > vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m=VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- > GeVJiVS49opBwwCFs-FKQrQ7rc&e= > > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180126_e291af63_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=bYnf-7v0CxYUkGth-QaVeUQdIlG8f1Gro-hwOxok7Qw&e=> > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: not available > Type: image/jpeg > Size: 26117 bytes > Desc: not available > URL: u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180126_e291af63_attachment.jpe&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=jYdnqhQBlnpf58oxunzBcTs9XdcbeOtLDQdgnASidDA&e=> > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > End of gpfsug-discuss Digest, Vol 72, Issue 69 > ********************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alevin at gmail.com Sat Jan 27 05:02:40 2018 From: alevin at gmail.com (Alex Levin) Date: Sat, 27 Jan 2018 00:02:40 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Steve, I've read that "mmfsctl suspend or suspend-write" should be executed, but in real life it is impossible for DR scenario. We tested both cases, the graceful one when the failover to another site is planned, applications stopped and i/o suspended and the case when there was no advanced notice about the disaster in the primary site. Both worked and for the second case various loads were simulated including heavy writes and reads/writes. In disaster model as expected some data were lost (due to incomplete writes, replication latency ... ), but mmfsck was always able to repair and the applications databases located on the affected filesystem were in an acceptable state. it is possible that we were just lucky and the test case didn't cover all possible scenarios. Harold, In our case, it is Linux, not AIX, but it shouldn't matter And our DR cluster is fully configured ( different IPs, hostnames and cluster name ) and running without filesystems at all or with a differently named filesystem. Before running mmimport , make sure that all expected LUNs are visible and writable. You can verify if the device is correct by reading first blocks of the device ( for example dd if=/dev/NSD_LUN_device bs=1M count=16 | strings ) not sure where you are " moved the mmsdrfs" you don't need to move/modify mmsdrfs file on the target ( disaster recovery ) cluster Just copy the one from primary to /tmp or /var/tmp and try to run mmimportfs fs_name -i copy_of_mmsdrfs /tmp/mmsdrfs Glen, as Harold has no IP connectivity between the clusters "mmfsctl syncFSconfig" is not an option ... Thanks --Alex On Fri, Jan 26, 2018 at 4:04 PM, Steve Xiao wrote: > When using this method of replication, you need to either issue "mmfsctl > suspend or suspend-write" command before replication or setup a single > consistency group for all LUNs. This is needed to ensure replica contain > a consistent copy of GPFS data. > > Steve Y. Xiao > > gpfsug-discuss-bounces at spectrumscale.org wrote on 01/26/2018 03:21:23 PM: > > > From: gpfsug-discuss-request at spectrumscale.org > > To: gpfsug-discuss at spectrumscale.org > > Date: 01/26/2018 03:21 PM > > Subject: gpfsug-discuss Digest, Vol 72, Issue 69 > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Send gpfsug-discuss mailing list submissions to > > gpfsug-discuss at spectrumscale.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > or, via email, send a message with subject or body 'help' to > > gpfsug-discuss-request at spectrumscale.org > > > > You can reach the person managing the list at > > gpfsug-discuss-owner at spectrumscale.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of gpfsug-discuss digest..." > > > > > > Today's Topics: > > > > 1. Re: storage-based replication for Spectrum Scale (Harold Morales) > > 2. Re: storage-based replication for Spectrum Scale (Glen Corneau) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Fri, 26 Jan 2018 13:29:09 -0500 > > From: Harold Morales > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > > Scale > > Message-ID: > > > > Content-Type: text/plain; charset="utf-8" > > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > > > With regards to this kind of replication: any prereqs need to be met in > the > > target environment for this to work? for example, should disk devices > > naming on AIX be the same as in the source environment? when importing > the > > mmsdrfs file, how is Scale going to know which disks should it assign to > > the cluster? by their hdisk name alone? > > > > Thanks again, > > > > > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > > > > > Hi, > > > > > > We are using a similar type of replication. > > > I assume the site B is the cold site prepared for DR > > > > > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX > replication > > > group to ensure consistency. > > > > > > The cluster name, IP addresses , hostnames of the cluster nodes are > > > different on another site - it can be a pre-configured cluster without > > > gpfs filesystems or with another filesystem. > > > Same names and addresses shouldn't be a problem. > > > > > > Additionally to the replicated LUNs/NSDs you need to deliver copy > > > of /var/mmfs/gen/mmsdrfs file from A to B site. > > > There is no need to replicate it in real-time, only after the change of > > > the cluster configuration. > > > > > > To activate site B - present replicated LUNs to the nodes in the DR > > > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > > > > > Tested with multiples LUNs and filesystems on various workloads - > seems > > > to be working > > > > > > --Alex > > > > > > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales < > hmorales at optimizeit.co> > > > wrote: > > > > > >> Thanks for answering. > > >> > > >> Essentially, the idea being explored is to replicate LUNs between > > >> identical storage hardware (HP 3PAR volumesrein) on both sites. There > is an > > >> IP connection between storage boxes but not between servers on both > sites, > > >> there is a dark fiber connecting both sites. Here they dont want to > explore > > >> the idea of a scaled-based. > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > >> > > >> > > > > > > _______________________________________________ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_ > attachments_20180126_1503995b_attachment-2D0001.html&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=SKdMmQae8uzHNWZq3vuRTp5UVwYFeeusLAxtbaposX0&e=> > > > > ------------------------------ > > > > Message: 2 > > Date: Fri, 26 Jan 2018 14:21:15 -0600 > > From: "Glen Corneau" > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > > Scale > > Message-ID: > > > 006FCEEE at notes.na.collabserv.com> > > > > Content-Type: text/plain; charset="us-ascii" > > > > Scale will walk across all discovered disks upon start time and attempt > to > > read the NSD identifiers from the disks. Once it finds them, it makes a > > local map file that correlates the NSD id and the hdiskX identifier. > The > > names do not have to be the same as either the source cluster or even > from > > node-to-node. > > > > The main thing to keep in mind is to keep the file system definitions in > > sync between the source and destination clusters. The "syncFSconfig" > user > > exit is the best way to do it because it's automatic. You generally > > shouldn't be shuffling the mmsdrfs file between sites, that's what the > > "mmfsctl syncFSconfig" does for you, on a per-file system basis. > > > > GPFS+AIX customers have been using this kind of storage replication for > > over 10 years, it's business as usual. > > > > ------------------ > > Glen Corneau > > Power Systems > > Washington Systems Center > > gcorneau at us.ibm.com > > > > > > > > > > > > From: Harold Morales > > To: gpfsug main discussion list > > Date: 01/26/2018 12:30 PM > > Subject: Re: [gpfsug-discuss] storage-based replication for > > Spectrum Scale > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > > > With regards to this kind of replication: any prereqs need to be met in > > the target environment for this to work? for example, should disk > devices > > naming on AIX be the same as in the source environment? when importing > the > > mmsdrfs file, how is Scale going to know which disks should it assign to > > the cluster? by their hdisk name alone? > > > > Thanks again, > > > > > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > > Hi, > > > > We are using a similar type of replication. > > I assume the site B is the cold site prepared for DR > > > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX > replication > > group to ensure consistency. > > > > The cluster name, IP addresses , hostnames of the cluster nodes are > > different on another site - it can be a pre-configured cluster without > > gpfs filesystems or with another filesystem. > > Same names and addresses shouldn't be a problem. > > > > Additionally to the replicated LUNs/NSDs you need to deliver copy > > of /var/mmfs/gen/mmsdrfs file from A to B site. > > There is no need to replicate it in real-time, only after the change of > > the cluster configuration. > > > > To activate site B - present replicated LUNs to the nodes in the DR > > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > > > Tested with multiples LUNs and filesystems on various workloads - seems > > to be working > > > > --Alex > > > > > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > > > wrote: > > Thanks for answering. > > > > Essentially, the idea being explored is to replicate LUNs between > > identical storage hardware (HP 3PAR volumesrein) on both sites. There is > > an IP connection between storage boxes but not between servers on both > > sites, there is a dark fiber connecting both sites. Here they dont want > to > > explore the idea of a scaled-based. > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=d- > > vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m= > VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- > > GeVJiVS49opBwwCFs-FKQrQ7rc&e= > > > > > > > > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_ > attachments_20180126_e291af63_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=bYnf-7v0CxYUkGth-QaVeUQdIlG8f1Gro-hwOxok7Qw&e=> > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: not available > > Type: image/jpeg > > Size: 26117 bytes > > Desc: not available > > URL: > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_ > attachments_20180126_e291af63_attachment.jpe&d=DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=jYdnqhQBlnpf58oxunzBcTs9XdcbeOtLDQdgnASidDA&e=> > > > > ------------------------------ > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > > End of gpfsug-discuss Digest, Vol 72, Issue 69 > > ********************************************** > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hmorales at optimizeit.co Sat Jan 27 09:11:44 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Sat, 27 Jan 2018 04:11:44 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Thanks for your insights. Alex, we did as you mentioned but after using mmimportfs there are a lot of errors on all commands having to do with filesystems: GPFS: 6027-419 Failed to read a file system descriptor. There is an input or output error that occurs for: mmlsfs mmlsdisk mmdf obviously, fs won't mount. 2018-01-27 0:02 GMT-05:00 Alex Levin : > Steve, > I've read that "mmfsctl suspend or suspend-write" should be executed, > but in real life it is impossible for DR scenario. > > We tested both cases, > the graceful one when the failover to another site is planned, > applications stopped and i/o suspended > and the case when there was no advanced notice about the disaster in the > primary site. > > Both worked and for the second case various loads were simulated including > heavy writes and reads/writes. > In disaster model as expected some data were lost (due to incomplete > writes, replication latency ... ), > but mmfsck was always able to repair and the applications databases > located on the affected filesystem were in an acceptable state. > > > it is possible that we were just lucky and the test case didn't cover all > possible scenarios. > > Harold, > In our case, it is Linux, not AIX, but it shouldn't matter > And our DR cluster is fully configured ( different IPs, hostnames and > cluster name ) and running without filesystems at all or with > a differently named filesystem. > > Before running mmimport , make sure that all expected LUNs are visible > and writable. > You can verify if the device is correct by reading first blocks of the > device ( for example dd if=/dev/NSD_LUN_device bs=1M count=16 | strings ) > > not sure where you are " moved the mmsdrfs" > you don't need to move/modify mmsdrfs file on the target ( disaster > recovery ) cluster > > Just copy the one from primary to /tmp or /var/tmp and try to run mmimportfs > fs_name -i copy_of_mmsdrfs /tmp/mmsdrfs > > > > Glen, as Harold has no IP connectivity between the clusters "mmfsctl > syncFSconfig" is not an option ... > > Thanks > --Alex > > > > > On Fri, Jan 26, 2018 at 4:04 PM, Steve Xiao wrote: > >> When using this method of replication, you need to either issue "mmfsctl >> suspend or suspend-write" command before replication or setup a single >> consistency group for all LUNs. This is needed to ensure replica contain >> a consistent copy of GPFS data. >> >> Steve Y. Xiao >> >> gpfsug-discuss-bounces at spectrumscale.org wrote on 01/26/2018 03:21:23 PM: >> >> > From: gpfsug-discuss-request at spectrumscale.org >> > To: gpfsug-discuss at spectrumscale.org >> > Date: 01/26/2018 03:21 PM >> > Subject: gpfsug-discuss Digest, Vol 72, Issue 69 >> > Sent by: gpfsug-discuss-bounces at spectrumscale.org >> > >> > Send gpfsug-discuss mailing list submissions to >> > gpfsug-discuss at spectrumscale.org >> > >> > To subscribe or unsubscribe via the World Wide Web, visit >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > or, via email, send a message with subject or body 'help' to >> > gpfsug-discuss-request at spectrumscale.org >> > >> > You can reach the person managing the list at >> > gpfsug-discuss-owner at spectrumscale.org >> > >> > When replying, please edit your Subject line so it is more specific >> > than "Re: Contents of gpfsug-discuss digest..." >> > >> > >> > Today's Topics: >> > >> > 1. Re: storage-based replication for Spectrum Scale (Harold Morales) >> > 2. Re: storage-based replication for Spectrum Scale (Glen Corneau) >> > >> > >> > ---------------------------------------------------------------------- >> > >> > Message: 1 >> > Date: Fri, 26 Jan 2018 13:29:09 -0500 >> > From: Harold Morales >> > To: gpfsug main discussion list >> > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum >> > Scale >> > Message-ID: >> > >> > Content-Type: text/plain; charset="utf-8" >> >> > >> > Hi Alex, This set up seems close to what I am trying to achieve. >> > >> > With regards to this kind of replication: any prereqs need to be met in >> the >> > target environment for this to work? for example, should disk devices >> > naming on AIX be the same as in the source environment? when importing >> the >> > mmsdrfs file, how is Scale going to know which disks should it assign to >> > the cluster? by their hdisk name alone? >> > >> > Thanks again, >> > >> > >> > >> > 2018-01-24 2:30 GMT-05:00 Alex Levin : >> > >> > > Hi, >> > > >> > > We are using a similar type of replication. >> > > I assume the site B is the cold site prepared for DR >> > > >> > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. >> > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX >> replication >> > > group to ensure consistency. >> > > >> > > The cluster name, IP addresses , hostnames of the cluster nodes are >> > > different on another site - it can be a pre-configured cluster without >> > > gpfs filesystems or with another filesystem. >> > > Same names and addresses shouldn't be a problem. >> > > >> > > Additionally to the replicated LUNs/NSDs you need to deliver copy >> > > of /var/mmfs/gen/mmsdrfs file from A to B site. >> > > There is no need to replicate it in real-time, only after the change >> of >> > > the cluster configuration. >> > > >> > > To activate site B - present replicated LUNs to the nodes in the DR >> > > cluster and run mmimportfs as "mmimportfs fs_name -i >> copy_of_mmsdrfs" >> > > >> > > Tested with multiples LUNs and filesystems on various workloads - >> seems >> > > to be working >> > > >> > > --Alex >> > > >> > > >> > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales < >> hmorales at optimizeit.co> >> > > wrote: >> > > >> > >> Thanks for answering. >> > >> >> > >> Essentially, the idea being explored is to replicate LUNs between >> > >> identical storage hardware (HP 3PAR volumesrein) on both sites. >> There is an >> > >> IP connection between storage boxes but not between servers on both >> sites, >> > >> there is a dark fiber connecting both sites. Here they dont want to >> explore >> > >> the idea of a scaled-based. >> > >> >> > >> >> > >> _______________________________________________ >> > >> gpfsug-discuss mailing list >> > >> gpfsug-discuss at spectrumscale.org >> > >> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> >> > >> >> > > >> > > _______________________________________________ >> > > gpfsug-discuss mailing list >> > > gpfsug-discuss at spectrumscale.org >> > > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > > >> > > >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: > > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_1503995b_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=SKdMmQae8uzHNWZq3vuRTp5UVwYFeeusLAxtbaposX0&e=> >> > >> > ------------------------------ >> > >> > Message: 2 >> > Date: Fri, 26 Jan 2018 14:21:15 -0600 >> > From: "Glen Corneau" >> > To: gpfsug main discussion list >> > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum >> > Scale >> > Message-ID: >> > > > 006FCEEE at notes.na.collabserv.com> >> > >> > Content-Type: text/plain; charset="us-ascii" >> > >> > Scale will walk across all discovered disks upon start time and attempt >> to >> > read the NSD identifiers from the disks. Once it finds them, it makes >> a >> > local map file that correlates the NSD id and the hdiskX identifier. >> The >> > names do not have to be the same as either the source cluster or even >> from >> > node-to-node. >> > >> > The main thing to keep in mind is to keep the file system definitions >> in >> > sync between the source and destination clusters. The "syncFSconfig" >> user >> > exit is the best way to do it because it's automatic. You generally >> > shouldn't be shuffling the mmsdrfs file between sites, that's what the >> > "mmfsctl syncFSconfig" does for you, on a per-file system basis. >> > >> > GPFS+AIX customers have been using this kind of storage replication for >> > over 10 years, it's business as usual. >> > >> > ------------------ >> > Glen Corneau >> > Power Systems >> > Washington Systems Center >> > gcorneau at us.ibm.com >> > >> > >> > >> > >> > >> > From: Harold Morales >> > To: gpfsug main discussion list >> > Date: 01/26/2018 12:30 PM >> > Subject: Re: [gpfsug-discuss] storage-based replication for >> > Spectrum Scale >> > Sent by: gpfsug-discuss-bounces at spectrumscale.org >> > >> > >> > >> > Hi Alex, This set up seems close to what I am trying to achieve. >> > >> > With regards to this kind of replication: any prereqs need to be met in >> > the target environment for this to work? for example, should disk >> devices >> > naming on AIX be the same as in the source environment? when importing >> the >> > mmsdrfs file, how is Scale going to know which disks should it assign >> to >> > the cluster? by their hdisk name alone? >> > >> > Thanks again, >> > >> > >> > >> > 2018-01-24 2:30 GMT-05:00 Alex Levin : >> > Hi, >> > >> > We are using a similar type of replication. >> > I assume the site B is the cold site prepared for DR >> > >> > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. >> > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX >> replication >> > group to ensure consistency. >> > >> > The cluster name, IP addresses , hostnames of the cluster nodes are >> > different on another site - it can be a pre-configured cluster without >> > gpfs filesystems or with another filesystem. >> > Same names and addresses shouldn't be a problem. >> > >> > Additionally to the replicated LUNs/NSDs you need to deliver copy >> > of /var/mmfs/gen/mmsdrfs file from A to B site. >> > There is no need to replicate it in real-time, only after the change of >> > the cluster configuration. >> > >> > To activate site B - present replicated LUNs to the nodes in the DR >> > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" >> > >> > Tested with multiples LUNs and filesystems on various workloads - >> seems >> > to be working >> > >> > --Alex >> > >> > >> > >> > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales >> >> > wrote: >> > Thanks for answering. >> > >> > Essentially, the idea being explored is to replicate LUNs between >> > identical storage hardware (HP 3PAR volumesrein) on both sites. There >> is >> > an IP connection between storage boxes but not between servers on both >> > sites, there is a dark fiber connecting both sites. Here they dont want >> to >> > explore the idea of a scaled-based. >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=d- >> > vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m=VbfWaftYSVjx8fMb >> 2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- >> > GeVJiVS49opBwwCFs-FKQrQ7rc&e= >> > >> > >> > >> > >> > >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: > > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_e291af63_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=bYnf-7v0CxYUkGth-QaVeUQdIlG8f1Gro-hwOxok7Qw&e=> >> > -------------- next part -------------- >> > A non-text attachment was scrubbed... >> > Name: not available >> > Type: image/jpeg >> > Size: 26117 bytes >> > Desc: not available >> > URL: > > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_e291af63_attachment.jpe&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=jYdnqhQBlnpf58oxunzBcTs9XdcbeOtLDQdgnASidDA&e=> >> > >> > ------------------------------ >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > >> > End of gpfsug-discuss Digest, Vol 72, Issue 69 >> > ********************************************** >> > >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egonle at aim.com Sat Jan 27 15:18:24 2018 From: egonle at aim.com (Eg. Bo.) Date: Sat, 27 Jan 2018 10:18:24 -0500 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Message-ID: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> Hello, while operating a GPFS filesystem for HPC some workload of few clients is able to eat available storage and NSD performance. Compute Nodes and NSD servers are connected to Infiniband.This question somehow is weird but is there a way to limit performance / I/O / bandwidth consumption for nodes so that other node still get "good" performance? Thanks, Eg. Bo.egonle at aim.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Sun Jan 28 15:53:47 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sun, 28 Jan 2018 12:53:47 -0300 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth In-Reply-To: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> References: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> Message-ID: Depends... While the current version of mmchqos is primarily geared towards controlling "maintenance" tasks. It can be used to limit (cap) the IOP rate for IOs coming from a particular GFPFS client node. You would need a version that supports the -N option on the mmchqos command and then configure each node that will mount the FS. Caution: try some experiments on a test system -- incorrect configuration can severely impact performance to the point where it can appear that things are totally "stuck"! mmchqos FS --disable --reset should get you back to the default config for gpfs/QOS. From: "Eg. Bo." To: gpfsug-discuss at spectrumscale.org Date: 01/27/2018 12:23 PM Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, while operating a GPFS filesystem for HPC some workload of few clients is able to eat available storage and NSD performance. Compute Nodes and NSD servers are connected to Infiniband. This question somehow is weird but is there a way to limit performance / I/O / bandwidth consumption for nodes so that other node still get "good" performance? Thanks, Eg. Bo. egonle at aim.com _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=r0AtLTBmDo_uZhIdntBO8k2-1Alg_2LBLPoJamV-zsY&s=BXGAuET7K4UhA24P72F4YRsAdcdJR3H02gqGOERENlE&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From vpuvvada at in.ibm.com Mon Jan 29 07:04:56 2018 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Mon, 29 Jan 2018 12:34:56 +0530 Subject: [gpfsug-discuss] AFM and RHEL 7.4 In-Reply-To: References: Message-ID: This is testing effort, AFM with RHEL 7.4 will be officially supported in next PTF releases 5.0.0.1 and 4.2.3.7. ~Venkat (vpuvvada at in.ibm.com) From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 01/25/2018 01:57 PM Subject: [gpfsug-discuss] AFM and RHEL 7.4 Sent by: gpfsug-discuss-bounces at spectrumscale.org The FAQ has a note stating: 1. AFM, Asynch Disaster Recovery with AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4. Could someone please clarify this sentence ? It can't be right that none of these features are supported with RHEL 7.4, or .. ? -jf_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=WsrKUTbd_huFePIa8dbAt4aezWpKHV902b-BlZTwGiA&s=OMRkxSzqA8U8J8mo3ap8I5TKscWunNEA2KZV3Z5_6HA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From richardb+gpfsUG at ellexus.com Mon Jan 29 10:27:37 2018 From: richardb+gpfsUG at ellexus.com (Richard Booth) Date: Mon, 29 Jan 2018 10:27:37 +0000 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Message-ID: Hi Without wishing to blow our own trumpet, Ellexus tools do this with GPFS, There is a youtube video that shows our tool Mistral managing a fair usage policy, link below, https://youtu.be/LJFcBUkcn3c Richard ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 27 Jan 2018 10:18:24 -0500 > From: "Eg. Bo." > To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth > Message-ID: <1613832bc3c-1721-1396f9 at webjas-vaa235.srv.aolmail.net> > Content-Type: text/plain; charset="utf-8" > > > Hello, > > > while operating a GPFS filesystem for HPC some workload of few clients is > able to eat available storage and NSD performance. Compute Nodes and NSD > servers are connected to Infiniband.This question somehow is weird but is > there a way to limit performance / I/O / bandwidth consumption for nodes so > that other node still get "good" performance? > > > > > Thanks, > > > Eg. Bo.egonle at aim.com > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: 20180127/dcc7cd79/attachment-0001.html> > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 72, Issue 74 > ********************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Mon Jan 29 09:54:46 2018 From: john.hearns at asml.com (John Hearns) Date: Mon, 29 Jan 2018 09:54:46 +0000 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth In-Reply-To: References: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> Message-ID: Hello Egonle. I looked at this scenario recently (H Mark!) Maybe you could send me an email off-list? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Marc A Kaplan Sent: Sunday, January 28, 2018 4:54 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] limit / fair share of GPFS bandwidth Depends... While the current version of mmchqos is primarily geared towards controlling "maintenance" tasks. It can be used to limit (cap) the IOP rate for IOs coming from a particular GFPFS client node. You would need a version that supports the -N option on the mmchqos command and then configure each node that will mount the FS. Caution: try some experiments on a test system -- incorrect configuration can severely impact performance to the point where it can appear that things are totally "stuck"! mmchqos FS --disable --reset should get you back to the default config for gpfs/QOS. From: "Eg. Bo." > To: gpfsug-discuss at spectrumscale.org Date: 01/27/2018 12:23 PM Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hello, while operating a GPFS filesystem for HPC some workload of few clients is able to eat available storage and NSD performance. Compute Nodes and NSD servers are connected to Infiniband. This question somehow is weird but is there a way to limit performance / I/O / bandwidth consumption for nodes so that other node still get "good" performance? Thanks, Eg. Bo. egonle at aim.com_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=r0AtLTBmDo_uZhIdntBO8k2-1Alg_2LBLPoJamV-zsY&s=BXGAuET7K4UhA24P72F4YRsAdcdJR3H02gqGOERENlE&e= -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkr at lbl.gov Wed Jan 31 21:20:04 2018 From: kkr at lbl.gov (Kristy Kallback-Rose) Date: Wed, 31 Jan 2018 13:20:04 -0800 Subject: [gpfsug-discuss] US Spectrum Scale/GPFS User Group - Save the Date ~May 16 - ~May 17 - Boston Message-ID: <46BCDF1D-72C8-4FD1-8375-DD4207732F78@lbl.gov> Please mark your calendars, we?re planning for our next US Spectrum Scale/GPFS User Group meeting. We should be able to provide more exact details in a week or so, but loosely the event will be two days in Boston around May 16 - May 17, 2018. Send agenda suggestions now. Feel free to reply to the list to generate discussion. Best, Kristy From chris.schlipalius at pawsey.org.au Wed Jan 31 23:21:02 2018 From: chris.schlipalius at pawsey.org.au (Chris Schlipalius) Date: Thu, 01 Feb 2018 07:21:02 +0800 Subject: [gpfsug-discuss] 2018 March 26th Singapore Spectrum Scale User Group event announced - Final call for user stories - submission deadline 9th Feb Message-ID: Hello, First, let me apologise to all who have responded. This is a follow up call out for user use cases or presentations for those wishing to present at this Usergroup ? namely client use cases or client stories, from those who haven?t yet contacted me to organise to do so. At the inaugural Spectrum Scale Usergroup Singapore on the Monday 26th March 2018, Sentosa, Singapore. This is being held in conjunction with SCA18 https://sc-asia.org/ All current Spectrum Scale User Group event details can be found here: http://goo.gl/dXtqvS Feel free to circulate this event link to all that may need it. Please ensure a speaker?s ticket is reserved, and please email me the title and duration of the talk and speakers name details so I can add this to the agenda on Eventbrite and promote on the discussion list and spectrumscale.org website. Accommodation and conference registration - both are open ? please see https://www.sc-asia.org/accommodation/ for the accommodation booking form, and https://www.sc-asia.org/register/ for the early bird conference registration (early bird rate closes soon 9th Feb). Please reserve a ticket in the Eventbrite link above ASAP and if you wish, email me separately if you have accommodation sorted, or need Sentosa accommodation (in addition to completing the form linked above). As I am in contact with the organisers of the event and I can also ensure there are places for presenters at this group and attendees. The user group location is still located at Resorts World Convention We are looking forwards to a great new Usergroup soon in a fabulous venue. Thanks again to NSCC and IBM for helping to arrange the venue and event booking. Regards, Chris Schlipalius Team Lead, Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO) 12 Burvill Court Kensington WA 6151 Australia From mail at arif-ali.co.uk Tue Jan 2 10:29:11 2018 From: mail at arif-ali.co.uk (Arif Ali) Date: Tue, 2 Jan 2018 10:29:11 +0000 Subject: [gpfsug-discuss] Testing to see the mailing list is working -- please ignore Message-ID: <42f7a66e-ff42-ef26-8ece-888fdc932f8e@arif-ali.co.uk> Just a quick test message, to ensure mailing list is working, as we've not heard since 22nd December, So please ignore -- regards, Arif Ali -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From r.sobey at imperial.ac.uk Wed Jan 3 10:37:54 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 3 Jan 2018 10:37:54 +0000 Subject: [gpfsug-discuss] mmchfileset syntax Message-ID: Hi all, Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. mmchfileset gpfs filesetname -J /gpfs/new_path mmchfileset: Options -J and FilesetName cannot be specified at the same time. Which leads to what?s the point of the following command as the error suggests it will work: mmchfileset device -J /gpfs/new_path ?as not specifying a FilesetName to the input of mmchfileset seems odd. I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jan 3 11:49:42 2018 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 3 Jan 2018 11:49:42 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: Message-ID: -J is to define which fileset you are talking about. The alternative is give the fileset name straight after the ?device? argument. Once the fileset is unambiguous, you then use other command line option to change things e.g. -j to change the *name* of this fileset. -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. Daniel Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com > On 3 Jan 2018, at 10:38, Sobey, Richard A wrote: > > Hi all, > > Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. > > mmchfileset gpfs filesetname -J /gpfs/new_path > mmchfileset: Options -J and FilesetName cannot be specified at the same time. > > Which leads to what?s the point of the following command as the error suggests it will work: > > mmchfileset device -J /gpfs/new_path > > ?as not specifying a FilesetName to the input of mmchfileset seems odd. > > I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? > > Cheers > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Wed Jan 3 12:37:46 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 3 Jan 2018 12:37:46 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: References: Message-ID: Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and mmchfileset gets the filesetname from that). Ultimately though I cannot change the mount point of the fileset without mmunlink / mmlink, right? Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger Sent: 03 January 2018 11:50 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmchfileset syntax -J is to define which fileset you are talking about. The alternative is give the fileset name straight after the ?device? argument. Once the fileset is unambiguous, you then use other command line option to change things e.g. -j to change the *name* of this fileset. -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. Daniel [/spectrum_storage-banne] [Spectrum Scale Logo] Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com On 3 Jan 2018, at 10:38, Sobey, Richard A > wrote: Hi all, Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. mmchfileset gpfs filesetname -J /gpfs/new_path mmchfileset: Options -J and FilesetName cannot be specified at the same time. Which leads to what?s the point of the following command as the error suggests it will work: mmchfileset device -J /gpfs/new_path ?as not specifying a FilesetName to the input of mmchfileset seems odd. I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? Cheers Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From tconnelly at pixitmedia.com Wed Jan 3 13:27:13 2018 From: tconnelly at pixitmedia.com (Tim Connelly) Date: Wed, 3 Jan 2018 13:27:13 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: References: Message-ID: >From the mmlinkfileset docs; "The user may use the mv command on the directory to move to a new location in the parent fileset, but the mv command is not allowed to move the junction to a different fileset." # mmlsfileset mmfs1 |grep timt sas1-timtest Linked /mmfs1/another_test/timtest # mv /mmfs1/another_test/timtest /mmfs1/another_test/timtest2 # mmlsfileset mmfs1 |grep timt sas1-timtest Linked /mmfs1/another_test/timtest2 Hope this helps! Tim On 3 January 2018 at 12:37, Sobey, Richard A wrote: > Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and > mmchfileset gets the filesetname from that). > > > > Ultimately though I cannot change the mount point of the fileset without > mmunlink / mmlink, right? > > > > Thanks > > Richard > > > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: > gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Daniel Kidger > *Sent:* 03 January 2018 11:50 > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] mmchfileset syntax > > > > -J is to define which fileset you are talking about. > > The alternative is give the fileset name straight after the ?device? > argument. > > > > Once the fileset is unambiguous, you then use other command line option to > change things > > e.g. -j to change the *name* of this fileset. > > -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the > mount point either. > > > > Daniel > > [image: /spectrum_storage-banne] > > > > > [image: Spectrum Scale Logo] > > > > > > *Dr Daniel Kidger* > IBM Technical Sales Specialist > Software Defined Solution Sales > > + <+%2044-7818%20522%20266> 44-(0)7818 522 266 <+%2044-7818%20522%20266> > daniel.kidger at uk.ibm.com > > > On 3 Jan 2018, at 10:38, Sobey, Richard A wrote: > > Hi all, > > > > Attempting to change the junction path of a fileset and I?m getting some > strange output. It may just not be possible but the documentation on > mmchfileset itself doesn?t make it clear. > > > > mmchfileset gpfs filesetname -J /gpfs/new_path > > mmchfileset: Options -J and FilesetName cannot be specified at the same > time. > > > > Which leads to what?s the point of the following command as the error > suggests it will work: > > > > mmchfileset device -J /gpfs/new_path > > > > ?as not specifying a FilesetName to the input of mmchfileset seems odd. > > > > I do of course know that I can just unlink and relink but hey.. trying to > save a few precious seconds ? > > > > Cheers > > Richard > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jan 3 13:13:03 2018 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 3 Jan 2018 13:13:03 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: Message-ID: Richard, The documentation says that that you can simply use ?mv? on a live fileset too. I have just tried this and yes it appears to work. I can move a fileset into another subdirectory of the same filesystem and mmlsfileset confirms the new mount point. Daniel Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com > On 3 Jan 2018, at 12:37, Sobey, Richard A wrote: > > Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and mmchfileset gets the filesetname from that). > > Ultimately though I cannot change the mount point of the fileset without mmunlink / mmlink, right? > > Thanks > Richard > > > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger > Sent: 03 January 2018 11:50 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] mmchfileset syntax > > -J is to define which fileset you are talking about. > The alternative is give the fileset name straight after the ?device? argument. > > Once the fileset is unambiguous, you then use other command line option to change things > e.g. -j to change the *name* of this fileset. > -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. > > > Daniel > > > > > > > Dr Daniel Kidger > IBM Technical Sales Specialist > Software Defined Solution Sales > > + 44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > On 3 Jan 2018, at 10:38, Sobey, Richard A wrote: > > Hi all, > > Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. > > mmchfileset gpfs filesetname -J /gpfs/new_path > mmchfileset: Options -J and FilesetName cannot be specified at the same time. > > Which leads to what?s the point of the following command as the error suggests it will work: > > mmchfileset device -J /gpfs/new_path > > ?as not specifying a FilesetName to the input of mmchfileset seems odd. > > I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? > > Cheers > Richard > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Wed Jan 3 13:58:20 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 3 Jan 2018 13:58:20 +0000 Subject: [gpfsug-discuss] mmchfileset syntax In-Reply-To: References: Message-ID: Thank you Daniel and Tim, most helpful. Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger Sent: 03 January 2018 13:13 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmchfileset syntax Richard, The documentation says that that you can simply use ?mv? on a live fileset too. I have just tried this and yes it appears to work. I can move a fileset into another subdirectory of the same filesystem and mmlsfileset confirms the new mount point. Daniel [Image removed by sender. /spectrum_storage-banne] [Image removed by sender. Spectrum Scale Logo] Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com On 3 Jan 2018, at 12:37, Sobey, Richard A > wrote: Ah. So I can specify ?device filesetname? OR ?device -J junctionpath? (and mmchfileset gets the filesetname from that). Ultimately though I cannot change the mount point of the fileset without mmunlink / mmlink, right? Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Daniel Kidger Sent: 03 January 2018 11:50 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] mmchfileset syntax -J is to define which fileset you are talking about. The alternative is give the fileset name straight after the ?device? argument. Once the fileset is unambiguous, you then use other command line option to change things e.g. -j to change the *name* of this fileset. -j doesn?t change the mount point, and indeed mmcrfileset doesn?t set the mount point either. Daniel [Image removed by sender. /spectrum_storage-banne] [Image removed by sender. Spectrum Scale Logo] Dr Daniel Kidger IBM Technical Sales Specialist Software Defined Solution Sales + 44-(0)7818 522 266 daniel.kidger at uk.ibm.com On 3 Jan 2018, at 10:38, Sobey, Richard A > wrote: Hi all, Attempting to change the junction path of a fileset and I?m getting some strange output. It may just not be possible but the documentation on mmchfileset itself doesn?t make it clear. mmchfileset gpfs filesetname -J /gpfs/new_path mmchfileset: Options -J and FilesetName cannot be specified at the same time. Which leads to what?s the point of the following command as the error suggests it will work: mmchfileset device -J /gpfs/new_path ?as not specifying a FilesetName to the input of mmchfileset seems odd. I do of course know that I can just unlink and relink but hey.. trying to save a few precious seconds ? Cheers Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.orghttps://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=75vJhcWv1Hpd6tMuB3qZ2waqZZtyZy9OKQ0LEECaPUM&s=Qnv7oXB0V-pe1q5evqLFP6uZPxplkHonKfRwnT1Inpc&e= Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ~WRD029.jpg Type: image/jpeg Size: 823 bytes Desc: ~WRD029.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2519 bytes Desc: image001.jpg URL: From hoov at us.ibm.com Wed Jan 3 17:10:51 2018 From: hoov at us.ibm.com (Theodore Hoover Jr) Date: Wed, 3 Jan 2018 17:10:51 +0000 Subject: [gpfsug-discuss] Fw: Spectrum Scale on AWS - Join Sponsor User Program Message-ID: An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Thu Jan 4 11:16:57 2018 From: p.childs at qmul.ac.uk (Peter Childs) Date: Thu, 4 Jan 2018 11:16:57 +0000 Subject: [gpfsug-discuss] Use AFM for migration of many small files In-Reply-To: References: <467FB293-D33B-46F4-BA1B-A5CB4D28DDE6@psi.ch> Message-ID: <1515064616.28898.32.camel@qmul.ac.uk> We are doing something very similar using 4.2.3-4 and GPFS 4.2.1-1 on the nfs backend. Did you have any success? The plan is to load the new cache from the old GPFS then dump once the cache is full. We've already increase numThreashThreads from 4 to 8 and seen only marginal improvements, we could attempt to increase this further. I'm also wondering if its worth increasing the Refresh Intervals to speed up read of already cache files. At this stage we want to fill the cache and don't care about write back until we switch the target to the new NFS/GPFS from our old GPFS storage to a new box back at our off-site location, (otherwise known as the office) [root at afmgateway1 scratch]# mmlsfileset home home -L --afm Filesets in file system 'home': Attributes for fileset home: ============================= Status Linked Path /data2/home Id 42 Root inode 1343225859 Parent Id 0 Created Wed Jan 3 12:32:33 2018 Comment Inode space 41 Maximum number of inodes 100000000 Allocated inodes 15468544 Permission change flag chmodAndSetacl afm-associated Yes Target nfs://afm1/gpfs/data1/afm/home Mode single-writer File Lookup Refresh Interval 30 (default) File Open Refresh Interval 30 (default) Dir Lookup Refresh Interval 60 (default) Dir Open Refresh Interval 60 (default) Async Delay 15 (default) Last pSnapId 0 Display Home Snapshots no Number of Gateway Flush Threads 8 Prefetch Threshold 0 (default) Eviction Enabled no Thanks in advance. Peter Childs On Tue, 2017-09-05 at 19:57 +0530, Venkateswara R Puvvada wrote: Which version of Spectrum Scale ? What is the fileset mode ? >We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here >I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. How was the performance measured ? If parallel IO is enabled, AFM uses multiple gateway nodes to prefetch the large files (if file size if more than 1GB). Performance difference between small and lager file is huge (1000MB - 150MB = 850MB) here, and generally it is not the case. How many files were present in list file for prefetch ? Could you also share full internaldump from the gateway node ? >I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few >read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. AFM prefetches the files on multiple threads. Default flush threads for prefetch are 36 (fileset.afmNumFlushThreads (default 4) + afmNumIOFlushThreads (default 32)). >Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and >I need to look elsewhere to get better performance for prefetch of many smaller files? See above, AFM reads files on multiple threads parallelly. Try increasing the afmNumFlushThreads on fileset and verify if it improves the performance. ~Venkat (vpuvvada at in.ibm.com) From: "Billich Heinrich Rainer (PSI)" To: "gpfsug-discuss at spectrumscale.org" Date: 09/04/2017 10:18 PM Subject: [gpfsug-discuss] Use AFM for migration of many small files Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and I need to look elsewhere to get better performance for prefetch of many smaller files? We will migrate several filesets in parallel, but still with individual filesets up to 350TB in size 150MB/s isn?t fun. Also just about 150 files/s seconds looks poor. The setup is quite new, hence there may be other places to look at. It?s all RHEL7 an spectrum scale 4.2.2-3 on the afm cache. Thank you, Heiner --, Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://urldefense.proofpoint.com/v2/url?u=https-3A__www.psi.ch&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=eHcVdovN10-m-Qk0Ln2qvol3pkKNFwrzz2wgf1zXVXE&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=LbRyuSM_djs0FDXr27hPottQHAn3OGcivpyRcIDBN3U&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Peter Childs ITS Research Storage Queen Mary, University of London -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Thu Jan 4 17:57:14 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Thu, 4 Jan 2018 17:57:14 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Message-ID: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bipcuds at gmail.com Thu Jan 4 20:34:30 2018 From: bipcuds at gmail.com (Keith Ball) Date: Thu, 4 Jan 2018 15:34:30 -0500 Subject: [gpfsug-discuss] Conflicting RHEL compatibility information in the Spectrum Scale FAQ Message-ID: Hi All, Here is a repeat post of my question on GSS 2.0.7 compatibility with RHEL 6.7. FAQ is contradictory. I know GSS/ESS is usually quite sensitive to use a specific RHEL minor version). And at this point, RHEL 6.5 isn't really supported anymore (no updates provided). On Tue, Dec 19, 2017 at 6:08 PM, Keith Ball wrote: I was recently trying to determine the latest RHEL release that will work > with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code > uses Scale 4.1.0.8. > > A specific X.Y GSS code build, from my experience, is intended to use a > specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), > which no longer has EUS support from RedHat (only 6.7 still does). > > GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be > used, which is a surprising statement given GSS/ESS code's sensitivity to > OS levels (any ESS I have ever seen has never been supported on more than > one version of RHEL). > > According to the Scale FAQ (https://www.ibm.com/support/ > knowledgecenter/STXKQY/gpfsclustersfaq.html#linux), A 2.2, Table 27, > Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and > 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 > is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 > release" ... which contradicts Table 27! > > Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install > GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is > supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option > here). > > Many Thanks, > Keith > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skylar2 at u.washington.edu Fri Jan 5 00:52:31 2018 From: skylar2 at u.washington.edu (Skylar Thompson) Date: Thu, 4 Jan 2018 16:52:31 -0800 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: <85c8b317-a7cb-ce84-81bd-0a7e8b75bc7e@u.washington.edu> While I'm not fully versed on the vulnerability or the proposed fixes, my understanding is that most of the performance impact from the fix is around having kernel memory completely separately from a process's user-space, which means every system call will have cache/TLB misses. This might mean clusters which are using RDMA won't have as big of a performance hit versus ones that aren't able to use RDMA, since they can do a lot more in user-space without involving the kernel. On 01/04/2018 09:57 AM, Buterbaugh, Kevin L wrote: > Happy New Year everyone, > > I?m sure that everyone is aware of Meltdown and Spectre by now ? we, > like many other institutions, will be patching for it at the earliest > possible opportunity. > > Our understanding is that the most serious of the negative performance > impacts of these patches will be for things like I/O (disk / network) > ? given that, we are curious if IBM has any plans for a GPFS update > that could help mitigate those impacts? ?Or is there simply nothing > that can be done? > > If there is a GPFS update planned for this we?d be interested in > knowing so that we could coordinate the kernel and GPFS upgrades on > our cluster. > > Thanks? > > Kevin > > P.S. ?The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a > good year for everyone despite how it?s starting out. ?:-O > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu > ?- (615)875-9633 > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Thu Jan 4 22:55:18 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Thu, 4 Jan 2018 17:55:18 -0500 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=m7Pdt9KL82CJT_AT-PwkmO3PbHg88-IQ7Jq-dwhDOdY&s=5i66Rx3vse5LcaN4-WlyCwi_TDTOQGQR2-X_XyjbBpw&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From richardb+gpfsUG at ellexus.com Fri Jan 5 08:09:38 2018 From: richardb+gpfsUG at ellexus.com (Richard Booth) Date: Fri, 5 Jan 2018 08:09:38 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Message-ID: Dear Kevin The company I work for might be able to help get back the performance lost from the fixes for Meltdown and Spectre. I work for Ellexus, an I/O profiling company. Our tools could help identify any inefficient I/O patterns in your applications to reduce the impact of this. We can set up demos and a free trial if you're interested in seeing what we do and how we can help. Kind regards Richard Message: 1 Date: Thu, 4 Jan 2018 17:57:14 +0000 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Message-ID: <5D655862-7F60-47F6-8BD2-A5298F73F70F at vanderbilt.edu> Content-Type: text/plain; charset="utf-8" Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -- Richard Booth Account Manager, Ellexus Ltd richard.booth at ellexus.com +44 (0)1223 42 16 46 Ellexus is the I/O profiling company. www.ellexus.com *Ellexus Ltd is a limited company registered in England & Wales* *Company registration no. 07166034* *Registered address: 198 High Street, Tonbridge, Kent TN9 1BE, UK* *Operating address: St John's Innovation Centre, Cowley Road, Cambridge CB4 0WS, UK* -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Fri Jan 5 12:39:11 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Fri, 5 Jan 2018 18:09:11 +0530 Subject: [gpfsug-discuss] Password to GUI forgotten In-Reply-To: <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu> References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu><2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com><26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu><348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu><1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov> <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu> Message-ID: Hi Kevin, If you are stuck then please open a PMR and work with the IBM support folks to get this resolved. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Buterbaugh, Kevin L" To: "Hanley, Jesse A." Cc: gpfsug main discussion list Date: 12/19/2017 01:42 AM Subject: Re: [gpfsug-discuss] Password to GUI forgotten Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Jesse, Thanks for the suggestion ? I find the following error very interesting: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/rmuser admin EFSSP0010C CLI parser: The object "admin" specified for "userID" does not exist. /root root at testnsd1# That says to me that I don?t have an admin user, which - if true - would explain why not a single password I can think of works. ;-) But as I mentioned in my original post I had this up and working earlier this fall. While I can?t prove anything, I can?t imagine a scenario where I would deliberately choose a non-default username. So if ?admin? has been the default login for the GPFS GUI all along then I am really mystified. Thanks! Kevin On Dec 18, 2017, at 1:58 PM, Hanley, Jesse A. wrote: Kevin, I ran into this a couple times using 4.2.3. This is what we used to get around it: /usr/lpp/mmfs/gui/cli/rmuser admin /usr/lpp/mmfs/gui/cli/mkuser admin -p -g Administrator,SecurityAdmin You may need to run the initgui command if those objects are present. That typically gets run on first login to the GUI. Thanks, -- Jesse From: on behalf of "Buterbaugh, Kevin L" Reply-To: gpfsug main discussion list Date: Monday, December 18, 2017 at 2:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten Hi All, Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up. Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this. Richards? suggestion below errors for me on the ?-g Administrator? part. Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked. No matter what I do, I cannot log in to the GUI. Thanks for the suggestions, though? Kevin On Dec 7, 2017, at 6:10 AM, Sobey, Richard A wrote: Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error. That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself. [root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin EFSSG0019I The user admin2 has been successfully created. EFSSG1000I The command completed successfully. Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A Sent: 07 December 2017 11:57 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out. Get Outlook for Android From: gpfsug-discuss-bounces at spectrumscale.org < gpfsug-discuss-bounces at spectrumscale.org> on behalf of Buterbaugh, Kevin L Sent: Wednesday, December 6, 2017 10:41:12 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist. /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist. /root root at testnsd1# I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case. Thanks! Kevin On Dec 6, 2017, at 4:32 PM, Joshua Kwedar wrote: Hmm.. odd. Here?s what the lsuser output should look like. # /usr/lpp/mmfs/gui/cli/lsuser Name Long name Password status Group names Failed login attempts admin active Administrator,SecurityAdmin 0 EFSSG1000I The command completed successfully. Can you try something like? # /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin From: on behalf of "Buterbaugh, Kevin L" Reply-To: gpfsug main discussion list Date: Wednesday, December 6, 2017 at 5:15 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, Sorry - should?ve mentioned that: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 EFSSG0001C Cannot validate option: login /root root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts: /root root at testnsd1# Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November. Thanks... Kevin On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) wrote: The GUI password can be changed via command line using chuser. /usr/lpp/mmfs/gui/cli/chuser Usage is as follows (where userID = admin) chuser userID {-p | -l | -a | -d | -g | --expirePassword} [-o ] Josh K On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L < Kevin.Buterbaugh at Vanderbilt.Edu> wrote: Hi All, So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall. However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI. :-( Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch? If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route. Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-) Thanks? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=B07CFG_8Wsf_UKWDoD32OlUNVRy3yFTESgTgZPki4Wg&s=A5cFcjLgIwulsbnh_cPIBawmU32lkOd4dqyl9_yzELA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jan 5 14:44:04 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 5 Jan 2018 14:44:04 +0000 Subject: [gpfsug-discuss] Password to GUI forgotten In-Reply-To: References: <5D543068-7BC1-4D7D-B7B9-D8C16EA8F4C1@vanderbilt.edu> <2CB55589-5CBD-4C75-B261-0E3B4C293014@gmail.com> <26ED3F01-AB60-4CDA-BEFC-1CB9DB716168@vanderbilt.edu> <348B3C35-E093-4EA8-8059-9671EBCFE128@vanderbilt.edu> <1790FF79-238C-4D44-9648-76B5B6D9CE13@ornl.gov> <9E821D66-8B42-4B5A-AFCD-CEBD5DFC92E2@vanderbilt.edu> Message-ID: Hi GPFS team, I did open a PMR and they (mainly Matthais) did help me get that issue resolved. Thanks for following up! Kevin On Jan 5, 2018, at 6:39 AM, IBM Spectrum Scale > wrote: Hi Kevin, If you are stuck then please open a PMR and work with the IBM support folks to get this resolved. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Buterbaugh, Kevin L" > To: "Hanley, Jesse A." > Cc: gpfsug main discussion list > Date: 12/19/2017 01:42 AM Subject: Re: [gpfsug-discuss] Password to GUI forgotten Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi Jesse, Thanks for the suggestion ? I find the following error very interesting: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/rmuser admin EFSSP0010C CLI parser: The object "admin" specified for "userID" does not exist. /root root at testnsd1# That says to me that I don?t have an admin user, which - if true - would explain why not a single password I can think of works. ;-) But as I mentioned in my original post I had this up and working earlier this fall. While I can?t prove anything, I can?t imagine a scenario where I would deliberately choose a non-default username. So if ?admin? has been the default login for the GPFS GUI all along then I am really mystified. Thanks! Kevin On Dec 18, 2017, at 1:58 PM, Hanley, Jesse A. > wrote: Kevin, I ran into this a couple times using 4.2.3. This is what we used to get around it: /usr/lpp/mmfs/gui/cli/rmuser admin /usr/lpp/mmfs/gui/cli/mkuser admin -p -g Administrator,SecurityAdmin You may need to run the initgui command if those objects are present. That typically gets run on first login to the GUI. Thanks, -- Jesse From: > on behalf of "Buterbaugh, Kevin L" > Reply-To: gpfsug main discussion list > Date: Monday, December 18, 2017 at 2:52 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Password to GUI forgotten Hi All, Sorry for the delay in getting back with you all ? didn?t mean to leave this hanging, but some higher priority things came up. Bottom line - I?m still stuck and probably going to open up a PMR with IBM after sending this. Richards? suggestion below errors for me on the ?-g Administrator? part. Other suggestions sent directly to me up to and including completely deleting the GPFS GUI and reinstalling have also not worked. No matter what I do, I cannot log in to the GUI. Thanks for the suggestions, though? Kevin On Dec 7, 2017, at 6:10 AM, Sobey, Richard A > wrote: Sorry I need to learn to read? didn?t see the ?object ?Administrator? does not exist? error. That said, my workaround for the problem of forgetting the password was to create a new ?admin2? user and use that to reset the password on admin itself. [root at gpfs cli]# ./mkuser admin2 -p Passw0rd -g Administrator,SecurityAdmin EFSSG0019I The user admin2 has been successfully created. EFSSG1000I The command completed successfully. Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org[mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A Sent: 07 December 2017 11:57 To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Password to GUI forgotten This happened to me a while back, I opened a pmr to get it sorted but it's just a case of running some cli commands. I'll dig it out. Get Outlook for Android ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org> on behalf of Buterbaugh, Kevin L > Sent: Wednesday, December 6, 2017 10:41:12 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g Administrator,SecurityAdmin EFSSP0010C CLI parser: The object "Administrator" specified for "-g" does not exist. /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 -g SecurityAdmin EFSSP0010C CLI parser: The object "SecurityAdmin" specified for "-g" does not exist. /root root at testnsd1# I?ll also add that all of the work I did earlier in the fall was with the test cluster running an earlier version of GPFS and it?s subsequently been updated to GPFS 4.2.3.5 ? not sure that?s relevant but wanted to mention it just in case. Thanks! Kevin On Dec 6, 2017, at 4:32 PM, Joshua Kwedar > wrote: Hmm.. odd. Here?s what the lsuser output should look like. # /usr/lpp/mmfs/gui/cli/lsuser Name Long name Password status Group names Failed login attempts admin active Administrator,SecurityAdmin 0 EFSSG1000I The command completed successfully. Can you try something like? # /usr/lpp/mmfs/gui/cli/mkuser admin -p abc1231 -g Administrator,SecurityAdmin From: > on behalf of "Buterbaugh, Kevin L" > Reply-To: gpfsug main discussion list > Date: Wednesday, December 6, 2017 at 5:15 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Password to GUI forgotten All, Sorry - should?ve mentioned that: /root root at testnsd1# /usr/lpp/mmfs/gui/cli/chuser admin -p abc1231 EFSSG0001C Cannot validate option: login /root root at testnsd1# /usr/lpp/mmfs/gui/cli/lsuser -Y lsuser:user:HEADER:version:reserved:reserved:Name:Long name:Password status:Group names:Failed login attempts: /root root at testnsd1# Weird - it?s like the login doesn?t exist ? but like I said, I had logged into it prior to November. Thanks... Kevin On Dec 6, 2017, at 4:10 PM, Joshua Kwedar (froz1) > wrote: The GUI password can be changed via command line using chuser. /usr/lpp/mmfs/gui/cli/chuser Usage is as follows (where userID = admin) chuser userID {-p | -l | -a | -d | -g | --expirePassword} [-o ] Josh K On Dec 6, 2017, at 4:56 PM, Buterbaugh, Kevin L > wrote: Hi All, So this is embarrassing to admit but I was playing around with setting up the GPFS GUI on our test cluster earlier this fall. However, I was gone pretty much the entire month of November for a combination of vacation and SC17 and the vacation was so relaxing that I?ve forgotten the admin password for the GPFS GUI. :-( Is there anything I can do to recover from this short of deleting the GPFS GUI related RPM?s, re-installing, and starting over from scratch? If that?s what I have to do, it?s no big deal as this is just our little 6-node test cluster, but I thought I?d ask before going down that route. Oh, and if someone has a way to accomplish this that they?d rather not share in a public mailing list for any reason, please feel free to e-mail me directly, let me know, and I won?t tell if you won?t tell (and hopefully Michael Flynn won?t tell either!)?. ;-) Thanks? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2c4a1bef0e00499c674b08d53cf622f5%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481950193934604&sdata=Nr824%2F2JVtw4EosfKUypg3mvvaxTJOeHxZETl3mN2tI%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cb77cd03d335947ea677008d53cf93ccf%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636481963514931393&sdata=Fp7gFRtowc%2BULDIPP2Wy09gdnKi7A%2BTNs8OC%2FuXpb%2Fs%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cba030691159e473668f408d53d6b930f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636482454631155492&sdata=QIpMo2L1PTQMjUDdgmf9S3WPj6ZnJs%2FEVLDumcFuqDw%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=B07CFG_8Wsf_UKWDoD32OlUNVRy3yFTESgTgZPki4Wg&s=A5cFcjLgIwulsbnh_cPIBawmU32lkOd4dqyl9_yzELA&e= ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Jan 6 10:46:16 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Sat, 6 Jan 2018 16:16:16 +0530 Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ In-Reply-To: References: Message-ID: Hi Lyle, Can you please forward this to the right folks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Keith Ball To: gpfsug-discuss at spectrumscale.org Date: 12/20/2017 04:39 AM Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was recently trying to determine the latest RHEL release that will work with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code uses Scale 4.1.0.8. A specific X.Y GSS code build, from my experience, is intended to use a specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), which no longer has EUS support from RedHat (only 6.7 still does). GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be used, which is a surprising statement given GSS/ESS code's sensitivity to OS levels (any ESS I have ever seen has never been supported on more than one version of RHEL). According to the Scale FAQ ( https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux ), A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27! Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option here). Many Thanks, Keith_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=u7hLzIRwApuSZgw6uZ_j3uewYEyRTPxRmWJhpM_UAwY&s=bBgN8s6sKOU36_ivUi5NHGarxlF3TNLPyXS-7PA34F0&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Jan 6 11:17:34 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Sat, 6 Jan 2018 16:47:34 +0530 Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group without takeover In-Reply-To: References: Message-ID: Hi Veera, Can you please help in answering Damir's query. --------------------------- My question is, is there a way of brining back the IO server into the mix without the recoverygroup takeover happening? Could I just start a gpfs and have it back in the mix as a backup server for the recoverygroup and if so, how do you do that. Right now that server is designated as primary server for the recovery group. I would like to have both IO servers in the mix for redundancy purposes. --------------------------- Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Damir Krstic To: gpfsug main discussion list Date: 12/22/2017 11:15 PM Subject: [gpfsug-discuss] ESS bring up the GPFS in recovery group without takeover Sent by: gpfsug-discuss-bounces at spectrumscale.org It's been a very frustrating couple of months with our 2 ESS systems. IBM tells us we had blueflame bug and they came on site and updated our ESS to the latest version back in middle of November. Wednesday night one of the NSD servers in one of our ESS building blocks kernel panicked. No idea why and none of the logs are insightful. We have a PMR open with IBM. I am not very confident we will get to the bottom of what's causing kernel panics on our IO servers. The system has gone down over 4 times now in 2 months. When we tried brining it back up, it rejoined the recovery group and the IO on the entire cluster locked up until we were able to find couple of compute nodes with pending state in mmfsadm dump tscomm. Killing gpfs on those nodes resolved the issue of the filesystem locking up. So far we have never been successful in brining back an IO server and not having a filesystem lock up until we find a node with pending state with tscomm. Anyway, the system was stable for few minutes until the same IO server that went down on Wednesday night went into an arbitrating mode. It never recovered. We stopped gpfs on that server and IO recovered again. We left gpfs down and cluster seems to be OK. My question is, is there a way of brining back the IO server into the mix without the recoverygroup takeover happening? Could I just start a gpfs and have it back in the mix as a backup server for the recoverygroup and if so, how do you do that. Right now that server is designated as primary server for the recovery group. I would like to have both IO servers in the mix for redundancy purposes. This ESS situation is beyond frustrating and I don't see end in sight. Any help is appreciated._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=vGabwIUw-ziMAtM7VfTppRp3S16NsgGOk5qMe50gtIQ&s=eplQuGhWVZMQ3tBLeqhCKpZ0w0rIiU-2R-UuqHYSsVA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Sat Jan 6 10:21:14 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Sat, 6 Jan 2018 15:51:14 +0530 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35 In-Reply-To: <44E99F55-25CC-48DB-9AD6-E7D6794694DC@nasa.gov> References: , <4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75@SDEB-EXC02.meteo.dz> <44E99F55-25CC-48DB-9AD6-E7D6794694DC@nasa.gov> Message-ID: Hi Lyle, Can you forward this to the right person. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]" To: gpfsug main discussion list Date: 12/19/2017 02:01 PM Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 71, Issue 35 Sent by: gpfsug-discuss-bounces at spectrumscale.org It?s not supported on SLES11 either. IBM didn?t (that I saw) talk much about this publicly or give customers a chance to provide feedback about the decision. I know it was raised at the UG in NY and I recall a number of people saying it would be a significant issue for them (myself included) as is the fact they no longer support Debian with scale 5.0. I?d raised the issue on the mailing list after the UG trying to start the discussion but IBM said they weren?t ready to talk about it publicly and I can only guess they had already set their sights and didn?t actually want feedback. This is actually pretty frustrating. I?m tempted to open an RFE but most of my RFEs either have been rejected or just sit idle so I?m not clear there?s a benefit. On December 19, 2017 at 03:08:27 EST, atmane khiredine wrote: IBM Spectrum Scale V5.0 not support RHEL 6.x Only RHEL 7.1 or later https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#linux Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz ________________________________________ De : gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] de la part de gpfsug-discuss-request at spectrumscale.org [gpfsug-discuss-request at spectrumscale.org] Envoy? : lundi 18 d?cembre 2017 22:46 ? : gpfsug-discuss at spectrumscale.org Objet : gpfsug-discuss Digest, Vol 71, Issue 35 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: FW: Spectrum Scale 5.0 now available on Fix Central (Michael L Taylor) 2. Re: gpfs 4.2.3.5 and RHEL 7.4... (Frederick Stock) 3. Re: gpfs 4.2.3.5 and RHEL 7.4... (Eric Horst) ---------------------------------------------------------------------- Message: 1 Date: Mon, 18 Dec 2017 13:27:42 -0700 From: "Michael L Taylor" To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central Message-ID: Content-Type: text/plain; charset="us-ascii" Hi Bob, Thanks for the note on 5.0.0 One correction however.... clusters can do a rolling upgrade to 5.0.0 from any 4.2.x level (not just 4.2.3). Today's Topics: 1. FW: Spectrum Scale 5.0 now available on Fix Central (Oesterlin, Robert) ---------------------------------------------------------------------- Message: 1 Date: Mon, 18 Dec 2017 19:43:35 +0000 From: "Oesterlin, Robert" To: gpfsug main discussion list Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix Central Message-ID: Content-Type: text/plain; charset="utf-8" The Scale 5.0 fix level is now up on Fix Central. You need to be at Scale 4.2.3 (cluster level) to do a rolling upgrade to this level. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: "dW-notify at us.ibm.com" Reply-To: "dW-notify at us.ibm.com" Date: Monday, December 18, 2017 at 1:27 PM Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum Scale V5.0 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum. [mid:4B32CB5C696F2849BDEF7DF9EACE884B72ACDC75 at SDEB-EXC02.meteo.dz/forums.png] gpfs at us.ibm.com< https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=YIpO2jniMJVXI1EqifZ-k4fMI36-_p1K5LqWeOadBT8&e= > replied to the IBM Spectrum Scale V5.0 announcements< https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3D2ad27846-2D6a54-2D46ba-2D96f4-2D5d6afa0df3ab&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=05bRl_SHFZieId6ukqofk_XzwZ2TSg3u-cqcGNRtobg&e= > topic thread in the General Parallel File System - Announce (GPFS - Announce)< https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0 000-2D0000-2D000000001606&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=zTY2WRO7GKP5fnLAU4K3cXg1K1VGjYOzoIDeei4xr_U&e=> forum. IBM Spectrum Scale 5.0.0.0 is now available from IBM Fix Central: http://www-933.ibm.com/support/fixcentral< https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D933.ibm.com_support_fixcentral&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=NhoaaeH3JplrJ1i1QspT5guZgy9z5td9aMxzwKGQHXk&s=iHlfdUOajEj49dqjhXGjZLG-1gZmSCZX2ZaKXFzn7n4&e= > This topic summarizes changes to the IBM Spectrum Scale licensed program and the IBM Spectrum Scale library. Summary of changes for IBM Spectrum Scale version 5 release 0.0 -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/157250d2/attachment-0001.html > ------------------------------ Message: 2 Date: Mon, 18 Dec 2017 16:10:55 -0500 From: "Frederick Stock" To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... Message-ID: Content-Type: text/plain; charset="us-ascii" Yes the integrated protocols are the Samba and Ganesha that are bundled with Spectrum Scale. These require the use of the CES component for monitoring the protocols. If you do use them then you need to wait for a release of Spectrum Scale in which the integrated protocols are also supported on RHEL 7.4. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug-discuss at spectrumscale.org Date: 12/18/2017 03:09 PM Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... Sent by: gpfsug-discuss-bounces at spectrumscale.org Currently, the IBM support matrix says: https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux that 4.2.3.5 is supported on RHEL 7.4, but with a footnote: "AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4." We don't use AFM or the install toolkit. But we *do* make fairly heavy use of mmces and nfs-ganesha - is that what they mean by "Integrated Protocols"? (We're looking at doing upgrades next month while our HPC clusters are doing their upgrades - and going to 7.4 would be nice. If there's a mine field there, I need to make sure we stay at 7.3 - plus applicable non-7.4 updates) _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_I4NKvn8aFGXCqcJARCbu4AOE&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/7c272339/attachment-0001.html > ------------------------------ Message: 3 Date: Mon, 18 Dec 2017 21:46:02 +0000 From: Eric Horst To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... Message-ID: Content-Type: text/plain; charset="utf-8" Grr, this might explain why I experienced unhappiness when I tried to start my long-delayed AFM based migration over the weekend. I had previously tested AFM and found everything working, but 7.4 may have slipped in last month. The AFM relationship seems to work but `mmafmctl premigrate` commands fail. I would revert packages if I could figure out where the issue lies. -Eric On Mon, Dec 18, 2017 at 9:10 PM, Frederick Stock wrote: > Yes the integrated protocols are the Samba and Ganesha that are bundled > with Spectrum Scale. These require the use of the CES component for > monitoring the protocols. If you do use them then you need to wait for a > release of Spectrum Scale in which the integrated protocols are also > supported on RHEL 7.4. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821> > stockf at us.ibm.com > > > > From: valdis.kletnieks at vt.edu > To: gpfsug-discuss at spectrumscale.org > Date: 12/18/2017 03:09 PM > Subject: [gpfsug-discuss] gpfs 4.2.3.5 and RHEL 7.4... > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Currently, the IBM support matrix says: > > https://www.ibm.com/support/knowledgecenter/STXKQY/ > gpfsclustersfaq.html#linux > > that 4.2.3.5 is supported on RHEL 7.4, but with a footnote: > > "AFM, Integrated Protocols, and Installation Toolkit are not supported on > RHEL 7.4." > > We don't use AFM or the install toolkit. But we *do* make fairly heavy use > of mmces and nfs-ganesha - is that what they mean by "Integrated > Protocols"? > > (We're looking at doing upgrades next month while our HPC clusters are > doing > their upgrades - and going to 7.4 would be nice. If there's a mine field > there, I need to > make sure we stay at 7.3 - plus applicable non-7.4 updates) > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m= > 3Z9HrSAviMivcR98fNZ28F-RQq7ZPp-1UZtazzLnaUU&s=HlT2amKtCbngYmKNb3_ > I4NKvn8aFGXCqcJARCbu4AOE&e= > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: < http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20171218/c01cc906/attachment.html > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss End of gpfsug-discuss Digest, Vol 71, Issue 35 ********************************************** _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=8ayFP1mF84gza8fbX73eZnWv6MoAQ-ZNctxuN0EsFjQ&s=kEhQxw29VlbFlbUt5Ml-bieqcsRiCv9aTx1cgdDS7Vk&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Mon Jan 8 10:38:14 2018 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Mon, 8 Jan 2018 18:38:14 +0800 Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ In-Reply-To: References: Message-ID: This has been asked to Yong Ze to address through another email thread, so ok to ignore this thread request. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "IBM Spectrum Scale" To: "Lyle Gayne" Cc: gpfsug-discuss-bounces at spectrumscale.org, gpfsug main discussion list Date: 01/06/2018 06:46 PM Subject: Re: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Lyle, Can you please forward this to the right folks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Keith Ball To: gpfsug-discuss at spectrumscale.org Date: 12/20/2017 04:39 AM Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was recently trying to determine the latest RHEL release that will work with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code uses Scale 4.1.0.8. A specific X.Y GSS code build, from my experience, is intended to use a specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), which no longer has EUS support from RedHat (only 6.7 still does). GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be used, which is a surprising statement given GSS/ESS code's sensitivity to OS levels (any ESS I have ever seen has never been supported on more than one version of RHEL). According to the Scale FAQ ( https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux ), A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27! Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option here). Many Thanks, Keith_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=u7hLzIRwApuSZgw6uZ_j3uewYEyRTPxRmWJhpM_UAwY&s=bBgN8s6sKOU36_ivUi5NHGarxlF3TNLPyXS-7PA34F0&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=4uHjPqPoMOnTJmSUAO5lPaI-I9TxUZG7tosuUx85sIg&s=tPqgg4TTZ9J88YaLOrAT74IcV_33GWv11K2oywQOGDw&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From lgayne at us.ibm.com Mon Jan 8 14:53:59 2018 From: lgayne at us.ibm.com (Lyle Gayne) Date: Mon, 8 Jan 2018 09:53:59 -0500 Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ In-Reply-To: References: Message-ID: Yong Ze Chen has already been responding to these. Thanks, Lyle From: IBM Spectrum Scale/Poughkeepsie/IBM To: Lyle Gayne/Poughkeepsie/IBM at IBMUS Cc: gpfsug-discuss-bounces at spectrumscale.org, gpfsug main discussion list Date: 01/06/2018 05:46 AM Subject: Re: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: Huzefa H Pancha Hi Lyle, Can you please forward this to the right folks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: Keith Ball To: gpfsug-discuss at spectrumscale.org Date: 12/20/2017 04:39 AM Subject: [gpfsug-discuss] Conflicting RHEL compatability information in the Spectrum Scale FAQ Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was recently trying to determine the latest RHEL release that will work with GSS 2.0.7 (the latest IBM version of GSS code for x86_64). This code uses Scale 4.1.0.8. A specific X.Y GSS code build, from my experience, is intended to use a specific RHEL version. For GSS 2.0, that's RHEL 6.5 (unless I am mistaken), which no longer has EUS support from RedHat (only 6.7 still does). GSS 2.0 release notes/install docs say that "RHEL 6.5 or later" can be used, which is a surprising statement given GSS/ESS code's sensitivity to OS levels (any ESS I have ever seen has never been supported on more than one version of RHEL). According to the Scale FAQ ( https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html#linux ), A 2.2, Table 27, Scale 4.1.0.x is supported on RHEL 6.2 and above (implying RHEL 6.5 and 6.7). But Table 30 indicates that the latest RHEL6 supported by Scale 4.1.0 is 6.6: for RHEL 6.7 kernel, however, indicates "From V4.1.1.2 in the 4.1.1 release" ... which contradicts Table 27! Anyone know the truth of the matter? Should I stick to RHEL 6.5 to install GSS 2.0.7, or has it been demonstrated that RHEL 6.7 works (and is supported)? (and no, Lenovo-sourced code (GSS >= 2.5) is not an option here). Many Thanks, ? Keith_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=u7hLzIRwApuSZgw6uZ_j3uewYEyRTPxRmWJhpM_UAwY&s=bBgN8s6sKOU36_ivUi5NHGarxlF3TNLPyXS-7PA34F0&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Jan 8 16:52:05 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 8 Jan 2018 16:52:05 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we?d love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again? Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Stephan.Peinkofer at lrz.de Mon Jan 8 21:10:30 2018 From: Stephan.Peinkofer at lrz.de (Peinkofer, Stephan) Date: Mon, 8 Jan 2018 21:10:30 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> , Message-ID: Dear List, my very personal experience today, using the patched kernel for SLES 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS (4.2.3-4) did not even start (the kernel modules seemed to compile fine using mmbuildgpl). Interestingly, even when I disabled PTI explicitely, using the nopti kernel option, GPFS refused to start with the same error!? mmfs.log always showed something like this: ... /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: InsModWrapper[95]: eval: line 1: 3915: Memory fault ... 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading the mmfs kernel extension ... Since I had no time to investigate the issue further and raise a ticket right now, I just downgraded to the previous kernel and everything worked again. As we have to patch at least the login nodes of our HPC clusters asap, I would also appreciate if we could get a statement from IBM how the KPTI patches are expected to interact with GPFS and if there are any (general) problems, when we can expect updated GPFS packages. Many thanks in advance. Best Regards, Stephan Peinkofer ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Buterbaugh, Kevin L Sent: Monday, January 8, 2018 5:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we?d love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again? Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Mon Jan 8 22:57:20 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 8 Jan 2018 17:57:20 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) Message-ID: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> I was thinking some more about the >32 subblock feature in scale 5.0. As mentioned by IBM the biggest advantage of that feature is on filesystems with large blocks (e.g. multiple MB). The majority of our filesystems have a block size of 1MB which got me thinking... wouldn't it be nice if they had a larger block size (there seem to be compelling performance reasons for large file I/O to do this). I'm wondering, what's the feasibility is of supporting filesystem pools of varying block sizes within a single filesystem? I thought allocation maps exist on a per-pool basis which gives me some hope it's not too hard. If one could do this then, yes, you'd still need new hardware to migrate to a larger block size (and >32 subblocks), but it could be done as part of a refresh cycle *and* (and this is the part most important to me) it could be driven entirely by the policy engine which means storage admins are largely hands off and the migration is by and large transparent to the end user. This doesn't really address the need for a tool to address a filesystem migration to 4k metadata blocks (although I wonder if something could be done to create a system_4k pool that contains 4k-aligned metadata NSDs where key data structures get re-written during a restripe in a 4k-aligned manner, but that's really grasping at straws for me). -Aaorn -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From bbanister at jumptrading.com Mon Jan 8 23:48:57 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Mon, 8 Jan 2018 23:48:57 +0000 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. Cheers, -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Monday, January 08, 2018 4:57 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) Note: External Email ------------------------------------------------- I was thinking some more about the >32 subblock feature in scale 5.0. As mentioned by IBM the biggest advantage of that feature is on filesystems with large blocks (e.g. multiple MB). The majority of our filesystems have a block size of 1MB which got me thinking... wouldn't it be nice if they had a larger block size (there seem to be compelling performance reasons for large file I/O to do this). I'm wondering, what's the feasibility is of supporting filesystem pools of varying block sizes within a single filesystem? I thought allocation maps exist on a per-pool basis which gives me some hope it's not too hard. If one could do this then, yes, you'd still need new hardware to migrate to a larger block size (and >32 subblocks), but it could be done as part of a refresh cycle *and* (and this is the part most important to me) it could be driven entirely by the policy engine which means storage admins are largely hands off and the migration is by and large transparent to the end user. This doesn't really address the need for a tool to address a filesystem migration to 4k metadata blocks (although I wonder if something could be done to create a system_4k pool that contains 4k-aligned metadata NSDs where key data structures get re-written during a restripe in a 4k-aligned manner, but that's really grasping at straws for me). -Aaorn -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From aaron.s.knister at nasa.gov Tue Jan 9 00:13:40 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Mon, 8 Jan 2018 19:13:40 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: Thanks, Bryan! That's a great use case I hadn't thought of. GPFS can already support a different block size for the system pool so in my very simplistic view of the world it's already possible (unless there's some implementation detail about the system pool that lends itself to a different block size from all other pools that doesn't apply to other non-system pools differing from each other). -Aaron On 1/8/18 6:48 PM, Bryan Banister wrote: > Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. > > I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). > > NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. > > Cheers, > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister > Sent: Monday, January 08, 2018 4:57 PM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) > > Note: External Email > ------------------------------------------------- > > I was thinking some more about the >32 subblock feature in scale 5.0. As > mentioned by IBM the biggest advantage of that feature is on filesystems > with large blocks (e.g. multiple MB). The majority of our filesystems > have a block size of 1MB which got me thinking... wouldn't it be nice if > they had a larger block size (there seem to be compelling performance > reasons for large file I/O to do this). > > I'm wondering, what's the feasibility is of supporting filesystem pools > of varying block sizes within a single filesystem? I thought allocation > maps exist on a per-pool basis which gives me some hope it's not too hard. > > If one could do this then, yes, you'd still need new hardware to migrate > to a larger block size (and >32 subblocks), but it could be done as part > of a refresh cycle *and* (and this is the part most important to me) it > could be driven entirely by the policy engine which means storage admins > are largely hands off and the migration is by and large transparent to > the end user. > > This doesn't really address the need for a tool to address a filesystem > migration to 4k metadata blocks (although I wonder if something could be > done to create a system_4k pool that contains 4k-aligned metadata NSDs > where key data structures get re-written during a restripe in a > 4k-aligned manner, but that's really grasping at straws for me). > > -Aaorn > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From Greg.Lehmann at csiro.au Tue Jan 9 03:46:48 2018 From: Greg.Lehmann at csiro.au (Greg.Lehmann at csiro.au) Date: Tue, 9 Jan 2018 03:46:48 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> , Message-ID: This had me wondering, so I tried SLES 12 SP3 and thankfully GPFS v5 still runs after the kernel patch and an mmbuildgpl. It was just a test box I had at the time, so I don't have any comments on performance. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peinkofer, Stephan Sent: Tuesday, 9 January 2018 7:11 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Dear List, my very personal experience today, using the patched kernel for SLES 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS (4.2.3-4) did not even start (the kernel modules seemed to compile fine using mmbuildgpl). Interestingly, even when I disabled PTI explicitely, using the nopti kernel option, GPFS refused to start with the same error!? mmfs.log always showed something like this: ... /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: InsModWrapper[95]: eval: line 1: 3915: Memory fault ... 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading the mmfs kernel extension ... Since I had no time to investigate the issue further and raise a ticket right now, I just downgraded to the previous kernel and everything worked again. As we have to patch at least the login nodes of our HPC clusters asap, I would also appreciate if we could get a statement from IBM how the KPTI patches are expected to interact with GPFS and if there are any (general) problems, when we can expect updated GPFS packages. Many thanks in advance. Best Regards, Stephan Peinkofer ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of Buterbaugh, Kevin L > Sent: Monday, January 8, 2018 5:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we'd love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again... Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS - until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I'm sure that everyone is aware of Meltdown and Spectre by now ... we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I'm sure that everyone is aware of Meltdown and Spectre by now ... we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ... given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we'd be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks... Kevin P.S. The "Happy New Year" wasn't intended as sarcasm ... I hope it is a good year for everyone despite how it's starting out. :-O - Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From heiner.billich at psi.ch Tue Jan 9 08:24:22 2018 From: heiner.billich at psi.ch (Billich Heinrich Rainer (PSI)) Date: Tue, 9 Jan 2018 08:24:22 +0000 Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Message-ID: Hello, I just upgraded to 4.2.3-5 and now see many failures ?ib_rdma_nic_unrecognized? in mmhealth, like Component Status Status Change Reasons ------------------------------------------------------------------------------------------ NETWORK DEGRADED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized(mlx4_0/1) mlx4_0/1 FAILED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized I didn?t see this messages with 4.2.3-4. The relevant lines in /usr/lpp/mmfs/lib/mmsysmon/NetworkService.py changed between -4 and -5. What seems to happen: I have Mellanox VPI cards with one port Infiniband and one port Ethernet. mmhealth complains about the Ethernet port. Hmm ? I did specify the active Infiniband ports only in verbsPorts, I don?t see why mmhealth cares about any other ports when it checks RDMA. So probably a bug, I?ll open a PMR unless somebody points me to a different solution. I tried but I can?t hide this event in mmhealth. Cheers, Heiner -- Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://www.psi.ch From orichards at pixitmedia.com Tue Jan 9 09:09:47 2018 From: orichards at pixitmedia.com (Orlando Richards) Date: Tue, 9 Jan 2018 09:09:47 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> Message-ID: <22812c5d-5013-18b9-7178-7a2c620770dc@pixitmedia.com> Hi all, If it helps - 4.2.3-6 compiles and starts fine on the RHEL 7.4 patched kernel: 3.10.0-693.11.6.el7.x86_64. ______ Orlando On 09/01/2018 03:46, Greg.Lehmann at csiro.au wrote: > > This had me wondering, so I tried SLES 12 SP3 and thankfully GPFS v5 > still runs after the kernel patch and an mmbuildgpl. It was just a > test box I had at the time, so I don?t have any comments on performance. > > *From:*gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of > *Peinkofer, Stephan > *Sent:* Tuesday, 9 January 2018 7:11 AM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS > > Dear List, > > my very?personal?experience today, using the patched kernel for SLES > 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS > (4.2.3-4)?did not even start (the kernel modules seemed to?compile > fine using mmbuildgpl). Interestingly, even when I disabled PTI > explicitely, using the nopti kernel option, GPFS?refused to start with > the same error!? > > mmfs.log always showed something like this: > > ... > > /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: > InsModWrapper[95]: eval: line 1: 3915: Memory fault > > ... > > 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading > the mmfs kernel extension > > ... > > Since I had no time to investigate the issue further and raise a > ticket right now, I just downgraded?to the previous kernel and > everything worked again. > > As we have to patch at least?the login nodes of our HPC clusters asap, > I would also appreciate if we could get a statement from IBM how the > KPTI patches are expected to interact with GPFS and if there are any > (general)?problems, when we can expect updated GPFS packages. > > Many thanks in advance. > Best Regards, > > Stephan Peinkofer > > ------------------------------------------------------------------------ > > *From:*gpfsug-discuss-bounces at spectrumscale.org > > > on behalf of > Buterbaugh, Kevin L > > *Sent:* Monday, January 8, 2018 5:52 PM > *To:* gpfsug main discussion list > *Subject:* Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS > > Hi GPFS Team, > > Thanks for this response. ?If it is at all possible I know that we > (and I would suspect many others are in this same boat) would greatly > appreciate a update from IBM on how a patched kernel impacts GPFS > functionality. ?Yes, we?d love to know the performance impact of the > patches on GPFS, but that pales in significance to knowing whether > GPFS version 4.x.x.x will even *start* with the patched kernel(s). > > Thanks again? > > Kevin > > > > On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: > > Kevin, > > The team is aware of Meltdown and Spectre. Due to the late > availability of production-ready test patches (they became > available today) we started today working on evaluating the impact > of applying these patches. The focus would be both on any > potential functional impacts (especially to the kernel modules > shipped with GPFS) and on the performance degradation which > affects user/kernel mode transitions. Performance characterization > will be complex, as some system calls which may get invoked often > by the mmfsd daemon will suddenly become significantly more > expensive because of the kernel changes. Depending on the main > areas affected, code changes might be possible to alleviate the > impact, by reducing frequency of certain calls, etc. Any such > changes will be deployed over time. > > At this point, we can't say what impact this will have on > stability or Performance on systems running GPFS ? until IBM > issues an official statement on this topic. We hope to have some > basic answers soon. > > > > Regards, The Spectrum Scale (GPFS) team > > ------------------------------------------------------------------------------------------------------------------ > If you feel that your question can benefit other users of Spectrum > Scale (GPFS), then please post it to the public IBM developerWroks > Forum at > https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 > . > > > If your query concerns a potential software error in Spectrum > Scale (GPFS) and you have an IBM software maintenance contract > please contact 1-800-237-5511 in the United States or your local > IBM Service Center in other countries. > > The forum is informally monitored as time permits and should not > be used for priority messages to the Spectrum Scale (GPFS) team. > > "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 > PM---Happy New Year everyone, I?m sure that everyone is aware of > Meltdown and Spectre by now ? we, like m > > From: "Buterbaugh, Kevin L" > > To: gpfsug main discussion list > > Date: 01/04/2018 01:11 PM > Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > ------------------------------------------------------------------------ > > > > > Happy New Year everyone, > > I?m sure that everyone is aware of Meltdown and Spectre by now ? > we, like many other institutions, will be patching for it at the > earliest possible opportunity. > > Our understanding is that the most serious of the negative > performance impacts of these patches will be for things like I/O > (disk / network) ? given that, we are curious if IBM has any plans > for a GPFS update that could help mitigate those impacts? Or is > there simply nothing that can be done? > > If there is a GPFS update planned for this we?d be interested in > knowing so that we could coordinate the kernel and GPFS upgrades > on our cluster. > > Thanks? > > Kevin > > P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it > is a good year for everyone despite how it?s starting out. :-O > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu > - (615)875-9633 > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- *Orlando Richards* VP Product Development, Pixit Media 07930742808|orichards at pixitmedia.com www.pixitmedia.com |Tw:@pixitmedia -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From MDIETZ at de.ibm.com Tue Jan 9 09:43:58 2018 From: MDIETZ at de.ibm.com (Mathias Dietz) Date: Tue, 9 Jan 2018 10:43:58 +0100 Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized In-Reply-To: References: Message-ID: Hello Heiner, with 4.2.3-5 mmhealth is always monitoring all ports of a configured IB adapter even if the port is not specified in verbsPorts. Development has implemented a fix which is planned to be part of 4.2.3-7 (February). To get rid of the false alarm in the meantime you could disable the Infiniband monitoring altogether. To disable Infiniband monitoring on a node: 1. Open the file /var/mmfs/mmsysmon/mmsysmonitor.conf 2. Locate the [network]section 3. Add below: ib_rdma_enable_monitoring=False 4. Save file and run "mmsysmoncontrol restart" If you have questions feel free to contact me directly by email. Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale RAS Architect & Release Lead Architect (4.2.3/5.0) --------------------------------------------------------------------------- IBM Deutschland Am Weiher 24 65451 Kelsterbach Phone: +49 70342744105 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ----------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk WittkoppSitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Billich Heinrich Rainer (PSI)" To: gpfsug main discussion list Date: 01/09/2018 09:31 AM Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I just upgraded to 4.2.3-5 and now see many failures ?ib_rdma_nic_unrecognized? in mmhealth, like Component Status Status Change Reasons ------------------------------------------------------------------------------------------ NETWORK DEGRADED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized(mlx4_0/1) mlx4_0/1 FAILED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized I didn?t see this messages with 4.2.3-4. The relevant lines in /usr/lpp/mmfs/lib/mmsysmon/NetworkService.py changed between -4 and -5. What seems to happen: I have Mellanox VPI cards with one port Infiniband and one port Ethernet. mmhealth complains about the Ethernet port. Hmm ? I did specify the active Infiniband ports only in verbsPorts, I don?t see why mmhealth cares about any other ports when it checks RDMA. So probably a bug, I?ll open a PMR unless somebody points me to a different solution. I tried but I can?t hide this event in mmhealth. Cheers, Heiner -- Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://www.psi.ch _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.childs at qmul.ac.uk Tue Jan 9 10:19:49 2018 From: p.childs at qmul.ac.uk (Peter Childs) Date: Tue, 9 Jan 2018 10:19:49 +0000 Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS In-Reply-To: <22812c5d-5013-18b9-7178-7a2c620770dc@pixitmedia.com> References: <5D655862-7F60-47F6-8BD2-A5298F73F70F@vanderbilt.edu> <22812c5d-5013-18b9-7178-7a2c620770dc@pixitmedia.com> Message-ID: <1515493189.28898.65.camel@qmul.ac.uk> On Tue, 2018-01-09 at 09:09 +0000, Orlando Richards wrote: Hi all, If it helps - 4.2.3-6 compiles and starts fine on the RHEL 7.4 patched kernel: 3.10.0-693.11.6.el7.x86_64. We have 4.2.3-4 working fine on CentOS 7.4 with patched kernel 3.10-0-693.11.6.el7.x86_64. So far we have found CentOs 7.4 (patched and unpatched) to be faster than CentOS 7.3, but CentOs 7.4 patched to be slower than unpatched CentOs 7.4. But I've not seen the details as to this testing so can't really give any further details. Peter ______ Orlando On 09/01/2018 03:46, Greg.Lehmann at csiro.au wrote: This had me wondering, so I tried SLES 12 SP3 and thankfully GPFS v5 still runs after the kernel patch and an mmbuildgpl. It was just a test box I had at the time, so I don?t have any comments on performance. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peinkofer, Stephan Sent: Tuesday, 9 January 2018 7:11 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Dear List, my very personal experience today, using the patched kernel for SLES 12.1 LTS (3.12.74-60.64.69.1) on one single VM, was that GPFS (4.2.3-4) did not even start (the kernel modules seemed to compile fine using mmbuildgpl). Interestingly, even when I disabled PTI explicitely, using the nopti kernel option, GPFS refused to start with the same error!? mmfs.log always showed something like this: ... /usr/lpp/mmfs/bin/runmmfs[336]: .[213]: loadKernelExt[674]: InsModWrapper[95]: eval: line 1: 3915: Memory fault ... 2018-01-08_09:01:27.520+0100 runmmfs: error in loading or unloading the mmfs kernel extension ... Since I had no time to investigate the issue further and raise a ticket right now, I just downgraded to the previous kernel and everything worked again. As we have to patch at least the login nodes of our HPC clusters asap, I would also appreciate if we could get a statement from IBM how the KPTI patches are expected to interact with GPFS and if there are any (general) problems, when we can expect updated GPFS packages. Many thanks in advance. Best Regards, Stephan Peinkofer ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of Buterbaugh, Kevin L > Sent: Monday, January 8, 2018 5:52 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Hi GPFS Team, Thanks for this response. If it is at all possible I know that we (and I would suspect many others are in this same boat) would greatly appreciate a update from IBM on how a patched kernel impacts GPFS functionality. Yes, we?d love to know the performance impact of the patches on GPFS, but that pales in significance to knowing whether GPFS version 4.x.x.x will even *start* with the patched kernel(s). Thanks again? Kevin On Jan 4, 2018, at 4:55 PM, IBM Spectrum Scale > wrote: Kevin, The team is aware of Meltdown and Spectre. Due to the late availability of production-ready test patches (they became available today) we started today working on evaluating the impact of applying these patches. The focus would be both on any potential functional impacts (especially to the kernel modules shipped with GPFS) and on the performance degradation which affects user/kernel mode transitions. Performance characterization will be complex, as some system calls which may get invoked often by the mmfsd daemon will suddenly become significantly more expensive because of the kernel changes. Depending on the main areas affected, code changes might be possible to alleviate the impact, by reducing frequency of certain calls, etc. Any such changes will be deployed over time. At this point, we can't say what impact this will have on stability or Performance on systems running GPFS ? until IBM issues an official statement on this topic. We hope to have some basic answers soon. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. "Buterbaugh, Kevin L" ---01/04/2018 01:11:59 PM---Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like m From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/04/2018 01:11 PM Subject: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Happy New Year everyone, I?m sure that everyone is aware of Meltdown and Spectre by now ? we, like many other institutions, will be patching for it at the earliest possible opportunity. Our understanding is that the most serious of the negative performance impacts of these patches will be for things like I/O (disk / network) ? given that, we are curious if IBM has any plans for a GPFS update that could help mitigate those impacts? Or is there simply nothing that can be done? If there is a GPFS update planned for this we?d be interested in knowing so that we could coordinate the kernel and GPFS upgrades on our cluster. Thanks? Kevin P.S. The ?Happy New Year? wasn?t intended as sarcasm ? I hope it is a good year for everyone despite how it?s starting out. :-O ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Orlando Richards VP Product Development, Pixit Media 07930742808 | orichards at pixitmedia.com www.pixitmedia.com | Tw:@pixitmedia [http://pixitmedia.com/sig/pixitmedia-9-2018.png] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Peter Childs ITS Research Storage Queen Mary, University of London -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcram at ddn.com Tue Jan 9 15:30:36 2018 From: jcram at ddn.com (Jeno Cram) Date: Tue, 9 Jan 2018 15:30:36 +0000 Subject: [gpfsug-discuss] SMB with LDAP Message-ID: Has anyone had any luck implementing CES with SMB using LDAP? This link doesn?t seem to have all of the information required to getting it working properly. https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_updateldapsmb.htm 1. I?m assuming smbpasswd -a is still required for all users? 2. What keeps the LDAP/SMB passwords in sync? 3. What access does the bind user need in LDAP? 4. What special requirements are required for Windows 10 clients to connect? Jeno Cram | Lead Application Support Engineer jcram at ddn.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Jan 9 15:51:03 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 9 Jan 2018 15:51:03 +0000 Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized In-Reply-To: References: Message-ID: <7828e86d6c494dfdb95a6dbbd6439f09@jumptrading.com> I can't help but comment that it's amazing that GPFS is using a txt config file instead of requiring a command run that stores config data into a non-editable (but still editable) flat file database... Wow 2018!! Hahahahaha! -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Mathias Dietz Sent: Tuesday, January 09, 2018 3:44 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Note: External Email ________________________________ Hello Heiner, with 4.2.3-5 mmhealth is always monitoring all ports of a configured IB adapter even if the port is not specified in verbsPorts. Development has implemented a fix which is planned to be part of 4.2.3-7 (February). To get rid of the false alarm in the meantime you could disable the Infiniband monitoring altogether. To disable Infiniband monitoring on a node: 1. Open the file /var/mmfs/mmsysmon/mmsysmonitor.conf 2. Locate the [network]section 3. Add below: ib_rdma_enable_monitoring=False 4. Save file and run "mmsysmoncontrol restart" If you have questions feel free to contact me directly by email. Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale RAS Architect & Release Lead Architect (4.2.3/5.0) --------------------------------------------------------------------------- IBM Deutschland Am Weiher 24 65451 Kelsterbach Phone: +49 70342744105 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ----------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk WittkoppSitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Billich Heinrich Rainer (PSI)" > To: gpfsug main discussion list > Date: 01/09/2018 09:31 AM Subject: [gpfsug-discuss] mmhealth with 4.2.3-5 gives many false alarms ib_rdma_nic_unrecognized Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hello, I just upgraded to 4.2.3-5 and now see many failures 'ib_rdma_nic_unrecognized' in mmhealth, like Component Status Status Change Reasons ------------------------------------------------------------------------------------------ NETWORK DEGRADED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized(mlx4_0/1) mlx4_0/1 FAILED 2018-01-06 15:57:21 ib_rdma_nic_unrecognized I didn't see this messages with 4.2.3-4. The relevant lines in /usr/lpp/mmfs/lib/mmsysmon/NetworkService.py changed between -4 and -5. What seems to happen: I have Mellanox VPI cards with one port Infiniband and one port Ethernet. mmhealth complains about the Ethernet port. Hmm - I did specify the active Infiniband ports only in verbsPorts, I don't see why mmhealth cares about any other ports when it checks RDMA. So probably a bug, I'll open a PMR unless somebody points me to a different solution. I tried but I can't hide this event in mmhealth. Cheers, Heiner -- Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://www.psi.ch _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jan 9 22:44:04 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 17:44:04 -0500 Subject: [gpfsug-discuss] interrupted inode expansion Message-ID: I just ran into this in our test environment with 4.2.3.6: http://www-01.ibm.com/support/docview.wss?crawler=1&uid=isg1IV94666 Has anyone else run into this? I'm about to request an efix for it. -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Tue Jan 9 22:47:59 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 17:47:59 -0500 Subject: [gpfsug-discuss] interrupted inode expansion In-Reply-To: References: Message-ID: <0b0daec8-11aa-8d6d-961e-75120520a125@nasa.gov> Question for the Scale team-- is this bug present in the 4.1 stream (or the 5.0 stream for that matter)? On 1/9/18 5:44 PM, Aaron Knister wrote: > I just ran into this in our test environment with 4.2.3.6: > > http://www-01.ibm.com/support/docview.wss?crawler=1&uid=isg1IV94666 > > Has anyone else run into this? I'm about to request an efix for it. > > -Aaron > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From christof.schmitt at us.ibm.com Tue Jan 9 23:34:06 2018 From: christof.schmitt at us.ibm.com (Christof Schmitt) Date: Tue, 9 Jan 2018 23:34:06 +0000 Subject: [gpfsug-discuss] SMB with LDAP In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Wed Jan 10 00:56:08 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 19:56:08 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: <3a1a5b22-76cc-2aa1-7d89-c03619f8d679@nasa.gov> Brian, I stole your wording and created an RFE for this: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115012 -Aaron On 1/8/18 6:48 PM, Bryan Banister wrote: > Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. > > I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). > > NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. > > Cheers, > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister > Sent: Monday, January 08, 2018 4:57 PM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) > > Note: External Email > ------------------------------------------------- > > I was thinking some more about the >32 subblock feature in scale 5.0. As > mentioned by IBM the biggest advantage of that feature is on filesystems > with large blocks (e.g. multiple MB). The majority of our filesystems > have a block size of 1MB which got me thinking... wouldn't it be nice if > they had a larger block size (there seem to be compelling performance > reasons for large file I/O to do this). > > I'm wondering, what's the feasibility is of supporting filesystem pools > of varying block sizes within a single filesystem? I thought allocation > maps exist on a per-pool basis which gives me some hope it's not too hard. > > If one could do this then, yes, you'd still need new hardware to migrate > to a larger block size (and >32 subblocks), but it could be done as part > of a refresh cycle *and* (and this is the part most important to me) it > could be driven entirely by the policy engine which means storage admins > are largely hands off and the migration is by and large transparent to > the end user. > > This doesn't really address the need for a tool to address a filesystem > migration to 4k metadata blocks (although I wonder if something could be > done to create a system_4k pool that contains 4k-aligned metadata NSDs > where key data structures get re-written during a restripe in a > 4k-aligned manner, but that's really grasping at straws for me). > > -Aaorn > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Wed Jan 10 00:57:48 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 19:57:48 -0500 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: <3a1a5b22-76cc-2aa1-7d89-c03619f8d679@nasa.gov> References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> <3a1a5b22-76cc-2aa1-7d89-c03619f8d679@nasa.gov> Message-ID: <6a1912f4-d519-f20e-ceb1-af55e2cb7f31@nasa.gov> *Bryan (sorry for the typo) On 1/9/18 7:56 PM, Aaron Knister wrote: > Brian, > > I stole your wording and created an RFE for this: > > http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115012 > > -Aaron > > On 1/8/18 6:48 PM, Bryan Banister wrote: >> Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. >> >> I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). >> >> NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. >> >> Cheers, >> -Bryan >> >> -----Original Message----- >> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister >> Sent: Monday, January 08, 2018 4:57 PM >> To: gpfsug main discussion list >> Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) >> >> Note: External Email >> ------------------------------------------------- >> >> I was thinking some more about the >32 subblock feature in scale 5.0. As >> mentioned by IBM the biggest advantage of that feature is on filesystems >> with large blocks (e.g. multiple MB). The majority of our filesystems >> have a block size of 1MB which got me thinking... wouldn't it be nice if >> they had a larger block size (there seem to be compelling performance >> reasons for large file I/O to do this). >> >> I'm wondering, what's the feasibility is of supporting filesystem pools >> of varying block sizes within a single filesystem? I thought allocation >> maps exist on a per-pool basis which gives me some hope it's not too hard. >> >> If one could do this then, yes, you'd still need new hardware to migrate >> to a larger block size (and >32 subblocks), but it could be done as part >> of a refresh cycle *and* (and this is the part most important to me) it >> could be driven entirely by the policy engine which means storage admins >> are largely hands off and the migration is by and large transparent to >> the end user. >> >> This doesn't really address the need for a tool to address a filesystem >> migration to 4k metadata blocks (although I wonder if something could be >> done to create a system_4k pool that contains 4k-aligned metadata NSDs >> where key data structures get re-written during a restripe in a >> 4k-aligned manner, but that's really grasping at straws for me). >> >> -Aaorn >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ________________________________ >> >> Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From aaron.s.knister at nasa.gov Wed Jan 10 02:01:13 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 21:01:13 -0500 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central In-Reply-To: References: Message-ID: <7fe4019d-8034-489a-631b-0c7c1134cce2@nasa.gov> Hi Steve, I can definitely understand the want for a newer compiler, but here's a question for you and the Scale team-- what requires the newer compiler? The userspace tools or the kernel module? I ask because as we all know it can be quite easy to install a modern build chain even on an older operating system or even build binaries on a new operating system that will run on an older operating system. Surely the userspace components could be built with a newer compiler and the gplbin layer would just be built with whatever compiler is present on the system. I suspect there's more complexity and subtlety to the issue but I thought I'd at least ask. I know SLES12 has been out for a while now but that doesn't mean we can just pick up and move to a new OS without some difficulty, especially if the older OS is working just fine :) -Aaron On 12/19/17 8:18 AM, Steve Duersch wrote: > As Mike Taylor pointed out in a previous post this was an incorrect > statement. > You can be at 4.2.x (ie 4.2.0, 4.2.1, 4.2.2, or 4.2.3) and still do a > rolling upgrade. > The minReleaseLevel is not pertinent to a rolling upgrade. The running > daemon is the important part. So you can't have any 4.1.x nodes in your > cluster and do a rolling upgrade to 5.0. > > Also, Aaron, as to the OS support. This decision was not made without > some angst. As I mentioned at the user group meeting in NYC...the key > point is that we would like to get to a more current compiler. This will > allow us to take advantage of newer features and functions and hopefully > make the code better for customers. SLES 12 has been around for over 2 > years. > > I hope this helps give some thinking behind the decision. > > > Steve Duersch > Spectrum Scale > 845-433-7902 > IBM Poughkeepsie, New York > > >> Today's Topics: >> >> ? ?1. Re: Spectrum Scale 5.0 now available on Fix Central >> ? ? ? (Sobey, Richard A) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 19 Dec 2017 09:06:08 +0000 >> From: "Sobey, Richard A" >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix >> ? ?Central >> Message-ID: >> ? ? >> > >> ? ? >> Content-Type: text/plain; charset="utf-8" >> >> Hi Robert >> >> Do you mean the minReleaseLevel from mmlsconfig or just making sure >> all the nodes are running 4.2.3? >> >> Cheers! >> Richard >> >> From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug- >> discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert >> Sent: 18 December 2017 19:44 >> To: gpfsug main discussion list >> Subject: [gpfsug-discuss] FW: Spectrum Scale 5.0 now available on Fix > Central >> >> The Scale 5.0 fix level is now up on Fix Central. >> >> You need to be at Scale 4.2.3 (cluster level) to do a rolling >> upgrade to this level. >> >> >> Bob Oesterlin >> Sr Principal Storage Engineer, Nuance >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From Robert.Oesterlin at nuance.com Wed Jan 10 02:12:16 2018 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 10 Jan 2018 02:12:16 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central Message-ID: I'm in the same boat here with deprecated support for RH6. We have a number of older applications supporting product that won't run on RH7. I'm still pondering what I'm going to do here. Or stay perpetually stuck on 4.2.3 on some clusters. Bob Oesterlin Sr Principal Storage Engineer, Nuance ?On 1/9/18, 8:02 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: I know SLES12 has been out for a while now but that doesn't mean we can just pick up and move to a new OS without some difficulty, especially if the older OS is working just fine :) From aaron.s.knister at nasa.gov Wed Jan 10 02:32:24 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Tue, 9 Jan 2018 21:32:24 -0500 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central In-Reply-To: References: Message-ID: I just opened an RFE for RHEL6, SLES11 and Debian support in Scale 5.0: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115019 -Aaron On 1/9/18 9:12 PM, Oesterlin, Robert wrote: > I'm in the same boat here with deprecated support for RH6. We have a number of older applications supporting product that won't run on RH7. I'm still pondering what I'm going to do here. Or stay perpetually stuck on 4.2.3 on some clusters. > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > ?On 1/9/18, 8:02 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: > > I know SLES12 has been out for a while now but that doesn't mean we can > just pick up and move to a new OS without some difficulty, especially if > the older OS is working just fine :) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From sandeep.patil at in.ibm.com Wed Jan 10 06:43:11 2018 From: sandeep.patil at in.ibm.com (Sandeep Ramesh) Date: Wed, 10 Jan 2018 12:13:11 +0530 Subject: [gpfsug-discuss] Latest Technical Blogs on Spectrum Scale In-Reply-To: References: Message-ID: Dear User Group Members, Here are list of development blogs in the last quarter. Passing it to this email group as Doris had got a feedback in the UG meetings to notify the members with the latest updates periodically. Genomic Workloads ? How To Get it Right From Infrastructure Point Of View. https://developer.ibm.com/storage/2018/01/06/genomic-workloads-get-right-infrastructure-point-view/ IBM Spectrum Scale Versus Apache Hadoop HDFS https://developer.ibm.com/storage/2018/01/10/spectrumscale_vs_hdfs/ ESS Fault Tolerance https://developer.ibm.com/storage/2018/01/09/ess-fault-tolerance/ IBM Spectrum Scale MMFSCK ? Savvy Enhancements https://developer.ibm.com/storage/2018/01/05/ibm-spectrum-scale-mmfsck-savvy-enhancements/ ESS Disk Management https://developer.ibm.com/storage/2018/01/02/ess-disk-management/ IBM Spectrum Scale Object Protocol On Ubuntu https://developer.ibm.com/storage/2018/01/01/ibm-spectrum-scale-object-protocol-ubuntu/ IBM Spectrum Scale 5.0 ? Whats new in Unified File and Object https://developer.ibm.com/storage/2017/12/20/ibm-spectrum-scale-5-0-whats-new-object/ A Complete Guide to ? Protocol Problem Determination Guide for IBM Spectrum Scale? ? Part 1 https://developer.ibm.com/storage/2017/12/19/complete-guide-protocol-problem-determination-guide-ibm-spectrum-scale-1/ IBM Spectrum Scale installation toolkit ? enhancements over releases https://developer.ibm.com/storage/2017/12/15/ibm-spectrum-scale-installation-toolkit-enhancements-releases/ Network requirements in an Elastic Storage Server Setup https://developer.ibm.com/storage/2017/12/13/network-requirements-in-an-elastic-storage-server-setup/ Co-resident migration with Transparent cloud tierin https://developer.ibm.com/storage/2017/12/05/co-resident-migration-transparent-cloud-tierin/ IBM Spectrum Scale on Hortonworks HDP Hadoop clusters : A Complete Big Data Solution https://developer.ibm.com/storage/2017/12/05/ibm-spectrum-scale-hortonworks-hdp-hadoop-clusters-complete-big-data-solution/ Big data analytics with Spectrum Scale using remote cluster mount & multi-filesystem support https://developer.ibm.com/storage/2017/11/28/big-data-analytics-spectrum-scale-using-remote-cluster-mount-multi-filesystem-support/ IBM Spectrum Scale HDFS Transparency Short Circuit Write Support https://developer.ibm.com/storage/2017/11/28/ibm-spectrum-scale-hdfs-transparency-short-circuit-write-support/ IBM Spectrum Scale HDFS Transparency Federation Support https://developer.ibm.com/storage/2017/11/27/ibm-spectrum-scale-hdfs-transparency-federation-support/ How to configure and performance tuning different system workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-different-system-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Spark workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-spark-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning database workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/27/configure-performance-tuning-database-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ How to configure and performance tuning Hadoop workloads on IBM Spectrum Scale Sharing Nothing Cluster https://developer.ibm.com/storage/2017/11/24/configure-performance-tuning-hadoop-workloads-ibm-spectrum-scale-sharing-nothing-cluster/ IBM Spectrum Scale Sharing Nothing Cluster Performance Tuning https://developer.ibm.com/storage/2017/11/24/ibm-spectrum-scale-sharing-nothing-cluster-performance-tuning/ How to Configure IBM Spectrum Scale? with NIS based Authentication. https://developer.ibm.com/storage/2017/11/21/configure-ibm-spectrum-scale-nis-based-authentication/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media From: Sandeep Ramesh/India/IBM To: gpfsug-discuss at spectrumscale.org Cc: Doris Conti/Poughkeepsie/IBM at IBMUS Date: 11/16/2017 08:15 PM Subject: Latest Technical Blogs on Spectrum Scale Dear User Group members, Here are the Development Blogs in last 3 months on Spectrum Scale Technical Topics. Spectrum Scale Monitoring ? Know More ? https://developer.ibm.com/storage/2017/11/16/spectrum-scale-monitoring-know/ IBM Spectrum Scale 5.0 Release ? What?s coming ! https://developer.ibm.com/storage/2017/11/14/ibm-spectrum-scale-5-0-release-whats-coming/ Four Essentials things to know for managing data ACLs on IBM Spectrum Scale? from Windows https://developer.ibm.com/storage/2017/11/13/four-essentials-things-know-managing-data-acls-ibm-spectrum-scale-windows/ GSSUTILS: A new way of running SSR, Deploying or Upgrading ESS Server https://developer.ibm.com/storage/2017/11/13/gssutils/ IBM Spectrum Scale Object Authentication https://developer.ibm.com/storage/2017/11/02/spectrum-scale-object-authentication/ Video Surveillance ? Choosing the right storage https://developer.ibm.com/storage/2017/11/02/video-surveillance-choosing-right-storage/ IBM Spectrum scale object deep dive training with problem determination https://www.slideshare.net/SmitaRaut/ibm-spectrum-scale-object-deep-dive-training Spectrum Scale as preferred software defined storage for Ubuntu OpenStack https://developer.ibm.com/storage/2017/09/29/spectrum-scale-preferred-software-defined-storage-ubuntu-openstack/ IBM Elastic Storage Server 2U24 Storage ? an All-Flash offering, a performance workhorse https://developer.ibm.com/storage/2017/10/06/ess-5-2-flash-storage/ A Complete Guide to Configure LDAP-based authentication with IBM Spectrum Scale? for File Access https://developer.ibm.com/storage/2017/09/21/complete-guide-configure-ldap-based-authentication-ibm-spectrum-scale-file-access/ Deploying IBM Spectrum Scale on AWS Quick Start https://developer.ibm.com/storage/2017/09/18/deploy-ibm-spectrum-scale-on-aws-quick-start/ Monitoring Spectrum Scale Object metrics https://developer.ibm.com/storage/2017/09/14/monitoring-spectrum-scale-object-metrics/ Tier your data with ease to Spectrum Scale Private Cloud(s) using Moonwalk Universal https://developer.ibm.com/storage/2017/09/14/tier-data-ease-spectrum-scale-private-clouds-using-moonwalk-universal/ Why do I see owner as ?Nobody? for my export mounted using NFSV4 Protocol on IBM Spectrum Scale?? https://developer.ibm.com/storage/2017/09/08/see-owner-nobody-export-mounted-using-nfsv4-protocol-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory and LDAP https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-ldap/ IBM Spectrum Scale? Authentication using Active Directory and RFC2307 https://developer.ibm.com/storage/2017/09/01/ibm-spectrum-scale-authentication-using-active-directory-rfc2307/ High Availability Implementation with IBM Spectrum Virtualize and IBM Spectrum Scale https://developer.ibm.com/storage/2017/08/30/high-availability-implementation-ibm-spectrum-virtualize-ibm-spectrum-scale/ 10 Frequently asked Questions on configuring Authentication using AD + AUTO ID mapping on IBM Spectrum Scale?. https://developer.ibm.com/storage/2017/08/04/10-frequently-asked-questions-configuring-authentication-using-ad-auto-id-mapping-ibm-spectrum-scale/ IBM Spectrum Scale? Authentication using Active Directory https://developer.ibm.com/storage/2017/07/30/ibm-spectrum-scale-auth-using-active-directory/ Five cool things that you didn?t know Transparent Cloud Tiering on Spectrum Scale can do https://developer.ibm.com/storage/2017/07/29/five-cool-things-didnt-know-transparent-cloud-tiering-spectrum-scale-can/ IBM Spectrum Scale GUI videos https://developer.ibm.com/storage/2017/07/25/ibm-spectrum-scale-gui-videos/ IBM Spectrum Scale? Authentication ? Planning for NFS Access https://developer.ibm.com/storage/2017/07/24/ibm-spectrum-scale-planning-nfs-access/ For more : Search /browse here: https://developer.ibm.com/storage/blog Consolidation list: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/White%20Papers%20%26%20Media -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Wed Jan 10 14:32:09 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 10 Jan 2018 14:32:09 +0000 Subject: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central In-Reply-To: References: Message-ID: <089336cb58b6443e98d28bd4d6b16724@jumptrading.com> I felt some sort of obligation to stand by my own wording... +1 vote for me!!! -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister Sent: Tuesday, January 09, 2018 8:32 PM To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Spectrum Scale 5.0 now available on Fix Central Note: External Email ------------------------------------------------- I just opened an RFE for RHEL6, SLES11 and Debian support in Scale 5.0: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=115019 -Aaron On 1/9/18 9:12 PM, Oesterlin, Robert wrote: > I'm in the same boat here with deprecated support for RH6. We have a number of older applications supporting product that won't run on RH7. I'm still pondering what I'm going to do here. Or stay perpetually stuck on 4.2.3 on some clusters. > > > Bob Oesterlin > Sr Principal Storage Engineer, Nuance > > > ?On 1/9/18, 8:02 PM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Aaron Knister" wrote: > > I know SLES12 has been out for a while now but that doesn't mean we can > just pick up and move to a new OS without some difficulty, especially if > the older OS is working just fine :) > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From Achim.Rehor at de.ibm.com Wed Jan 10 15:57:48 2018 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Wed, 10 Jan 2018 16:57:48 +0100 Subject: [gpfsug-discuss] interrupted inode expansion In-Reply-To: <0b0daec8-11aa-8d6d-961e-75120520a125@nasa.gov> References: <0b0daec8-11aa-8d6d-961e-75120520a125@nasa.gov> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: From vpuvvada at in.ibm.com Fri Jan 12 06:50:49 2018 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Fri, 12 Jan 2018 12:20:49 +0530 Subject: [gpfsug-discuss] Use AFM for migration of many small files In-Reply-To: <1515064616.28898.32.camel@qmul.ac.uk> References: <467FB293-D33B-46F4-BA1B-A5CB4D28DDE6@psi.ch> <1515064616.28898.32.camel@qmul.ac.uk> Message-ID: >The plan is to load the new cache from the old GPFS then dump once the cache is full. >We've already increase numThreashThreads from 4 to 8 and seen only marginal improvements, we could attempt to increase this further. AFM have replication performance issues with small files on high latency networks. There is a plan to fix these issues. >I'm also wondering if its worth increasing the Refresh Intervals to speed up read of already cache files. At this stage we want to fill the cache and don't care about write back until we switch the target to the >new NFS/GPFS from our old GPFS storage to a new box back at our off-site location, (otherwise known as the office) Increasing the refresh intervals will improve the application performance at cache site. It is better to set large refresh intervals if the cache is the only writer. ~Venkat (vpuvvada at in.ibm.com) From: Peter Childs To: "gpfsug-discuss at spectrumscale.org" Date: 01/04/2018 04:47 PM Subject: Re: [gpfsug-discuss] Use AFM for migration of many small files Sent by: gpfsug-discuss-bounces at spectrumscale.org We are doing something very similar using 4.2.3-4 and GPFS 4.2.1-1 on the nfs backend. Did you have any success? The plan is to load the new cache from the old GPFS then dump once the cache is full. We've already increase numThreashThreads from 4 to 8 and seen only marginal improvements, we could attempt to increase this further. I'm also wondering if its worth increasing the Refresh Intervals to speed up read of already cache files. At this stage we want to fill the cache and don't care about write back until we switch the target to the new NFS/GPFS from our old GPFS storage to a new box back at our off-site location, (otherwise known as the office) [root at afmgateway1 scratch]# mmlsfileset home home -L --afm Filesets in file system 'home': Attributes for fileset home: ============================= Status Linked Path /data2/home Id 42 Root inode 1343225859 Parent Id 0 Created Wed Jan 3 12:32:33 2018 Comment Inode space 41 Maximum number of inodes 100000000 Allocated inodes 15468544 Permission change flag chmodAndSetacl afm-associated Yes Target nfs://afm1/gpfs/data1/afm/home Mode single-writer File Lookup Refresh Interval 30 (default) File Open Refresh Interval 30 (default) Dir Lookup Refresh Interval 60 (default) Dir Open Refresh Interval 60 (default) Async Delay 15 (default) Last pSnapId 0 Display Home Snapshots no Number of Gateway Flush Threads 8 Prefetch Threshold 0 (default) Eviction Enabled no Thanks in advance. Peter Childs On Tue, 2017-09-05 at 19:57 +0530, Venkateswara R Puvvada wrote: Which version of Spectrum Scale ? What is the fileset mode ? >We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here >I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. How was the performance measured ? If parallel IO is enabled, AFM uses multiple gateway nodes to prefetch the large files (if file size if more than 1GB). Performance difference between small and lager file is huge (1000MB - 150MB = 850MB) here, and generally it is not the case. How many files were present in list file for prefetch ? Could you also share full internaldump from the gateway node ? >I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few >read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. AFM prefetches the files on multiple threads. Default flush threads for prefetch are 36 (fileset.afmNumFlushThreads (default 4) + afmNumIOFlushThreads (default 32)). >Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and >I need to look elsewhere to get better performance for prefetch of many smaller files? See above, AFM reads files on multiple threads parallelly. Try increasing the afmNumFlushThreads on fileset and verify if it improves the performance. ~Venkat (vpuvvada at in.ibm.com) From: "Billich Heinrich Rainer (PSI)" To: "gpfsug-discuss at spectrumscale.org" Date: 09/04/2017 10:18 PM Subject: [gpfsug-discuss] Use AFM for migration of many small files Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, We use AFM prefetch to migrate data between two clusters (using NFS). This works fine with large files, say 1+GB. But we have millions of smaller files, about 1MB each. Here I see just ~150MB/s ? compare this to the 1000+MB/s we get for larger files. I assume that we would need more parallelism, does prefetch pull just one file at a time? So each file needs some or many metadata operations plus a single or just a few read and writes. Doing this sequentially adds up all the latencies of NFS+GPFS. This is my explanation. With larger files gpfs prefetch on home will help. Please can anybody comment: Is this right, does AFM prefetch handle one file at a time in a sequential manner? And is there any way to change this behavior? Or am I wrong and I need to look elsewhere to get better performance for prefetch of many smaller files? We will migrate several filesets in parallel, but still with individual filesets up to 350TB in size 150MB/s isn?t fun. Also just about 150 files/s seconds looks poor. The setup is quite new, hence there may be other places to look at. It?s all RHEL7 an spectrum scale 4.2.2-3 on the afm cache. Thank you, Heiner --, Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://urldefense.proofpoint.com/v2/url?u=https-3A__www.psi.ch&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=eHcVdovN10-m-Qk0Ln2qvol3pkKNFwrzz2wgf1zXVXE&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=4y79Y-3M5sHV1Fm6aUFPEDIl8W5jxVP64XSlBsAYBb4&s=LbRyuSM_djs0FDXr27hPottQHAn3OGcivpyRcIDBN3U&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=eMSpfXQgE3wMSVM0G0vCYr6LEgERhP8iGfw3uNk8lLs&s=mcQ13uhvwS4yPbA2uCmwccc7l4mjTmL2fAdPLimS0Hc&e= -- Peter Childs ITS Research Storage Queen Mary, University of London _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=07QQkI0Rg8NyUEgPIuJwfg3elEXqTpOjIFpy2WbaEg0&s=kGEDPbMo64yU7Tcwu61ggT89tfq_3QdX-r6NoANXh78&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From coetzee.ray at gmail.com Fri Jan 12 10:29:43 2018 From: coetzee.ray at gmail.com (Ray Coetzee) Date: Fri, 12 Jan 2018 10:29:43 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale Message-ID: I'd like to ask the group of their experiences in improving the performance of applications that use mmap calls against files on Spectrum Scale. Besides using an NFS export from CES instead of a native GPFS mount, or precaching the dataset into the pagepool, what other approaches are there to offset the performance hit of the 4K IO size? Kind regards Ray Coetzee -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Jan 12 14:45:29 2018 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 12 Jan 2018 14:45:29 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: Message-ID: what version of Scale are you using right now ? On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee wrote: > I'd like to ask the group of their experiences in improving the > performance of applications that use mmap calls against files on Spectrum > Scale. > > Besides using an NFS export from CES instead of a native GPFS mount, or > precaching the dataset into the pagepool, what other approaches are there > to offset the performance hit of the 4K IO size? > > Kind regards > > Ray Coetzee > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Thu Jan 11 14:16:26 2018 From: john.hearns at asml.com (John Hearns) Date: Thu, 11 Jan 2018 14:16:26 +0000 Subject: [gpfsug-discuss] System running out of memory - SUnreclaim is huge Message-ID: I am having problems with GPFS servers running out of memory. We have an open PMR for this, however if anyone has seen this or has any ideas I would be grateful for a heads up. Servers have 128 Gbytes f RAM, kernel 2.6.32-573.18.1.el6.x86_64, GPFS version 4.2.3.4 In the latest incident the free memory went to below 1Gbyte, and we started to have processes killed, including our monitoring setup. I shut down GPFS on that server and /proc/meminfo still shows: Slab: 111192296 kB SReclaimable: 29020 kB SUnreclaim: 111163276 kB Am I barking up a wrong tree here and pointing the finger at GPFS? Something is causing the scsi_data_buffer slab memory usage (see below). One thing I did yesterday was to change the disk scheduler for each disk from cfq to dealine (as recommended in the tuning guide) However the server was already in short memory at that point. Slabtop shows Active / Total Objects (% used) : -306803185 / -306722574 (100.0%) Active / Total Slabs (% used) : 27749714 / 27749719 (100.0%) Active / Total Caches (% used) : 115 / 198 (58.1%) Active / Total Size (% used) : 93857848.58K / 93872319.47K (100.0%) Minimum / Average / Maximum Object : 0.02K / 0.02K / 4096.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 3987822096 3987821817 0% 0.02K 27693209 144 110772836K scsi_data_buffer 91155 64448 70% 0.06K 1545 59 6180K size-64 36064 32035 88% 0.03K 322 112 1288K size-32 35505 34334 96% 0.25K 2367 15 9468K skbuff_head_cache 33876 33874 99% 8.00K 33876 1 271008K size-8192 33804 33615 99% 0.14K 1252 27 5008K sysfs_dir_cache -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Fri Jan 12 16:12:13 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 12 Jan 2018 16:12:13 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: Message-ID: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> You could put all of your data onto SSDs in a RAID1 configuration so that you don?t have insane read-modify-write penalties on writes (RAID1) and avoid horrible seek thrashing that spinning rust requires (SSD random access medium) for your 4K I/O operations. One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not to touch it?. He gave many reasons why mmap operations on a distributed file system is incredibly hard and not recommended. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sven Oehme Sent: Friday, January 12, 2018 8:45 AM To: coetzee.ray at gmail.com; gpfsug main discussion list Subject: Re: [gpfsug-discuss] mmap performance against Spectrum Scale Note: External Email ________________________________ what version of Scale are you using right now ? On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee > wrote: I'd like to ask the group of their experiences in improving the performance of applications that use mmap calls against files on Spectrum Scale. Besides using an NFS export from CES instead of a native GPFS mount, or precaching the dataset into the pagepool, what other approaches are there to offset the performance hit of the 4K IO size? Kind regards Ray Coetzee _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From coetzee.ray at gmail.com Fri Jan 12 20:51:13 2018 From: coetzee.ray at gmail.com (Ray Coetzee) Date: Fri, 12 Jan 2018 20:51:13 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. (Without the meltdown patch) Hey Bryan, I remember that quote from Yuri, that's why I hoped some "magic" may have been done since then. Other attempts to improve performance I've tried include: - Using LROC to have a larger chance of a cache hit (Unfortunately the entire dataset is multiple TB) - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for this purpose (Job runs halved in duration but nowhere near what NFS can give) - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and some others with little improvement. For those interested, as a performance comparison. The same job when run on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe scratch filesystem and over 60min on spindle based filesystem. Kind regards Ray Coetzee Email: coetzee.ray at gmail.com On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister wrote: > You could put all of your data onto SSDs in a RAID1 configuration so that > you don?t have insane read-modify-write penalties on writes (RAID1) and > avoid horrible seek thrashing that spinning rust requires (SSD random > access medium) for your 4K I/O operations. > > > > One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not > to touch it?. He gave many reasons why mmap operations on a distributed > file system is incredibly hard and not recommended. > > -Bryan > > > > *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org] *On Behalf Of *Sven Oehme > *Sent:* Friday, January 12, 2018 8:45 AM > *To:* coetzee.ray at gmail.com; gpfsug main discussion list < > gpfsug-discuss at spectrumscale.org> > *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale > > > > *Note: External Email* > ------------------------------ > > what version of Scale are you using right now ? > > > > On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee wrote: > > I'd like to ask the group of their experiences in improving the > performance of applications that use mmap calls against files on Spectrum > Scale. > > > > Besides using an NFS export from CES instead of a native GPFS mount, or > precaching the dataset into the pagepool, what other approaches are there > to offset the performance hit of the 4K IO size? > > > > Kind regards > > Ray Coetzee > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at calicolabs.com Fri Jan 12 20:56:00 2018 From: alex at calicolabs.com (Alex Chekholko) Date: Fri, 12 Jan 2018 12:56:00 -0800 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: Hi Ray, Sounds like you are better off hosting that workflow on the other storage system. Another thing I have done in the past is work with the software maintainer to replace all mmap() with lseek(). YMMV. I have previously told users that mmap "doesn't work" on GPFS. As I understand it, programmers use mmap to try to improve performance but here it has the opposite effect. Regards, Alex On Fri, Jan 12, 2018 at 12:51 PM, Ray Coetzee wrote: > Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. > (Without the meltdown patch) > > Hey Bryan, I remember that quote from Yuri, that's why I hoped some > "magic" may have been done since then. > > Other attempts to improve performance I've tried include: > > - Using LROC to have a larger chance of a cache hit (Unfortunately the > entire dataset is multiple TB) > - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for > this purpose (Job runs halved in duration but nowhere near what NFS can > give) > - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and > some others with little improvement. > > For those interested, as a performance comparison. The same job when run > on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe > scratch filesystem and over 60min on spindle based filesystem. > > Kind regards > > Ray Coetzee > Email: coetzee.ray at gmail.com > > > On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister > wrote: > >> You could put all of your data onto SSDs in a RAID1 configuration so that >> you don?t have insane read-modify-write penalties on writes (RAID1) and >> avoid horrible seek thrashing that spinning rust requires (SSD random >> access medium) for your 4K I/O operations. >> >> >> >> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not >> to touch it?. He gave many reasons why mmap operations on a distributed >> file system is incredibly hard and not recommended. >> >> -Bryan >> >> >> >> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: >> gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >> *Sent:* Friday, January 12, 2018 8:45 AM >> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >> gpfsug-discuss at spectrumscale.org> >> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >> >> >> >> *Note: External Email* >> ------------------------------ >> >> what version of Scale are you using right now ? >> >> >> >> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >> wrote: >> >> I'd like to ask the group of their experiences in improving the >> performance of applications that use mmap calls against files on Spectrum >> Scale. >> >> >> >> Besides using an NFS export from CES instead of a native GPFS mount, or >> precaching the dataset into the pagepool, what other approaches are there >> to offset the performance hit of the 4K IO size? >> >> >> >> Kind regards >> >> Ray Coetzee >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ------------------------------ >> >> Note: This email is for the confidential use of the named addressee(s) >> only and may contain proprietary, confidential or privileged information. >> If you are not the intended recipient, you are hereby notified that any >> review, dissemination or copying of this email is strictly prohibited, and >> to please notify the sender immediately and destroy this email and any >> attachments. Email transmission cannot be guaranteed to be secure or >> error-free. The Company, therefore, does not make any guarantees as to the >> completeness or accuracy of this email or any attachments. This email is >> for informational purposes only and does not constitute a recommendation, >> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >> or perform any type of transaction of a financial product. >> > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oehmes at gmail.com Fri Jan 12 20:57:24 2018 From: oehmes at gmail.com (Sven Oehme) Date: Fri, 12 Jan 2018 20:57:24 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: is this primary read or write ? On Fri, Jan 12, 2018, 12:51 PM Ray Coetzee wrote: > Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. > (Without the meltdown patch) > > Hey Bryan, I remember that quote from Yuri, that's why I hoped some > "magic" may have been done since then. > > Other attempts to improve performance I've tried include: > > - Using LROC to have a larger chance of a cache hit (Unfortunately the > entire dataset is multiple TB) > - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for > this purpose (Job runs halved in duration but nowhere near what NFS can > give) > - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and > some others with little improvement. > > For those interested, as a performance comparison. The same job when run > on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe > scratch filesystem and over 60min on spindle based filesystem. > > Kind regards > > Ray Coetzee > Email: coetzee.ray at gmail.com > > > On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister > wrote: > >> You could put all of your data onto SSDs in a RAID1 configuration so that >> you don?t have insane read-modify-write penalties on writes (RAID1) and >> avoid horrible seek thrashing that spinning rust requires (SSD random >> access medium) for your 4K I/O operations. >> >> >> >> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best not >> to touch it?. He gave many reasons why mmap operations on a distributed >> file system is incredibly hard and not recommended. >> >> -Bryan >> >> >> >> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: >> gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >> *Sent:* Friday, January 12, 2018 8:45 AM >> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >> gpfsug-discuss at spectrumscale.org> >> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >> >> >> >> *Note: External Email* >> ------------------------------ >> >> what version of Scale are you using right now ? >> >> >> >> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >> wrote: >> >> I'd like to ask the group of their experiences in improving the >> performance of applications that use mmap calls against files on Spectrum >> Scale. >> >> >> >> Besides using an NFS export from CES instead of a native GPFS mount, or >> precaching the dataset into the pagepool, what other approaches are there >> to offset the performance hit of the 4K IO size? >> >> >> >> Kind regards >> >> Ray Coetzee >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ------------------------------ >> >> Note: This email is for the confidential use of the named addressee(s) >> only and may contain proprietary, confidential or privileged information. >> If you are not the intended recipient, you are hereby notified that any >> review, dissemination or copying of this email is strictly prohibited, and >> to please notify the sender immediately and destroy this email and any >> attachments. Email transmission cannot be guaranteed to be secure or >> error-free. The Company, therefore, does not make any guarantees as to the >> completeness or accuracy of this email or any attachments. This email is >> for informational purposes only and does not constitute a recommendation, >> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >> or perform any type of transaction of a financial product. >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jan 12 22:58:39 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 12 Jan 2018 19:58:39 -0300 Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) In-Reply-To: References: <22c4a22d-a72e-4868-b2da-12ee24cda3fd@nasa.gov> Message-ID: Having multiple blocksizes in the same file system would unnecessarily complicate things. Consider migrating a file from one pool to another with different blocksizes... How to represent the indirect blocks (lists of blocks allocated to the file)? Especially consider that today, migration can proceed one block at a time, during migration a file is "mis-placed" -- has blocks spread over more than one pool.... The new feature that supports more than 32 sub-blocks per block - is a step in another direction but maybe addresses some of your concerns.... We do support different blocksizes for meta-data -- but meta-data is distinct from data and never migrates out of system pool. --marc K. From: Aaron Knister To: Date: 01/08/2018 09:25 PM Subject: Re: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) Sent by: gpfsug-discuss-bounces at spectrumscale.org Thanks, Bryan! That's a great use case I hadn't thought of. GPFS can already support a different block size for the system pool so in my very simplistic view of the world it's already possible (unless there's some implementation detail about the system pool that lends itself to a different block size from all other pools that doesn't apply to other non-system pools differing from each other). -Aaron On 1/8/18 6:48 PM, Bryan Banister wrote: > Hey Aaron... I have been talking about the same idea here and would say it would be a massive feature and management improvement. > > I would like to have many GPFS storage pools in my file system, each with tuned blocksize and subblock sizes to suite the application, using independent filesets and the data placement policy to store the data in the right GPFS storage pool. Migrating the data with the policy engine between these pools as you described would be a lot faster and a lot safer than trying to migrate files individually (like with rsync). > > NSDs can only belong to one storage pool, so I don't see why the block allocation map would be difficult to manage in this case. > > Cheers, > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Aaron Knister > Sent: Monday, January 08, 2018 4:57 PM > To: gpfsug main discussion list > Subject: [gpfsug-discuss] Multiple Block Sizes in a Filesystem (Was: Online data migration tool) > > Note: External Email > ------------------------------------------------- > > I was thinking some more about the >32 subblock feature in scale 5.0. As > mentioned by IBM the biggest advantage of that feature is on filesystems > with large blocks (e.g. multiple MB). The majority of our filesystems > have a block size of 1MB which got me thinking... wouldn't it be nice if > they had a larger block size (there seem to be compelling performance > reasons for large file I/O to do this). > > I'm wondering, what's the feasibility is of supporting filesystem pools > of varying block sizes within a single filesystem? I thought allocation > maps exist on a per-pool basis which gives me some hope it's not too hard. > > If one could do this then, yes, you'd still need new hardware to migrate > to a larger block size (and >32 subblocks), but it could be done as part > of a refresh cycle *and* (and this is the part most important to me) it > could be driven entirely by the policy engine which means storage admins > are largely hands off and the migration is by and large transparent to > the end user. > > This doesn't really address the need for a tool to address a filesystem > migration to 4k metadata blocks (although I wonder if something could be > done to create a system_4k pool that contains 4k-aligned metadata NSDs > where key data structures get re-written during a restripe in a > 4k-aligned manner, but that's really grasping at straws for me). > > -Aaorn > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=zfQZ_ymVgGc2EseA0szLiRxH-FgYnw7qMdx2qKo3Zes&s=2KsLTkZ-3MRyIMQhp8WwTn524NpfiKv8gwTy4P36xX4&e= > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=zfQZ_ymVgGc2EseA0szLiRxH-FgYnw7qMdx2qKo3Zes&s=2KsLTkZ-3MRyIMQhp8WwTn524NpfiKv8gwTy4P36xX4&e= > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=zfQZ_ymVgGc2EseA0szLiRxH-FgYnw7qMdx2qKo3Zes&s=2KsLTkZ-3MRyIMQhp8WwTn524NpfiKv8gwTy4P36xX4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Fri Jan 12 23:14:52 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Fri, 12 Jan 2018 18:14:52 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm Message-ID: Apologies if this has been covered elsewhere (I couldn't find it if it has). I'm curious how GPFS decides where to allocate new blocks. We've got a filesystem that we added some NSDs to a while back and it hurt there for a little while because it appeared as though GPFS was choosing to allocate new blocks much more frequently on the ~100% free LUNs than the existing LUNs (I can't recall how free they were at the time). Looking at it now, though, it seems GPFS is doing the opposite. There's now a ~10% difference between the LUNs added and the existing LUNs (20% free vs 30% free) and GPFS is choosing to allocate new writes at a ratio of about 3:1 on the disks with *fewer* free blocks than on the disks with more free blocks. That's completely inconsistent with what we saw when we initially added the disks which makes me wonder how GPFS is choosing to allocate new blocks (other than the obvious bits about failure group, and replication factor). Could someone explain (or point me at a whitepaper) what factors GPFS uses when allocating blocks, particularly as it pertains to choosing one NSD over another within the same failure group. Thanks! -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From janfrode at tanso.net Sat Jan 13 09:24:54 2018 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Sat, 13 Jan 2018 09:24:54 +0000 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: Message-ID: Don?t have documentation/whitepaper, but as I recall, it will first allocate round-robin over failureGroup, then round-robin over nsdServers, and then round-robin over volumes. So if these new NSDs are defined as different failureGroup from the old disks, that might explain it.. -jf l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister : > Apologies if this has been covered elsewhere (I couldn't find it if it > has). I'm curious how GPFS decides where to allocate new blocks. > > We've got a filesystem that we added some NSDs to a while back and it > hurt there for a little while because it appeared as though GPFS was > choosing to allocate new blocks much more frequently on the ~100% free > LUNs than the existing LUNs (I can't recall how free they were at the > time). Looking at it now, though, it seems GPFS is doing the opposite. > There's now a ~10% difference between the LUNs added and the existing > LUNs (20% free vs 30% free) and GPFS is choosing to allocate new writes > at a ratio of about 3:1 on the disks with *fewer* free blocks than on > the disks with more free blocks. That's completely inconsistent with > what we saw when we initially added the disks which makes me wonder how > GPFS is choosing to allocate new blocks (other than the obvious bits > about failure group, and replication factor). Could someone explain (or > point me at a whitepaper) what factors GPFS uses when allocating blocks, > particularly as it pertains to choosing one NSD over another within the > same failure group. > > Thanks! > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Sat Jan 13 13:18:36 2018 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Sat, 13 Jan 2018 13:18:36 +0000 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sat Jan 13 16:18:25 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 13 Jan 2018 11:18:25 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: Message-ID: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> Thanks Everyone! I whipped up a script to dump the block layout of a file and then join that with mmdf information. As part of my exploration I wrote one 2GB file to each of this particular filesystem's 4 data pools last night (using "touch $file; mmchattr $file -P $pool; dd of=$file") and have attached a dump of the layout/nsd information for each file/pool. The fields for the output are: diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, freePct, freeKbFragments, freeKbFragmentsPct Here's the highlight from pool1: 36 264 d13_06_006 23437934592 1213 4548935680 (19%) 83304320 (0%) 59 74 d10_41_025 23437934592 1011 6993759232 (30%) 58642816 (0%) For this file (And anecdotally what I've seen looking at NSD I/O data for other files written to this pool) the pattern of more blocks being allocated to the NSDs that are ~20% free vs the NSDs that are 30% free seems to be fairly consistent. Looking at a snippet of pool2: 101 238 d15_15_011 23437934592 1415 2008394752 (9%) 181699328 (1%) 102 235 d15_16_012 23437934592 1415 2009153536 (9%) 182165312 (1%) 116 248 d11_42_026 23437934592 1011 4146111488 (18%) 134941504 (1%) 117 249 d13_42_026 23437934592 1213 4147710976 (18%) 135203776 (1%) there are slightly more blocks allocated in general on the NSDs with twice the amount of free space, but it doesn't seem to be a significant amount relative to the delta in free space. The pattern from pool1 certainly doesn't hold true here. Pool4 isn't very interesting because all of the NSDs are well balanced in terms of free space (all 16% free). Pool3, however, *is* particularly interesting. Here's a snippet: 106 222 d15_24_016 23437934592 1415 2957561856 (13%) 37436768 (0%) 107 222 d15_25_017 23437934592 1415 2957537280 (13%) 37353984 (0%) 108 222 d15_26_018 23437934592 1415 2957539328 (13%) 37335872 (0%) 125 222 d11_44_028 23437934592 1011 13297235968 (57%) 20505568 (0%) 126 222 d12_44_028 23437934592 1213 13296712704 (57%) 20632768 (0%) 127 222 d12_45_029 23437934592 1213 13297404928 (57%) 20557408 (0%) GPFS consistently allocated the same number of blocks to disks with ~4x the free space than it did the other disks in the pool. Suffice it to say-- I'm *very* confused :) -Aaron On 1/13/18 8:18 AM, Daniel Kidger wrote: > Aaron, > ? > Also are your new NSDs the same size as your existing ones? > i.e. the NSDs that are at a?higher %age full might have more free blocks > than the other NSDs? > Daniel > > ? > IBM Storage Professional Badge > > ? > ? > *Dr Daniel Kidger* > IBM?Technical Sales Specialist > Software Defined Solution Sales > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > ? > ? > ? > > ----- Original message ----- > From: Jan-Frode Myklebust > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: Re: [gpfsug-discuss] pool block allocation algorithm > Date: Sat, Jan 13, 2018 9:25 AM > ? > Don?t have documentation/whitepaper, but as I recall, it will first > allocate round-robin over failureGroup, then round-robin over > nsdServers, and then round-robin over volumes. So if these new NSDs > are defined as different failureGroup from the old disks, that might > explain it.. > > > -jf > l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister > >: > > Apologies if this has been covered elsewhere (I couldn't find it > if it > has). I'm curious how GPFS decides where to allocate new blocks. > > We've got a filesystem that we added some NSDs to a while back > and it > hurt there for a little while because it appeared as though GPFS was > choosing to allocate new blocks much more frequently on the > ~100% free > LUNs than the existing LUNs (I can't recall how free they were > at the > time). Looking at it now, though, it seems GPFS is doing the > opposite. > There's now a ~10% difference between the LUNs added and the > existing > LUNs (20% free vs 30% free) and GPFS is choosing to allocate new > writes > at a ratio of about 3:1 on the disks with *fewer* free blocks > than on > the disks with more free blocks. That's completely inconsistent with > what we saw when we initially added the disks which makes me > wonder how > GPFS is choosing to allocate new blocks (other than the obvious bits > about failure group, and replication factor). Could someone > explain (or > point me at a whitepaper) what factors GPFS uses when allocating > blocks, > particularly as it pertains to choosing one NSD over another > within the > same failure group. > > Thanks! > > -Aaron > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= > > ? > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 -------------- next part -------------- 1 221 d12_01_001 23437934592 1213 4550865920 (19%) 83734976 (0%) 2 266 d12_02_002 23437934592 1213 4550282240 (19%) 83712000 (0%) 3 265 d12_03_003 23437934592 1213 4550419456 (19%) 83986432 (0%) 4 265 d12_04_004 23437934592 1213 4550207488 (19%) 83382624 (0%) 5 266 d12_05_005 23437934592 1213 4551135232 (19%) 83820832 (0%) 6 264 d12_06_006 23437934592 1213 4549501952 (19%) 83537984 (0%) 31 256 d13_01_001 23437934592 1213 4550247424 (19%) 83557184 (0%) 32 266 d13_02_002 23437934592 1213 4548240384 (19%) 83289888 (0%) 33 264 d13_03_003 23437934592 1213 4548679680 (19%) 83851360 (0%) 34 266 d13_04_004 23437934592 1213 4551052288 (19%) 83524320 (0%) 35 265 d13_05_005 23437934592 1213 4550079488 (19%) 83578624 (0%) 36 264 d13_06_006 23437934592 1213 4548935680 (19%) 83304320 (0%) 59 74 d10_41_025 23437934592 1011 6993759232 (30%) 58642816 (0%) 61 216 d14_01_001 23437934592 1415 4575688704 (20%) 83600032 (0%) 62 266 d14_02_002 23437934592 1415 4574487552 (20%) 83376960 (0%) 63 265 d14_03_003 23437934592 1415 4574819328 (20%) 83378240 (0%) 64 266 d14_04_004 23437934592 1415 4575392768 (20%) 83162080 (0%) 65 264 d14_05_005 23437934592 1415 4576001024 (20%) 83447968 (0%) 66 265 d14_06_006 23437934592 1415 4575699968 (20%) 83043040 (0%) 85 72 d11_41_025 23437934592 1011 6991225856 (30%) 58576768 (0%) 86 66 d12_41_025 23437934592 1213 6992339968 (30%) 58402688 (0%) 87 71 d12_42_026 23437934592 1213 6992146432 (30%) 58284032 (0%) 88 67 d13_41_025 23437934592 1213 6993167360 (30%) 58134624 (0%) 89 75 d14_41_025 23437934592 1415 6992531456 (30%) 58169600 (0%) 90 71 d14_42_026 23437934592 1415 6993435648 (30%) 58234720 (0%) 91 265 d15_01_001 23437934592 1415 4575676416 (20%) 83394144 (0%) 92 264 d15_02_002 23437934592 1415 4576221184 (20%) 82999168 (0%) 93 264 d15_03_003 23437934592 1415 4576231424 (20%) 83179680 (0%) 94 266 d15_04_004 23437934592 1415 4577104896 (20%) 83445088 (0%) 95 265 d15_05_005 23437934592 1415 4576627712 (20%) 82928064 (0%) 96 259 d15_06_006 23437934592 1415 4576740352 (20%) 83355776 (0%) 115 73 d15_41_025 23437934592 1415 6990783488 (30%) 58595296 (0%) 145 266 d10_01_001 23437934592 1011 4588489728 (20%) 83100448 (0%) 146 266 d10_02_002 23437934592 1011 4587276288 (20%) 83239488 (0%) 147 265 d10_03_003 23437934592 1011 4586872832 (20%) 83024000 (0%) 148 264 d10_04_004 23437934592 1011 4585822208 (20%) 82897920 (0%) 149 264 d10_05_005 23437934592 1011 4586415104 (20%) 83277056 (0%) 150 266 d10_06_006 23437934592 1011 4587424768 (20%) 83076544 (0%) 151 265 d11_01_001 23437934592 1011 4586209280 (20%) 83178368 (0%) 152 265 d11_02_002 23437934592 1011 4587372544 (20%) 83091232 (0%) 153 264 d11_03_003 23437934592 1011 4587196416 (20%) 83147040 (0%) 154 264 d11_04_004 23437934592 1011 4586633216 (20%) 83049024 (0%) 155 265 d11_05_005 23437934592 1011 4586576896 (20%) 83188768 (0%) 156 264 d11_06_006 23437934592 1011 4587261952 (20%) 83393952 (0%) -------------- next part -------------- 7 238 d12_11_007 23437934592 1213 2005351424 (9%) 184092352 (1%) 8 237 d12_12_008 23437934592 1213 2005532672 (9%) 183893472 (1%) 9 243 d12_13_009 23437934592 1213 2004583424 (9%) 183000704 (1%) 10 239 d12_14_010 23437934592 1213 2004461568 (9%) 182958048 (1%) 11 228 d12_15_011 23437934592 1213 2004750336 (9%) 183160064 (1%) 12 240 d12_16_012 23437934592 1213 2008772608 (9%) 183263936 (1%) 37 228 d13_11_007 23437934592 1213 2006091776 (9%) 183041024 (1%) 38 230 d13_12_008 23437934592 1213 2003761152 (9%) 182625024 (1%) 39 229 d13_13_009 23437934592 1213 2004123648 (9%) 182907552 (1%) 40 208 d13_14_010 23437934592 1213 2006372352 (9%) 182797376 (1%) 41 233 d13_15_011 23437934592 1213 2003358720 (9%) 182322880 (1%) 42 232 d13_16_012 23437934592 1213 2004774912 (9%) 182729792 (1%) 67 234 d14_11_007 23437934592 1415 2006553600 (9%) 182736800 (1%) 68 249 d14_12_008 23437934592 1415 2010278912 (9%) 182304832 (1%) 69 226 d14_13_009 23437934592 1415 2008749056 (9%) 182854464 (1%) 70 237 d14_14_010 23437934592 1415 2009348096 (9%) 182257024 (1%) 71 249 d14_15_011 23437934592 1415 2008017920 (9%) 182217152 (1%) 72 233 d14_16_012 23437934592 1415 2008158208 (9%) 182239680 (1%) 97 237 d15_11_007 23437934592 1415 2007257088 (9%) 181898304 (1%) 98 233 d15_12_008 23437934592 1415 2008309760 (9%) 182000928 (1%) 99 229 d15_13_009 23437934592 1415 2008755200 (9%) 182380544 (1%) 100 224 d15_14_010 23437934592 1415 2008460288 (9%) 181691616 (1%) 101 238 d15_15_011 23437934592 1415 2008394752 (9%) 181699328 (1%) 102 235 d15_16_012 23437934592 1415 2009153536 (9%) 182165312 (1%) 116 248 d11_42_026 23437934592 1011 4146111488 (18%) 134941504 (1%) 117 249 d13_42_026 23437934592 1213 4147710976 (18%) 135203776 (1%) 118 248 d15_42_026 23437934592 1415 4147094528 (18%) 134748320 (1%) 119 249 d10_43_027 23437934592 1011 4148652032 (18%) 135124288 (1%) 120 247 d11_43_027 23437934592 1011 4147335168 (18%) 134808256 (1%) 121 250 d12_43_027 23437934592 1213 4147177472 (18%) 134812096 (1%) 122 248 d13_43_027 23437934592 1213 4147270656 (18%) 134516192 (1%) 123 248 d14_43_027 23437934592 1415 4148153344 (18%) 134928896 (1%) 124 248 d15_43_027 23437934592 1415 4149500928 (18%) 134818880 (1%) 157 247 d10_11_007 23437934592 1011 2008516608 (9%) 182482592 (1%) 159 248 d10_13_009 23437934592 1011 2009745408 (9%) 182393184 (1%) 161 233 d10_15_011 23437934592 1011 2007539712 (9%) 182169856 (1%) 163 234 d11_11_007 23437934592 1011 2009283584 (9%) 182173824 (1%) 164 244 d11_12_008 23437934592 1011 2009091072 (9%) 182099200 (1%) 165 248 d11_13_009 23437934592 1011 2008827904 (9%) 182029632 (1%) 166 231 d11_14_010 23437934592 1011 2010264576 (9%) 181675424 (1%) 167 235 d11_15_011 23437934592 1011 2010696704 (9%) 181854304 (1%) 168 236 d11_16_012 23437934592 1011 2008183808 (9%) 182236800 (1%) -------------- next part -------------- 13 222 d12_21_013 23437934592 1213 2957768704 (13%) 37317824 (0%) 14 223 d12_22_014 23437934592 1213 2957680640 (13%) 37608576 (0%) 15 223 d12_23_015 23437934592 1213 2958575616 (13%) 37380672 (0%) 16 223 d12_24_016 23437934592 1213 2957406208 (13%) 37403648 (0%) 17 223 d12_25_017 23437934592 1213 2957883392 (13%) 37558016 (0%) 18 222 d12_26_018 23437934592 1213 2957815808 (13%) 37556416 (0%) 43 222 d13_21_013 23437934592 1213 2957149184 (13%) 37745280 (0%) 44 222 d13_22_014 23437934592 1213 2957812736 (13%) 37520736 (0%) 45 222 d13_23_015 23437934592 1213 2957990912 (13%) 37341952 (0%) 46 222 d13_24_016 23437934592 1213 2958204928 (13%) 37477536 (0%) 47 222 d13_25_017 23437934592 1213 2958315520 (13%) 37507072 (0%) 48 222 d13_26_018 23437934592 1213 2958086144 (13%) 37407648 (0%) 73 222 d14_21_013 23437934592 1415 2957812736 (13%) 37631488 (0%) 74 223 d14_22_014 23437934592 1415 2959673344 (13%) 37427680 (0%) 75 223 d14_23_015 23437934592 1415 2957627392 (13%) 37400992 (0%) 76 223 d14_24_016 23437934592 1415 2957408256 (13%) 37235776 (0%) 77 222 d14_25_017 23437934592 1415 2957407232 (13%) 37499200 (0%) 78 222 d14_26_018 23437934592 1415 2956661760 (13%) 37633696 (0%) 103 222 d15_21_013 23437934592 1415 2957427712 (13%) 37629664 (0%) 104 222 d15_22_014 23437934592 1415 2957506560 (13%) 37571968 (0%) 105 222 d15_23_015 23437934592 1415 2958114816 (13%) 37422400 (0%) 106 222 d15_24_016 23437934592 1415 2957561856 (13%) 37436768 (0%) 107 222 d15_25_017 23437934592 1415 2957537280 (13%) 37353984 (0%) 108 222 d15_26_018 23437934592 1415 2957539328 (13%) 37335872 (0%) 125 222 d11_44_028 23437934592 1011 13297235968 (57%) 20505568 (0%) 126 222 d12_44_028 23437934592 1213 13296712704 (57%) 20632768 (0%) 127 222 d12_45_029 23437934592 1213 13297404928 (57%) 20557408 (0%) 128 222 d13_44_028 23437934592 1213 13297271808 (57%) 20502304 (0%) 129 222 d14_44_028 23437934592 1415 13294722048 (57%) 20569824 (0%) 130 222 d14_45_029 23437934592 1415 13296071680 (57%) 20667104 (0%) 131 222 d15_44_028 23437934592 1415 13297007616 (57%) 20429536 (0%) 132 223 d10_44_028 23437934592 1011 13309036544 (57%) 20299552 (0%) 133 223 d10_45_029 23437934592 1011 13309213696 (57%) 20464576 (0%) 169 223 d10_21_013 23437934592 1011 2797943808 (12%) 36881600 (0%) 170 222 d10_22_014 23437934592 1011 2798030848 (12%) 37152256 (0%) 171 222 d10_23_015 23437934592 1011 2798152704 (12%) 36959840 (0%) 172 222 d10_24_016 23437934592 1011 2798002176 (12%) 36954976 (0%) 173 222 d10_25_017 23437934592 1011 2798524416 (12%) 36737120 (0%) 174 222 d10_26_018 23437934592 1011 2797768704 (12%) 36888096 (0%) 175 222 d11_21_013 23437934592 1011 2958944256 (13%) 37504288 (0%) 176 222 d11_22_014 23437934592 1011 2957225984 (13%) 37575648 (0%) 177 222 d11_23_015 23437934592 1011 2958100480 (13%) 37546336 (0%) 178 222 d11_24_016 23437934592 1011 2958453760 (13%) 37578784 (0%) 179 222 d11_25_017 23437934592 1011 2958053376 (13%) 37331808 (0%) 180 222 d11_26_018 23437934592 1011 2958290944 (13%) 37445824 (0%) -------------- next part -------------- 19 223 d12_31_019 23437934592 1213 3803873280 (16%) 117281408 (1%) 20 222 d12_32_020 23437934592 1213 3779237888 (16%) 116652576 (0%) 21 222 d12_33_021 23437934592 1213 3779206144 (16%) 116820288 (0%) 22 222 d12_34_022 23437934592 1213 3776828416 (16%) 116108032 (0%) 23 222 d12_35_023 23437934592 1213 3776950272 (16%) 116238752 (0%) 24 222 d12_36_024 23437934592 1213 3776524288 (16%) 116599744 (0%) 49 222 d13_31_019 23437934592 1213 3778293760 (16%) 116365600 (0%) 50 222 d13_32_020 23437934592 1213 3778381824 (16%) 115960448 (0%) 51 222 d13_33_021 23437934592 1213 3777104896 (16%) 116225984 (0%) 52 222 d13_34_022 23437934592 1213 3775632384 (16%) 116496576 (0%) 53 222 d13_35_023 23437934592 1213 3776899072 (16%) 116397120 (0%) 54 222 d13_36_024 23437934592 1213 3776140288 (16%) 116400000 (0%) 79 222 d14_31_019 23437934592 1415 3844638720 (16%) 113531904 (0%) 80 222 d14_32_020 23437934592 1415 3817139200 (16%) 113600928 (0%) 81 222 d14_33_021 23437934592 1415 3815447552 (16%) 113358272 (0%) 82 222 d14_34_022 23437934592 1415 3814561792 (16%) 113348160 (0%) 83 222 d14_35_023 23437934592 1415 3815387136 (16%) 113582432 (0%) 84 222 d14_36_024 23437934592 1415 3815303168 (16%) 112914816 (0%) 109 222 d15_31_019 23437934592 1415 3814614016 (16%) 113298240 (0%) 110 222 d15_32_020 23437934592 1415 3814936576 (16%) 113053664 (0%) 111 222 d15_33_021 23437934592 1415 3815122944 (16%) 113438656 (0%) 112 222 d15_34_022 23437934592 1415 3813474304 (16%) 113481216 (0%) 113 222 d15_35_023 23437934592 1415 3812912128 (16%) 113618624 (0%) 114 222 d15_36_024 23437934592 1415 3813067776 (16%) 113282848 (0%) 134 223 d11_45_029 23437934592 1011 3683404800 (16%) 100860000 (0%) 135 222 d13_45_029 23437934592 1213 3682211840 (16%) 101101824 (0%) 136 223 d15_45_029 23437934592 1415 3682046976 (16%) 101375328 (0%) 137 222 d10_46_030 23437934592 1011 3684398080 (16%) 100946912 (0%) 138 222 d11_46_030 23437934592 1011 3683755008 (16%) 101295200 (0%) 139 223 d12_46_030 23437934592 1213 3682649088 (16%) 100540832 (0%) 140 223 d13_46_030 23437934592 1213 3684602880 (16%) 100489376 (0%) 141 223 d14_46_030 23437934592 1415 3681666048 (16%) 100854112 (0%) 142 223 d15_46_030 23437934592 1415 3685487616 (16%) 100715744 (0%) 181 222 d10_31_019 23437934592 1011 3816681472 (16%) 113389344 (0%) 182 222 d10_32_020 23437934592 1011 3816535040 (16%) 113652672 (0%) 183 222 d10_33_021 23437934592 1011 3817772032 (16%) 113124224 (0%) 184 222 d10_34_022 23437934592 1011 3817069568 (16%) 113225504 (0%) 185 222 d10_35_023 23437934592 1011 3816517632 (16%) 113163488 (0%) 186 222 d10_36_024 23437934592 1011 3815605248 (16%) 113475648 (0%) 187 222 d11_31_019 23437934592 1011 3815664640 (16%) 113100992 (0%) 188 222 d11_32_020 23437934592 1011 3815985152 (16%) 113325504 (0%) 189 222 d11_33_021 23437934592 1011 3815148544 (16%) 113328480 (0%) 190 223 d11_34_022 23437934592 1011 3814780928 (16%) 113457440 (0%) 191 223 d11_35_023 23437934592 1011 3815219200 (16%) 113065504 (0%) 192 223 d11_36_024 23437934592 1011 3815383040 (16%) 113170592 (0%) From aaron.s.knister at nasa.gov Sat Jan 13 16:56:52 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 13 Jan 2018 11:56:52 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> Message-ID: Sorry, I didn't explicitly say it, but the output I sent should answer both Daniel's and Jan-Frode's questions. In short, though, the new NSDs were added to existing failure groups and they should all be (except in one or two cases where we re-formatted the LUN and the size changed slightly) the same size. -Aaron On 1/13/18 11:18 AM, Aaron Knister wrote: > Thanks Everyone! I whipped up a script to dump the block layout of a > file and then join that with mmdf information. As part of my exploration > I wrote one 2GB file to each of this particular filesystem's 4 data > pools last night (using "touch $file; mmchattr $file -P $pool; dd > of=$file") and have attached a dump of the layout/nsd information for > each file/pool. The fields for the output are: > > diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, > freePct, freeKbFragments, freeKbFragmentsPct > > > Here's the highlight from pool1: > > 36 264 d13_06_006 23437934592 1213 4548935680 (19%) > 83304320 (0%) > 59 74 d10_41_025 23437934592 1011 6993759232 (30%) > 58642816 (0%) > > For this file (And anecdotally what I've seen looking at NSD I/O data > for other files written to this pool) the pattern of more blocks being > allocated to the NSDs that are ~20% free vs the NSDs that are 30% free > seems to be fairly consistent. > > > Looking at a snippet of pool2: > 101 238 d15_15_011 23437934592 1415 2008394752 (9%) > 181699328 (1%) > 102 235 d15_16_012 23437934592 1415 2009153536 (9%) > 182165312 (1%) > 116 248 d11_42_026 23437934592 1011 4146111488 (18%) > 134941504 (1%) > 117 249 d13_42_026 23437934592 1213 4147710976 (18%) > 135203776 (1%) > > there are slightly more blocks allocated in general on the NSDs with > twice the amount of free space, but it doesn't seem to be a significant > amount relative to the delta in free space. The pattern from pool1 > certainly doesn't hold true here. > > Pool4 isn't very interesting because all of the NSDs are well balanced > in terms of free space (all 16% free). > > Pool3, however, *is* particularly interesting. Here's a snippet: > > 106 222 d15_24_016 23437934592 1415 2957561856 (13%) > 37436768 (0%) > 107 222 d15_25_017 23437934592 1415 2957537280 (13%) > 37353984 (0%) > 108 222 d15_26_018 23437934592 1415 2957539328 (13%) > 37335872 (0%) > 125 222 d11_44_028 23437934592 1011 13297235968 (57%) > 20505568 (0%) > 126 222 d12_44_028 23437934592 1213 13296712704 (57%) > 20632768 (0%) > 127 222 d12_45_029 23437934592 1213 13297404928 (57%) > 20557408 (0%) > > GPFS consistently allocated the same number of blocks to disks with ~4x > the free space than it did the other disks in the pool. > > Suffice it to say-- I'm *very* confused :) > > -Aaron > > On 1/13/18 8:18 AM, Daniel Kidger wrote: >> Aaron, >> ? >> Also are your new NSDs the same size as your existing ones? >> i.e. the NSDs that are at a?higher %age full might have more free blocks >> than the other NSDs? >> Daniel >> >> ? >> IBM Storage Professional Badge >> >> ? >> ? >> *Dr Daniel Kidger* >> IBM?Technical Sales Specialist >> Software Defined Solution Sales >> >> +44-(0)7818 522 266 >> daniel.kidger at uk.ibm.com >> >> ? >> ? >> ? >> >> ----- Original message ----- >> From: Jan-Frode Myklebust >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug main discussion list >> Cc: >> Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> Date: Sat, Jan 13, 2018 9:25 AM >> ? >> Don?t have documentation/whitepaper, but as I recall, it will first >> allocate round-robin over failureGroup, then round-robin over >> nsdServers, and then round-robin over volumes. So if these new NSDs >> are defined as different failureGroup from the old disks, that might >> explain it.. >> >> >> -jf >> l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >> >: >> >> Apologies if this has been covered elsewhere (I couldn't find it >> if it >> has). I'm curious how GPFS decides where to allocate new blocks. >> >> We've got a filesystem that we added some NSDs to a while back >> and it >> hurt there for a little while because it appeared as though GPFS was >> choosing to allocate new blocks much more frequently on the >> ~100% free >> LUNs than the existing LUNs (I can't recall how free they were >> at the >> time). Looking at it now, though, it seems GPFS is doing the >> opposite. >> There's now a ~10% difference between the LUNs added and the >> existing >> LUNs (20% free vs 30% free) and GPFS is choosing to allocate new >> writes >> at a ratio of about 3:1 on the disks with *fewer* free blocks >> than on >> the disks with more free blocks. That's completely inconsistent with >> what we saw when we initially added the disks which makes me >> wonder how >> GPFS is choosing to allocate new blocks (other than the obvious bits >> about failure group, and replication factor). Could someone >> explain (or >> point me at a whitepaper) what factors GPFS uses when allocating >> blocks, >> particularly as it pertains to choosing one NSD over another >> within the >> same failure group. >> >> Thanks! >> >> -Aaron >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= >> >> ? >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From peserocka at gmail.com Sat Jan 13 17:00:49 2018 From: peserocka at gmail.com (Peter Serocka) Date: Sat, 13 Jan 2018 18:00:49 +0100 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> Message-ID: <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Within reasonable capacity limits it would also expect to direct incoming data to disks that are best ?available? from a current performance perspective ? like doing least IOPS, having lowest latency and shortest filled queue length. You new NSDs, filled only with recent data, might quickly have become the most busy units before reaching capacity balance, simply because recent data tends to be more active than older stuff. Makes sense? ? Peter > On 2018 Jan 13 Sat, at 17:18, Aaron Knister wrote: > > Thanks Everyone! I whipped up a script to dump the block layout of a > file and then join that with mmdf information. As part of my exploration > I wrote one 2GB file to each of this particular filesystem's 4 data > pools last night (using "touch $file; mmchattr $file -P $pool; dd > of=$file") and have attached a dump of the layout/nsd information for > each file/pool. The fields for the output are: > > diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, > freePct, freeKbFragments, freeKbFragmentsPct > > > Here's the highlight from pool1: > > 36 264 d13_06_006 23437934592 1213 4548935680 (19%) > 83304320 (0%) > 59 74 d10_41_025 23437934592 1011 6993759232 (30%) > 58642816 (0%) > > For this file (And anecdotally what I've seen looking at NSD I/O data > for other files written to this pool) the pattern of more blocks being > allocated to the NSDs that are ~20% free vs the NSDs that are 30% free > seems to be fairly consistent. > > > Looking at a snippet of pool2: > 101 238 d15_15_011 23437934592 1415 2008394752 (9%) > 181699328 (1%) > 102 235 d15_16_012 23437934592 1415 2009153536 (9%) > 182165312 (1%) > 116 248 d11_42_026 23437934592 1011 4146111488 (18%) > 134941504 (1%) > 117 249 d13_42_026 23437934592 1213 4147710976 (18%) > 135203776 (1%) > > there are slightly more blocks allocated in general on the NSDs with > twice the amount of free space, but it doesn't seem to be a significant > amount relative to the delta in free space. The pattern from pool1 > certainly doesn't hold true here. > > Pool4 isn't very interesting because all of the NSDs are well balanced > in terms of free space (all 16% free). > > Pool3, however, *is* particularly interesting. Here's a snippet: > > 106 222 d15_24_016 23437934592 1415 2957561856 (13%) > 37436768 (0%) > 107 222 d15_25_017 23437934592 1415 2957537280 (13%) > 37353984 (0%) > 108 222 d15_26_018 23437934592 1415 2957539328 (13%) > 37335872 (0%) > 125 222 d11_44_028 23437934592 1011 13297235968 (57%) > 20505568 (0%) > 126 222 d12_44_028 23437934592 1213 13296712704 (57%) > 20632768 (0%) > 127 222 d12_45_029 23437934592 1213 13297404928 (57%) > 20557408 (0%) > > GPFS consistently allocated the same number of blocks to disks with ~4x > the free space than it did the other disks in the pool. > > Suffice it to say-- I'm *very* confused :) > > -Aaron > > On 1/13/18 8:18 AM, Daniel Kidger wrote: >> Aaron, >> >> Also are your new NSDs the same size as your existing ones? >> i.e. the NSDs that are at a higher %age full might have more free blocks >> than the other NSDs? >> Daniel >> >> >> IBM Storage Professional Badge >> >> >> >> *Dr Daniel Kidger* >> IBM Technical Sales Specialist >> Software Defined Solution Sales >> >> +44-(0)7818 522 266 >> daniel.kidger at uk.ibm.com >> >> >> >> >> >> ----- Original message ----- >> From: Jan-Frode Myklebust >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug main discussion list >> Cc: >> Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> Date: Sat, Jan 13, 2018 9:25 AM >> >> Don?t have documentation/whitepaper, but as I recall, it will first >> allocate round-robin over failureGroup, then round-robin over >> nsdServers, and then round-robin over volumes. So if these new NSDs >> are defined as different failureGroup from the old disks, that might >> explain it.. >> >> >> -jf >> l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >> >: >> >> Apologies if this has been covered elsewhere (I couldn't find it >> if it >> has). I'm curious how GPFS decides where to allocate new blocks. >> >> We've got a filesystem that we added some NSDs to a while back >> and it >> hurt there for a little while because it appeared as though GPFS was >> choosing to allocate new blocks much more frequently on the >> ~100% free >> LUNs than the existing LUNs (I can't recall how free they were >> at the >> time). Looking at it now, though, it seems GPFS is doing the >> opposite. >> There's now a ~10% difference between the LUNs added and the >> existing >> LUNs (20% free vs 30% free) and GPFS is choosing to allocate new >> writes >> at a ratio of about 3:1 on the disks with *fewer* free blocks >> than on >> the disks with more free blocks. That's completely inconsistent with >> what we saw when we initially added the disks which makes me >> wonder how >> GPFS is choosing to allocate new blocks (other than the obvious bits >> about failure group, and replication factor). Could someone >> explain (or >> point me at a whitepaper) what factors GPFS uses when allocating >> blocks, >> particularly as it pertains to choosing one NSD over another >> within the >> same failure group. >> >> Thanks! >> >> -Aaron >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= >> >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number >> 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 -------------- next part -------------- 1 221 d12_01_001 23437934592 1213 4550865920 (19%) 83734976 (0%) 2 266 d12_02_002 23437934592 1213 4550282240 (19%) 83712000 (0%) 3 265 d12_03_003 23437934592 1213 4550419456 (19%) 83986432 (0%) 4 265 d12_04_004 23437934592 1213 4550207488 (19%) 83382624 (0%) 5 266 d12_05_005 23437934592 1213 4551135232 (19%) 83820832 (0%) 6 264 d12_06_006 23437934592 1213 4549501952 (19%) 83537984 (0%) 31 256 d13_01_001 23437934592 1213 4550247424 (19%) 83557184 (0%) 32 266 d13_02_002 23437934592 1213 4548240384 (19%) 83289888 (0%) 33 264 d13_03_003 23437934592 1213 4548679680 (19%) 83851360 (0%) 34 266 d13_04_004 23437934592 1213 4551052288 (19%) 83524320 (0%) 35 265 d13_05_005 23437934592 1213 4550079488 (19%) 83578624 (0%) 36 264 d13_06_006 23437934592 1213 4548935680 (19%) 83304320 (0%) 59 74 d10_41_025 23437934592 1011 6993759232 (30%) 58642816 (0%) 61 216 d14_01_001 23437934592 1415 4575688704 (20%) 83600032 (0%) 62 266 d14_02_002 23437934592 1415 4574487552 (20%) 83376960 (0%) 63 265 d14_03_003 23437934592 1415 4574819328 (20%) 83378240 (0%) 64 266 d14_04_004 23437934592 1415 4575392768 (20%) 83162080 (0%) 65 264 d14_05_005 23437934592 1415 4576001024 (20%) 83447968 (0%) 66 265 d14_06_006 23437934592 1415 4575699968 (20%) 83043040 (0%) 85 72 d11_41_025 23437934592 1011 6991225856 (30%) 58576768 (0%) 86 66 d12_41_025 23437934592 1213 6992339968 (30%) 58402688 (0%) 87 71 d12_42_026 23437934592 1213 6992146432 (30%) 58284032 (0%) 88 67 d13_41_025 23437934592 1213 6993167360 (30%) 58134624 (0%) 89 75 d14_41_025 23437934592 1415 6992531456 (30%) 58169600 (0%) 90 71 d14_42_026 23437934592 1415 6993435648 (30%) 58234720 (0%) 91 265 d15_01_001 23437934592 1415 4575676416 (20%) 83394144 (0%) 92 264 d15_02_002 23437934592 1415 4576221184 (20%) 82999168 (0%) 93 264 d15_03_003 23437934592 1415 4576231424 (20%) 83179680 (0%) 94 266 d15_04_004 23437934592 1415 4577104896 (20%) 83445088 (0%) 95 265 d15_05_005 23437934592 1415 4576627712 (20%) 82928064 (0%) 96 259 d15_06_006 23437934592 1415 4576740352 (20%) 83355776 (0%) 115 73 d15_41_025 23437934592 1415 6990783488 (30%) 58595296 (0%) 145 266 d10_01_001 23437934592 1011 4588489728 (20%) 83100448 (0%) 146 266 d10_02_002 23437934592 1011 4587276288 (20%) 83239488 (0%) 147 265 d10_03_003 23437934592 1011 4586872832 (20%) 83024000 (0%) 148 264 d10_04_004 23437934592 1011 4585822208 (20%) 82897920 (0%) 149 264 d10_05_005 23437934592 1011 4586415104 (20%) 83277056 (0%) 150 266 d10_06_006 23437934592 1011 4587424768 (20%) 83076544 (0%) 151 265 d11_01_001 23437934592 1011 4586209280 (20%) 83178368 (0%) 152 265 d11_02_002 23437934592 1011 4587372544 (20%) 83091232 (0%) 153 264 d11_03_003 23437934592 1011 4587196416 (20%) 83147040 (0%) 154 264 d11_04_004 23437934592 1011 4586633216 (20%) 83049024 (0%) 155 265 d11_05_005 23437934592 1011 4586576896 (20%) 83188768 (0%) 156 264 d11_06_006 23437934592 1011 4587261952 (20%) 83393952 (0%) -------------- next part -------------- 7 238 d12_11_007 23437934592 1213 2005351424 (9%) 184092352 (1%) 8 237 d12_12_008 23437934592 1213 2005532672 (9%) 183893472 (1%) 9 243 d12_13_009 23437934592 1213 2004583424 (9%) 183000704 (1%) 10 239 d12_14_010 23437934592 1213 2004461568 (9%) 182958048 (1%) 11 228 d12_15_011 23437934592 1213 2004750336 (9%) 183160064 (1%) 12 240 d12_16_012 23437934592 1213 2008772608 (9%) 183263936 (1%) 37 228 d13_11_007 23437934592 1213 2006091776 (9%) 183041024 (1%) 38 230 d13_12_008 23437934592 1213 2003761152 (9%) 182625024 (1%) 39 229 d13_13_009 23437934592 1213 2004123648 (9%) 182907552 (1%) 40 208 d13_14_010 23437934592 1213 2006372352 (9%) 182797376 (1%) 41 233 d13_15_011 23437934592 1213 2003358720 (9%) 182322880 (1%) 42 232 d13_16_012 23437934592 1213 2004774912 (9%) 182729792 (1%) 67 234 d14_11_007 23437934592 1415 2006553600 (9%) 182736800 (1%) 68 249 d14_12_008 23437934592 1415 2010278912 (9%) 182304832 (1%) 69 226 d14_13_009 23437934592 1415 2008749056 (9%) 182854464 (1%) 70 237 d14_14_010 23437934592 1415 2009348096 (9%) 182257024 (1%) 71 249 d14_15_011 23437934592 1415 2008017920 (9%) 182217152 (1%) 72 233 d14_16_012 23437934592 1415 2008158208 (9%) 182239680 (1%) 97 237 d15_11_007 23437934592 1415 2007257088 (9%) 181898304 (1%) 98 233 d15_12_008 23437934592 1415 2008309760 (9%) 182000928 (1%) 99 229 d15_13_009 23437934592 1415 2008755200 (9%) 182380544 (1%) 100 224 d15_14_010 23437934592 1415 2008460288 (9%) 181691616 (1%) 101 238 d15_15_011 23437934592 1415 2008394752 (9%) 181699328 (1%) 102 235 d15_16_012 23437934592 1415 2009153536 (9%) 182165312 (1%) 116 248 d11_42_026 23437934592 1011 4146111488 (18%) 134941504 (1%) 117 249 d13_42_026 23437934592 1213 4147710976 (18%) 135203776 (1%) 118 248 d15_42_026 23437934592 1415 4147094528 (18%) 134748320 (1%) 119 249 d10_43_027 23437934592 1011 4148652032 (18%) 135124288 (1%) 120 247 d11_43_027 23437934592 1011 4147335168 (18%) 134808256 (1%) 121 250 d12_43_027 23437934592 1213 4147177472 (18%) 134812096 (1%) 122 248 d13_43_027 23437934592 1213 4147270656 (18%) 134516192 (1%) 123 248 d14_43_027 23437934592 1415 4148153344 (18%) 134928896 (1%) 124 248 d15_43_027 23437934592 1415 4149500928 (18%) 134818880 (1%) 157 247 d10_11_007 23437934592 1011 2008516608 (9%) 182482592 (1%) 159 248 d10_13_009 23437934592 1011 2009745408 (9%) 182393184 (1%) 161 233 d10_15_011 23437934592 1011 2007539712 (9%) 182169856 (1%) 163 234 d11_11_007 23437934592 1011 2009283584 (9%) 182173824 (1%) 164 244 d11_12_008 23437934592 1011 2009091072 (9%) 182099200 (1%) 165 248 d11_13_009 23437934592 1011 2008827904 (9%) 182029632 (1%) 166 231 d11_14_010 23437934592 1011 2010264576 (9%) 181675424 (1%) 167 235 d11_15_011 23437934592 1011 2010696704 (9%) 181854304 (1%) 168 236 d11_16_012 23437934592 1011 2008183808 (9%) 182236800 (1%) -------------- next part -------------- 13 222 d12_21_013 23437934592 1213 2957768704 (13%) 37317824 (0%) 14 223 d12_22_014 23437934592 1213 2957680640 (13%) 37608576 (0%) 15 223 d12_23_015 23437934592 1213 2958575616 (13%) 37380672 (0%) 16 223 d12_24_016 23437934592 1213 2957406208 (13%) 37403648 (0%) 17 223 d12_25_017 23437934592 1213 2957883392 (13%) 37558016 (0%) 18 222 d12_26_018 23437934592 1213 2957815808 (13%) 37556416 (0%) 43 222 d13_21_013 23437934592 1213 2957149184 (13%) 37745280 (0%) 44 222 d13_22_014 23437934592 1213 2957812736 (13%) 37520736 (0%) 45 222 d13_23_015 23437934592 1213 2957990912 (13%) 37341952 (0%) 46 222 d13_24_016 23437934592 1213 2958204928 (13%) 37477536 (0%) 47 222 d13_25_017 23437934592 1213 2958315520 (13%) 37507072 (0%) 48 222 d13_26_018 23437934592 1213 2958086144 (13%) 37407648 (0%) 73 222 d14_21_013 23437934592 1415 2957812736 (13%) 37631488 (0%) 74 223 d14_22_014 23437934592 1415 2959673344 (13%) 37427680 (0%) 75 223 d14_23_015 23437934592 1415 2957627392 (13%) 37400992 (0%) 76 223 d14_24_016 23437934592 1415 2957408256 (13%) 37235776 (0%) 77 222 d14_25_017 23437934592 1415 2957407232 (13%) 37499200 (0%) 78 222 d14_26_018 23437934592 1415 2956661760 (13%) 37633696 (0%) 103 222 d15_21_013 23437934592 1415 2957427712 (13%) 37629664 (0%) 104 222 d15_22_014 23437934592 1415 2957506560 (13%) 37571968 (0%) 105 222 d15_23_015 23437934592 1415 2958114816 (13%) 37422400 (0%) 106 222 d15_24_016 23437934592 1415 2957561856 (13%) 37436768 (0%) 107 222 d15_25_017 23437934592 1415 2957537280 (13%) 37353984 (0%) 108 222 d15_26_018 23437934592 1415 2957539328 (13%) 37335872 (0%) 125 222 d11_44_028 23437934592 1011 13297235968 (57%) 20505568 (0%) 126 222 d12_44_028 23437934592 1213 13296712704 (57%) 20632768 (0%) 127 222 d12_45_029 23437934592 1213 13297404928 (57%) 20557408 (0%) 128 222 d13_44_028 23437934592 1213 13297271808 (57%) 20502304 (0%) 129 222 d14_44_028 23437934592 1415 13294722048 (57%) 20569824 (0%) 130 222 d14_45_029 23437934592 1415 13296071680 (57%) 20667104 (0%) 131 222 d15_44_028 23437934592 1415 13297007616 (57%) 20429536 (0%) 132 223 d10_44_028 23437934592 1011 13309036544 (57%) 20299552 (0%) 133 223 d10_45_029 23437934592 1011 13309213696 (57%) 20464576 (0%) 169 223 d10_21_013 23437934592 1011 2797943808 (12%) 36881600 (0%) 170 222 d10_22_014 23437934592 1011 2798030848 (12%) 37152256 (0%) 171 222 d10_23_015 23437934592 1011 2798152704 (12%) 36959840 (0%) 172 222 d10_24_016 23437934592 1011 2798002176 (12%) 36954976 (0%) 173 222 d10_25_017 23437934592 1011 2798524416 (12%) 36737120 (0%) 174 222 d10_26_018 23437934592 1011 2797768704 (12%) 36888096 (0%) 175 222 d11_21_013 23437934592 1011 2958944256 (13%) 37504288 (0%) 176 222 d11_22_014 23437934592 1011 2957225984 (13%) 37575648 (0%) 177 222 d11_23_015 23437934592 1011 2958100480 (13%) 37546336 (0%) 178 222 d11_24_016 23437934592 1011 2958453760 (13%) 37578784 (0%) 179 222 d11_25_017 23437934592 1011 2958053376 (13%) 37331808 (0%) 180 222 d11_26_018 23437934592 1011 2958290944 (13%) 37445824 (0%) -------------- next part -------------- 19 223 d12_31_019 23437934592 1213 3803873280 (16%) 117281408 (1%) 20 222 d12_32_020 23437934592 1213 3779237888 (16%) 116652576 (0%) 21 222 d12_33_021 23437934592 1213 3779206144 (16%) 116820288 (0%) 22 222 d12_34_022 23437934592 1213 3776828416 (16%) 116108032 (0%) 23 222 d12_35_023 23437934592 1213 3776950272 (16%) 116238752 (0%) 24 222 d12_36_024 23437934592 1213 3776524288 (16%) 116599744 (0%) 49 222 d13_31_019 23437934592 1213 3778293760 (16%) 116365600 (0%) 50 222 d13_32_020 23437934592 1213 3778381824 (16%) 115960448 (0%) 51 222 d13_33_021 23437934592 1213 3777104896 (16%) 116225984 (0%) 52 222 d13_34_022 23437934592 1213 3775632384 (16%) 116496576 (0%) 53 222 d13_35_023 23437934592 1213 3776899072 (16%) 116397120 (0%) 54 222 d13_36_024 23437934592 1213 3776140288 (16%) 116400000 (0%) 79 222 d14_31_019 23437934592 1415 3844638720 (16%) 113531904 (0%) 80 222 d14_32_020 23437934592 1415 3817139200 (16%) 113600928 (0%) 81 222 d14_33_021 23437934592 1415 3815447552 (16%) 113358272 (0%) 82 222 d14_34_022 23437934592 1415 3814561792 (16%) 113348160 (0%) 83 222 d14_35_023 23437934592 1415 3815387136 (16%) 113582432 (0%) 84 222 d14_36_024 23437934592 1415 3815303168 (16%) 112914816 (0%) 109 222 d15_31_019 23437934592 1415 3814614016 (16%) 113298240 (0%) 110 222 d15_32_020 23437934592 1415 3814936576 (16%) 113053664 (0%) 111 222 d15_33_021 23437934592 1415 3815122944 (16%) 113438656 (0%) 112 222 d15_34_022 23437934592 1415 3813474304 (16%) 113481216 (0%) 113 222 d15_35_023 23437934592 1415 3812912128 (16%) 113618624 (0%) 114 222 d15_36_024 23437934592 1415 3813067776 (16%) 113282848 (0%) 134 223 d11_45_029 23437934592 1011 3683404800 (16%) 100860000 (0%) 135 222 d13_45_029 23437934592 1213 3682211840 (16%) 101101824 (0%) 136 223 d15_45_029 23437934592 1415 3682046976 (16%) 101375328 (0%) 137 222 d10_46_030 23437934592 1011 3684398080 (16%) 100946912 (0%) 138 222 d11_46_030 23437934592 1011 3683755008 (16%) 101295200 (0%) 139 223 d12_46_030 23437934592 1213 3682649088 (16%) 100540832 (0%) 140 223 d13_46_030 23437934592 1213 3684602880 (16%) 100489376 (0%) 141 223 d14_46_030 23437934592 1415 3681666048 (16%) 100854112 (0%) 142 223 d15_46_030 23437934592 1415 3685487616 (16%) 100715744 (0%) 181 222 d10_31_019 23437934592 1011 3816681472 (16%) 113389344 (0%) 182 222 d10_32_020 23437934592 1011 3816535040 (16%) 113652672 (0%) 183 222 d10_33_021 23437934592 1011 3817772032 (16%) 113124224 (0%) 184 222 d10_34_022 23437934592 1011 3817069568 (16%) 113225504 (0%) 185 222 d10_35_023 23437934592 1011 3816517632 (16%) 113163488 (0%) 186 222 d10_36_024 23437934592 1011 3815605248 (16%) 113475648 (0%) 187 222 d11_31_019 23437934592 1011 3815664640 (16%) 113100992 (0%) 188 222 d11_32_020 23437934592 1011 3815985152 (16%) 113325504 (0%) 189 222 d11_33_021 23437934592 1011 3815148544 (16%) 113328480 (0%) 190 223 d11_34_022 23437934592 1011 3814780928 (16%) 113457440 (0%) 191 223 d11_35_023 23437934592 1011 3815219200 (16%) 113065504 (0%) 192 223 d11_36_024 23437934592 1011 3815383040 (16%) 113170592 (0%) -------------- next part -------------- > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From aaron.s.knister at nasa.gov Sat Jan 13 17:26:51 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sat, 13 Jan 2018 12:26:51 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Message-ID: Thanks, Peter. That definitely makes sense and I was actually wondering if performance was a factor. Do you know where to look to see what GPFS' perception of "performance" is for a given NSD? -Aaron On 1/13/18 12:00 PM, Peter Serocka wrote: > Within reasonable capacity limits it would also expect > to direct incoming data to disks that are best ?available? > from a current performance perspective ? like doing least > IOPS, having lowest latency and shortest filled queue length. > > You new NSDs, filled only with recent data, might quickly have > become the most busy units before reaching capacity balance, > simply because recent data tends to be more active than older stuff. > > Makes sense? > > ? Peter > >> On 2018 Jan 13 Sat, at 17:18, Aaron Knister wrote: >> >> Thanks Everyone! I whipped up a script to dump the block layout of a >> file and then join that with mmdf information. As part of my exploration >> I wrote one 2GB file to each of this particular filesystem's 4 data >> pools last night (using "touch $file; mmchattr $file -P $pool; dd >> of=$file") and have attached a dump of the layout/nsd information for >> each file/pool. The fields for the output are: >> >> diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, >> freePct, freeKbFragments, freeKbFragmentsPct >> >> >> Here's the highlight from pool1: >> >> 36? 264? d13_06_006??? 23437934592? 1213??? 4548935680? (19%) >> 83304320?? (0%) >> 59?? 74? d10_41_025??? 23437934592? 1011??? 6993759232? (30%) >> 58642816?? (0%) >> >> For this file (And anecdotally what I've seen looking at NSD I/O data >> for other files written to this pool) the pattern of more blocks being >> allocated to the NSDs that are ~20% free vs the NSDs that are 30% free >> seems to be fairly consistent. >> >> >> Looking at a snippet of pool2: >> 101? 238? d15_15_011??? 23437934592? 1415??? 2008394752?? (9%) >> 181699328?? (1%) >> 102? 235? d15_16_012??? 23437934592? 1415??? 2009153536?? (9%) >> 182165312?? (1%) >> 116? 248? d11_42_026??? 23437934592? 1011??? 4146111488? (18%) >> 134941504?? (1%) >> 117? 249? d13_42_026??? 23437934592? 1213??? 4147710976? (18%) >> 135203776?? (1%) >> >> there are slightly more blocks allocated in general on the NSDs with >> twice the amount of free space, but it doesn't seem to be a significant >> amount relative to the delta in free space. The pattern from pool1 >> certainly doesn't hold true here. >> >> Pool4 isn't very interesting because all of the NSDs are well balanced >> in terms of free space (all 16% free). >> >> Pool3, however, *is* particularly interesting. Here's a snippet: >> >> 106? 222? d15_24_016??? 23437934592? 1415??? 2957561856? (13%) >> 37436768?? (0%) >> 107? 222? d15_25_017??? 23437934592? 1415??? 2957537280? (13%) >> 37353984?? (0%) >> 108? 222? d15_26_018??? 23437934592? 1415??? 2957539328? (13%) >> 37335872?? (0%) >> 125? 222? d11_44_028??? 23437934592? 1011?? 13297235968? (57%) >> 20505568?? (0%) >> 126? 222? d12_44_028??? 23437934592? 1213?? 13296712704? (57%) >> 20632768?? (0%) >> 127? 222? d12_45_029??? 23437934592? 1213?? 13297404928? (57%) >> 20557408?? (0%) >> >> GPFS consistently allocated the same number of blocks to disks with ~4x >> the free space than it did the other disks in the pool. >> >> Suffice it to say-- I'm *very* confused :) >> >> -Aaron >> >> On 1/13/18 8:18 AM, Daniel Kidger wrote: >>> Aaron, >>>? >>> Also are your new NSDs the same size as your existing ones? >>> i.e. the NSDs that are at a higher %age full might have more free blocks >>> than the other NSDs? >>> Daniel >>> >>>? >>> IBM Storage Professional Badge >>> >>>? >>>???????????????? >>> *Dr Daniel Kidger* >>> IBM Technical Sales Specialist >>> Software Defined Solution Sales >>> >>> +44-(0)7818 522 266 >>> daniel.kidger at uk.ibm.com >>> >>>? >>>? >>>? >>> >>>??? ----- Original message ----- >>>??? From: Jan-Frode Myklebust >>>??? Sent by: gpfsug-discuss-bounces at spectrumscale.org >>>??? To: gpfsug main discussion list >>>??? Cc: >>>??? Subject: Re: [gpfsug-discuss] pool block allocation algorithm >>>??? Date: Sat, Jan 13, 2018 9:25 AM >>>???? >>>??? Don?t have documentation/whitepaper, but as I recall, it will first >>>??? allocate round-robin over failureGroup, then round-robin over >>>??? nsdServers, and then round-robin over volumes. So if these new NSDs >>>??? are defined as different failureGroup from the old disks, that might >>>??? explain it.. >>> >>> >>>??? -jf >>>??? l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >>>??? >: >>> >>>??????? Apologies if this has been covered elsewhere (I couldn't find it >>>??????? if it >>>??????? has). I'm curious how GPFS decides where to allocate new blocks. >>> >>>??????? We've got a filesystem that we added some NSDs to a while back >>>??????? and it >>>??????? hurt there for a little while because it appeared as though GPFS was >>>??????? choosing to allocate new blocks much more frequently on the >>>??????? ~100% free >>>??????? LUNs than the existing LUNs (I can't recall how free they were >>>??????? at the >>>??????? time). Looking at it now, though, it seems GPFS is doing the >>>??????? opposite. >>>??????? There's now a ~10% difference between the LUNs added and the >>>??????? existing >>>??????? LUNs (20% free vs 30% free) and GPFS is choosing to allocate new >>>??????? writes >>>??????? at a ratio of about 3:1 on the disks with *fewer* free blocks >>>??????? than on >>>??????? the disks with more free blocks. That's completely inconsistent with >>>??????? what we saw when we initially added the disks which makes me >>>??????? wonder how >>>??????? GPFS is choosing to allocate new blocks (other than the obvious bits >>>??????? about failure group, and replication factor). Could someone >>>??????? explain (or >>>??????? point me at a whitepaper) what factors GPFS uses when allocating >>>??????? blocks, >>>??????? particularly as it pertains to choosing one NSD over another >>>??????? within the >>>??????? same failure group. >>> >>>??????? Thanks! >>> >>>??????? -Aaron >>> >>>??????? -- >>>??????? Aaron Knister >>>??????? NASA Center for Climate Simulation (Code 606.2) >>>??????? Goddard Space Flight Center >>>??????? (301) 286-2776 >>>??????? _______________________________________________ >>>??????? gpfsug-discuss mailing list >>>??????? gpfsug-discuss at spectrumscale.org >>>??????? >>>??????? http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>??????? >>> >>>??? _______________________________________________ >>>??? gpfsug-discuss mailing list >>>??? gpfsug-discuss at spectrumscale.org >>>??? https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe-GTf8EwJ6AkZQiTsRQZ73UH20&e= >>> >>>? >>> Unless stated otherwise above: >>> IBM United Kingdom Limited - Registered in England and Wales with number >>> 741598. >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From wsawdon at us.ibm.com Sat Jan 13 19:43:22 2018 From: wsawdon at us.ibm.com (Wayne Sawdon) Date: Sat, 13 Jan 2018 11:43:22 -0800 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov><48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Message-ID: Originally, GPFS used a strict round robin, first over failure groups, then over volumes within each failure group. That had performance issues when one or more volumes was low on space. Then for a while there were a variety of weighted stripe methods including by free space and by capacity. The file system had an option allowing the user to change the stripe method. That option was removed when we switched to a "best effort" round robin, which does a round robin over the failure groups then volumes based on the allocation regions that a node owns. When the stripe width at a node drops below half of the failure groups or half of the volumes that node will acquire new allocation regions. Basically we vary the stripe width to avoid searching for free space on specific volumes. It will eventually even itself out or you could restripe the file system to even it out immediately. -Wayne ps. And of course, allocation in FPO is completely different. gpfsug-discuss-bounces at spectrumscale.org wrote on 01/13/2018 09:26:51 AM: > From: Aaron Knister > To: gpfsug main discussion list > Date: 01/13/2018 09:27 AM > Subject: Re: [gpfsug-discuss] pool block allocation algorithm > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Thanks, Peter. That definitely makes sense and I was actually wondering > if performance was a factor. Do you know where to look to see what GPFS' > perception of "performance" is for a given NSD? > > -Aaron > > On 1/13/18 12:00 PM, Peter Serocka wrote: > > Within reasonable capacity limits it would also expect > > to direct incoming data to disks that are best ?available? > > from a current performance perspective ? like doing least > > IOPS, having lowest latency and shortest filled queue length. > > > > You new NSDs, filled only with recent data, might quickly have > > become the most busy units before reaching capacity balance, > > simply because recent data tends to be more active than older stuff. > > > > Makes sense? > > > > ? Peter > > > >> On 2018 Jan 13 Sat, at 17:18, Aaron Knister > wrote: > >> > >> Thanks Everyone! I whipped up a script to dump the block layout of a > >> file and then join that with mmdf information. As part of my exploration > >> I wrote one 2GB file to each of this particular filesystem's 4 data > >> pools last night (using "touch $file; mmchattr $file -P $pool; dd > >> of=$file") and have attached a dump of the layout/nsd information for > >> each file/pool. The fields for the output are: > >> > >> diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, > >> freePct, freeKbFragments, freeKbFragmentsPct > >> > >> > >> Here's the highlight from pool1: > >> > >> 36? 264? d13_06_006??? 23437934592? 1213??? 4548935680? (19%) > >> 83304320?? (0%) > >> 59?? 74? d10_41_025??? 23437934592? 1011??? 6993759232? (30%) > >> 58642816?? (0%) > >> > >> For this file (And anecdotally what I've seen looking at NSD I/O data > >> for other files written to this pool) the pattern of more blocks being > >> allocated to the NSDs that are ~20% free vs the NSDs that are 30% free > >> seems to be fairly consistent. > >> > >> > >> Looking at a snippet of pool2: > >> 101? 238? d15_15_011??? 23437934592? 1415??? 2008394752?? (9%) > >> 181699328?? (1%) > >> 102? 235? d15_16_012??? 23437934592? 1415??? 2009153536?? (9%) > >> 182165312?? (1%) > >> 116? 248? d11_42_026??? 23437934592? 1011??? 4146111488? (18%) > >> 134941504?? (1%) > >> 117? 249? d13_42_026??? 23437934592? 1213??? 4147710976? (18%) > >> 135203776?? (1%) > >> > >> there are slightly more blocks allocated in general on the NSDs with > >> twice the amount of free space, but it doesn't seem to be a significant > >> amount relative to the delta in free space. The pattern from pool1 > >> certainly doesn't hold true here. > >> > >> Pool4 isn't very interesting because all of the NSDs are well balanced > >> in terms of free space (all 16% free). > >> > >> Pool3, however, *is* particularly interesting. Here's a snippet: > >> > >> 106? 222? d15_24_016??? 23437934592? 1415??? 2957561856? (13%) > >> 37436768?? (0%) > >> 107? 222? d15_25_017??? 23437934592? 1415??? 2957537280? (13%) > >> 37353984?? (0%) > >> 108? 222? d15_26_018??? 23437934592? 1415??? 2957539328? (13%) > >> 37335872?? (0%) > >> 125? 222? d11_44_028??? 23437934592? 1011?? 13297235968? (57%) > >> 20505568?? (0%) > >> 126? 222? d12_44_028??? 23437934592? 1213?? 13296712704? (57%) > >> 20632768?? (0%) > >> 127? 222? d12_45_029??? 23437934592? 1213?? 13297404928? (57%) > >> 20557408?? (0%) > >> > >> GPFS consistently allocated the same number of blocks to disks with ~4x > >> the free space than it did the other disks in the pool. > >> > >> Suffice it to say-- I'm *very* confused :) > >> > >> -Aaron > >> > >> On 1/13/18 8:18 AM, Daniel Kidger wrote: > >>> Aaron, > >>> > >>> Also are your new NSDs the same size as your existing ones? > >>> i.e. the NSDs that are at a higher %age full might have more free blocks > >>> than the other NSDs? > >>> Daniel > >>> > >>> > >>> IBM Storage Professional Badge > >>> u=https-3A__www.youracclaim.com_user_danel-2Dkidger&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=hu8pcNGJmsITfq8y9fzxDf9WoXD1Kr5ptVLEEpbwcjU&e=> > >>> > >>> > >>> *Dr Daniel Kidger* > >>> IBM Technical Sales Specialist > >>> Software Defined Solution Sales > >>> > >>> +44-(0)7818 522 266 > >>> daniel.kidger at uk.ibm.com > >>> > >>> > >>> > >>> > >>> > >>>??? ----- Original message ----- > >>>??? From: Jan-Frode Myklebust > >>>??? Sent by: gpfsug-discuss-bounces at spectrumscale.org > >>>??? To: gpfsug main discussion list > >>>??? Cc: > >>>??? Subject: Re: [gpfsug-discuss] pool block allocation algorithm > >>>??? Date: Sat, Jan 13, 2018 9:25 AM > >>> > >>>??? Don?t have documentation/whitepaper, but as I recall, it will first > >>>??? allocate round-robin over failureGroup, then round-robin over > >>>??? nsdServers, and then round-robin over volumes. So if these new NSDs > >>>??? are defined as different failureGroup from the old disks, that might > >>>??? explain it.. > >>> > >>> > >>>??? -jf > >>>??? l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister > >>>??? >: > >>> > >>>??????? Apologies if this has been covered elsewhere (I couldn't find it > >>>??????? if it > >>>??????? has). I'm curious how GPFS decides where to allocate new blocks. > >>> > >>>??????? We've got a filesystem that we added some NSDs to a while back > >>>??????? and it > >>>??????? hurt there for a little while because it appeared as > though GPFS was > >>>??????? choosing to allocate new blocks much more frequently on the > >>>??????? ~100% free > >>>??????? LUNs than the existing LUNs (I can't recall how free they were > >>>??????? at the > >>>??????? time). Looking at it now, though, it seems GPFS is doing the > >>>??????? opposite. > >>>??????? There's now a ~10% difference between the LUNs added and the > >>>??????? existing > >>>??????? LUNs (20% free vs 30% free) and GPFS is choosing to allocate new > >>>??????? writes > >>>??????? at a ratio of about 3:1 on the disks with *fewer* free blocks > >>>??????? than on > >>>??????? the disks with more free blocks. That's completely > inconsistent with > >>>??????? what we saw when we initially added the disks which makes me > >>>??????? wonder how > >>>??????? GPFS is choosing to allocate new blocks (other than the > obvious bits > >>>??????? about failure group, and replication factor). Could someone > >>>??????? explain (or > >>>??????? point me at a whitepaper) what factors GPFS uses when allocating > >>>??????? blocks, > >>>??????? particularly as it pertains to choosing one NSD over another > >>>??????? within the > >>>??????? same failure group. > >>> > >>>??????? Thanks! > >>> > >>>??????? -Aaron > >>> > >>>??????? -- > >>>??????? Aaron Knister > >>>??????? NASA Center for Climate Simulation (Code 606.2) > >>>??????? Goddard Space Flight Center > >>>??????? (301) 286-2776 > >>>??????? _______________________________________________ > >>>??????? gpfsug-discuss mailing list > >>>??????? gpfsug-discuss at spectrumscale.org > >>>??????? u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=QYsXVDOdNRcII7FPtAbCXEyYJzNSd_UXq8bmreALKxs&e= > > > >>>??????? https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > >>>??????? u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- > GTf8EwJ6AkZQiTsRQZ73UH20&e=> > >>> > >>>??? _______________________________________________ > >>>??? gpfsug-discuss mailing list > >>>??? gpfsug-discuss at spectrumscale.org > >>>??? https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- > GTf8EwJ6AkZQiTsRQZ73UH20&e= > >>> > >>> > >>> Unless stated otherwise above: > >>> IBM United Kingdom Limited - Registered in England and Wales with number > >>> 741598. > >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > >>> > >>> > >>> > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > >>> > >> > >> -- > >> Aaron Knister > >> NASA Center for Climate Simulation (Code 606.2) > >> Goddard Space Flight Center > >> (301) 286-2776 > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- > Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sun Jan 14 22:22:15 2018 From: aaron.s.knister at nasa.gov (Aaron Knister) Date: Sun, 14 Jan 2018 17:22:15 -0500 Subject: [gpfsug-discuss] pool block allocation algorithm In-Reply-To: References: <74d7c702-9469-96f4-9829-e63232c45419@nasa.gov> <48355D5B-6F89-4443-9CCE-E3B5613F8514@gmail.com> Message-ID: <80dccad3-f531-3a8e-ab28-3ffa83dbd990@nasa.gov> Thanks, Wayne. What you said makes sense although I'm not sure I completely grok it. Can you comment on whether or not historic LUN performance factors into allocation decisions? -Aaron On 1/13/18 2:43 PM, Wayne Sawdon wrote: > Originally, GPFS used a strict round robin, first over failure groups, > then over volumes within each > failure group. That had performance issues when one or more volumes was > low on space. Then > for a while there were a variety of weighted stripe methods including by > free space and by capacity. > The file system had an option allowing the user to change the stripe > method. That option was > removed when we switched to a "best effort" round robin, which does a > round robin over the > failure groups then volumes based on the allocation regions that a node > owns. When the stripe width > at a node drops below half of the failure groups or half of the volumes > that node will acquire new > allocation regions. > > Basically we vary the stripe width to avoid searching for free space on > specific volumes. It will > eventually even itself out or you could restripe the file system to even > it out immediately. > > -Wayne > > ps. And of course, allocation in FPO is completely different. > > > gpfsug-discuss-bounces at spectrumscale.org wrote on 01/13/2018 09:26:51 AM: > >> From: Aaron Knister >> To: gpfsug main discussion list >> Date: 01/13/2018 09:27 AM >> Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> >> Thanks, Peter. That definitely makes sense and I was actually wondering >> if performance was a factor. Do you know where to look to see what GPFS' >> perception of "performance" is for a given NSD? >> >> -Aaron >> >> On 1/13/18 12:00 PM, Peter Serocka wrote: >> > Within reasonable capacity limits it would also expect >> > to direct incoming data to disks that are best ?available? >> > from a current performance perspective ? like doing least >> > IOPS, having lowest latency and shortest filled queue length. >> > >> > You new NSDs, filled only with recent data, might quickly have >> > become the most busy units before reaching capacity balance, >> > simply because recent data tends to be more active than older stuff. >> > >> > Makes sense? >> > >> > ? Peter >> > >> >> On 2018 Jan 13 Sat, at 17:18, Aaron Knister >> wrote: >> >> >> >> Thanks Everyone! I whipped up a script to dump the block layout of a >> >> file and then join that with mmdf information. As part of my > exploration >> >> I wrote one 2GB file to each of this particular filesystem's 4 data >> >> pools last night (using "touch $file; mmchattr $file -P $pool; dd >> >> of=$file") and have attached a dump of the layout/nsd information for >> >> each file/pool. The fields for the output are: >> >> >> >> diskId, numBlocksOnDisk, diskName, diskSize, failureGroup, freeBlocks, >> >> freePct, freeKbFragments, freeKbFragmentsPct >> >> >> >> >> >> Here's the highlight from pool1: >> >> >> >> 36? 264? d13_06_006??? 23437934592? 1213??? 4548935680? (19%) >> >> 83304320?? (0%) >> >> 59?? 74? d10_41_025??? 23437934592? 1011??? 6993759232? (30%) >> >> 58642816?? (0%) >> >> >> >> For this file (And anecdotally what I've seen looking at NSD I/O data >> >> for other files written to this pool) the pattern of more blocks being >> >> allocated to the NSDs that are ~20% free vs the NSDs that are 30% free >> >> seems to be fairly consistent. >> >> >> >> >> >> Looking at a snippet of pool2: >> >> 101? 238? d15_15_011??? 23437934592? 1415??? 2008394752?? (9%) >> >> 181699328?? (1%) >> >> 102? 235? d15_16_012??? 23437934592? 1415??? 2009153536?? (9%) >> >> 182165312?? (1%) >> >> 116? 248? d11_42_026??? 23437934592? 1011??? 4146111488? (18%) >> >> 134941504?? (1%) >> >> 117? 249? d13_42_026??? 23437934592? 1213??? 4147710976? (18%) >> >> 135203776?? (1%) >> >> >> >> there are slightly more blocks allocated in general on the NSDs with >> >> twice the amount of free space, but it doesn't seem to be a significant >> >> amount relative to the delta in free space. The pattern from pool1 >> >> certainly doesn't hold true here. >> >> >> >> Pool4 isn't very interesting because all of the NSDs are well balanced >> >> in terms of free space (all 16% free). >> >> >> >> Pool3, however, *is* particularly interesting. Here's a snippet: >> >> >> >> 106? 222? d15_24_016??? 23437934592? 1415??? 2957561856? (13%) >> >> 37436768?? (0%) >> >> 107? 222? d15_25_017??? 23437934592? 1415??? 2957537280? (13%) >> >> 37353984?? (0%) >> >> 108? 222? d15_26_018??? 23437934592? 1415??? 2957539328? (13%) >> >> 37335872?? (0%) >> >> 125? 222? d11_44_028??? 23437934592? 1011?? 13297235968? (57%) >> >> 20505568?? (0%) >> >> 126? 222? d12_44_028??? 23437934592? 1213?? 13296712704? (57%) >> >> 20632768?? (0%) >> >> 127? 222? d12_45_029??? 23437934592? 1213?? 13297404928? (57%) >> >> 20557408?? (0%) >> >> >> >> GPFS consistently allocated the same number of blocks to disks with ~4x >> >> the free space than it did the other disks in the pool. >> >> >> >> Suffice it to say-- I'm *very* confused :) >> >> >> >> -Aaron >> >> >> >> On 1/13/18 8:18 AM, Daniel Kidger wrote: >> >>> Aaron, >> >>>? >> >>> Also are your new NSDs the same size as your existing ones? >> >>> i.e. the NSDs that are at a higher %age full might have more free > blocks >> >>> than the other NSDs? >> >>> Daniel >> >>> >> >>>? >> >>> IBM Storage Professional Badge >> >>> > > u=https-3A__www.youracclaim.com_user_danel-2Dkidger&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=hu8pcNGJmsITfq8y9fzxDf9WoXD1Kr5ptVLEEpbwcjU&e=> >> >>>? >> >>>???????????????? >> >>> *Dr Daniel Kidger* >> >>> IBM Technical Sales Specialist >> >>> Software Defined Solution Sales >> >>> >> >>> +44-(0)7818 522 266 >> >>> daniel.kidger at uk.ibm.com >> >>> >> >>>? >> >>>? >> >>>? >> >>> >> >>>??? ----- Original message ----- >> >>>??? From: Jan-Frode Myklebust >> >>>??? Sent by: gpfsug-discuss-bounces at spectrumscale.org >> >>>??? To: gpfsug main discussion list >> >>>??? Cc: >> >>>??? Subject: Re: [gpfsug-discuss] pool block allocation algorithm >> >>>??? Date: Sat, Jan 13, 2018 9:25 AM >> >>>???? >> >>>??? Don?t have documentation/whitepaper, but as I recall, it will first >> >>>??? allocate round-robin over failureGroup, then round-robin over >> >>>??? nsdServers, and then round-robin over volumes. So if these new NSDs >> >>>??? are defined as different failureGroup from the old disks, that > might >> >>>??? explain it.. >> >>> >> >>> >> >>>??? -jf >> >>>??? l?r. 13. jan. 2018 kl. 00:15 skrev Aaron Knister >> >>>??? >: >> >>> >> >>>??????? Apologies if this has been covered elsewhere (I couldn't > find it >> >>>??????? if it >> >>>??????? has). I'm curious how GPFS decides where to allocate new > blocks. >> >>> >> >>>??????? We've got a filesystem that we added some NSDs to a while back >> >>>??????? and it >> >>>??????? hurt there for a little while because it appeared as >> though GPFS was >> >>>??????? choosing to allocate new blocks much more frequently on the >> >>>??????? ~100% free >> >>>??????? LUNs than the existing LUNs (I can't recall how free they were >> >>>??????? at the >> >>>??????? time). Looking at it now, though, it seems GPFS is doing the >> >>>??????? opposite. >> >>>??????? There's now a ~10% difference between the LUNs added and the >> >>>??????? existing >> >>>??????? LUNs (20% free vs 30% free) and GPFS is choosing to > allocate new >> >>>??????? writes >> >>>??????? at a ratio of about 3:1 on the disks with *fewer* free blocks >> >>>??????? than on >> >>>??????? the disks with more free blocks. That's completely >> inconsistent with >> >>>??????? what we saw when we initially added the disks which makes me >> >>>??????? wonder how >> >>>??????? GPFS is choosing to allocate new blocks (other than the >> obvious bits >> >>>??????? about failure group, and replication factor). Could someone >> >>>??????? explain (or >> >>>??????? point me at a whitepaper) what factors GPFS uses when > allocating >> >>>??????? blocks, >> >>>??????? particularly as it pertains to choosing one NSD over another >> >>>??????? within the >> >>>??????? same failure group. >> >>> >> >>>??????? Thanks! >> >>> >> >>>??????? -Aaron >> >>> >> >>>??????? -- >> >>>??????? Aaron Knister >> >>>??????? NASA Center for Climate Simulation (Code 606.2) >> >>>??????? Goddard Space Flight Center >> >>>??????? (301) 286-2776 >> >>>??????? _______________________________________________ >> >>>??????? gpfsug-discuss mailing list >> >>>??????? gpfsug-discuss at spectrumscale.org >> >>>??????? > u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=QYsXVDOdNRcII7FPtAbCXEyYJzNSd_UXq8bmreALKxs&e= >> > >> >>>??????? https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> >>>??????? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- >> GTf8EwJ6AkZQiTsRQZ73UH20&e=> >> >>> >> >>>??? _______________________________________________ >> >>>??? gpfsug-discuss mailing list >> >>>??? gpfsug-discuss at spectrumscale.org >> >>>??? https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=f89xvht1uMUzAcLpusakZb1snMOgweGu0KTkKp9oedI&s=NSd3e2hwIKBCwSxsKe- >> GTf8EwJ6AkZQiTsRQZ73UH20&e= >> >>> >> >>>? >> >>> Unless stated otherwise above: >> >>> IBM United Kingdom Limited - Registered in England and Wales with > number >> >>> 741598. >> >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > PO6 3AU >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> gpfsug-discuss mailing list >> >>> gpfsug-discuss at spectrumscale.org >> >>> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> >>> >> >> >> >> -- >> >> Aaron Knister >> >> NASA Center for Climate Simulation (Code 606.2) >> >> Goddard Space Flight Center >> >> (301) 286-2776 >> >> _______________________________________________ >> >> gpfsug-discuss mailing list >> >> gpfsug-discuss at spectrumscale.org >> >> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> >> -- >> Aaron Knister >> NASA Center for Climate Simulation (Code 606.2) >> Goddard Space Flight Center >> (301) 286-2776 >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=GtPIT10cORUM6qwFnTVtIiDUFmESkxW3I0wu8GDxmgc&m=JJtwh_OoxH4AMzXA5IDarV4i- >> Xi8IMuVUeAtBxlTznA&s=-7RY2fbJ0kmb6CgKfJIoXbHkE4pIZ4L9IDEap4AbyIQ&e= >> > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 From kaustubh.katruwar at in.ibm.com Mon Jan 15 09:50:37 2018 From: kaustubh.katruwar at in.ibm.com (Kaustubh I Katruwar) Date: Mon, 15 Jan 2018 15:20:37 +0530 Subject: [gpfsug-discuss] SMB with LDAP In-Reply-To: References: Message-ID: Hi, Can you also describe your problem? The link talks about updating the LDAP server entries to help integrate with Spectrum Scale to serve SMB connections. Thanks and Regards, Kaustubh I. Katruwar From: Jeno Cram To: "gpfsug-discuss at spectrumscale.org" Date: 09/01/2018 09:06 PM Subject: [gpfsug-discuss] SMB with LDAP Sent by: gpfsug-discuss-bounces at spectrumscale.org Has anyone had any luck implementing CES with SMB using LDAP? This link doesn?t seem to have all of the information required to getting it working properly. https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adm_updateldapsmb.htm I?m assuming smbpasswd -a is still required for all users? What keeps the LDAP/SMB passwords in sync? What access does the bind user need in LDAP? What special requirements are required for Windows 10 clients to connect? Jeno Cram | Lead Application Support Engineer jcram at ddn.com _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=gTocoNEzBZA5cNptWoAoyoucYhc_7VA6hXTPOXtubUc&m=NEFCJtsgntiVgznfKbVs0BqwQJOubKG8dhgNlMXvQzs&s=0U9-rvVW4HxBCh0tdCYaElQer5JBxJJ9KFFWkkutUQU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From a.khiredine at meteo.dz Mon Jan 15 10:41:12 2018 From: a.khiredine at meteo.dz (atmane khiredine) Date: Mon, 15 Jan 2018 10:41:12 +0000 Subject: [gpfsug-discuss] temperature disk gss Message-ID: <4B32CB5C696F2849BDEF7DF9EACE884B85A15F93@SDEB-EXC01.meteo.dz> Dear All, is there a way to display the temperature of the drives in GSS Thanks Everyone! Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz From coetzee.ray at gmail.com Mon Jan 15 12:26:43 2018 From: coetzee.ray at gmail.com (Ray Coetzee) Date: Mon, 15 Jan 2018 12:26:43 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: Hi Sven yes, it's primarily reads. Kind regards Ray Coetzee Mob: +44 759 704 7060 Skype: ray.coetzee Email: coetzee.ray at gmail.com On Fri, Jan 12, 2018 at 8:57 PM, Sven Oehme wrote: > is this primary read or write ? > > > On Fri, Jan 12, 2018, 12:51 PM Ray Coetzee wrote: > >> Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. >> (Without the meltdown patch) >> >> Hey Bryan, I remember that quote from Yuri, that's why I hoped some >> "magic" may have been done since then. >> >> Other attempts to improve performance I've tried include: >> >> - Using LROC to have a larger chance of a cache hit (Unfortunately >> the entire dataset is multiple TB) >> - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for >> this purpose (Job runs halved in duration but nowhere near what NFS can >> give) >> - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and >> some others with little improvement. >> >> For those interested, as a performance comparison. The same job when run >> on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe >> scratch filesystem and over 60min on spindle based filesystem. >> >> Kind regards >> >> Ray Coetzee >> Email: coetzee.ray at gmail.com >> >> >> On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister < >> bbanister at jumptrading.com> wrote: >> >>> You could put all of your data onto SSDs in a RAID1 configuration so >>> that you don?t have insane read-modify-write penalties on writes (RAID1) >>> and avoid horrible seek thrashing that spinning rust requires (SSD random >>> access medium) for your 4K I/O operations. >>> >>> >>> >>> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best >>> not to touch it?. He gave many reasons why mmap operations on a >>> distributed file system is incredibly hard and not recommended. >>> >>> -Bryan >>> >>> >>> >>> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- >>> bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >>> *Sent:* Friday, January 12, 2018 8:45 AM >>> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >>> gpfsug-discuss at spectrumscale.org> >>> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >>> >>> >>> >>> *Note: External Email* >>> ------------------------------ >>> >>> what version of Scale are you using right now ? >>> >>> >>> >>> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >>> wrote: >>> >>> I'd like to ask the group of their experiences in improving the >>> performance of applications that use mmap calls against files on Spectrum >>> Scale. >>> >>> >>> >>> Besides using an NFS export from CES instead of a native GPFS mount, or >>> precaching the dataset into the pagepool, what other approaches are there >>> to offset the performance hit of the 4K IO size? >>> >>> >>> >>> Kind regards >>> >>> Ray Coetzee >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> >>> ------------------------------ >>> >>> Note: This email is for the confidential use of the named addressee(s) >>> only and may contain proprietary, confidential or privileged information. >>> If you are not the intended recipient, you are hereby notified that any >>> review, dissemination or copying of this email is strictly prohibited, and >>> to please notify the sender immediately and destroy this email and any >>> attachments. Email transmission cannot be guaranteed to be secure or >>> error-free. The Company, therefore, does not make any guarantees as to the >>> completeness or accuracy of this email or any attachments. This email is >>> for informational purposes only and does not constitute a recommendation, >>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >>> or perform any type of transaction of a financial product. >>> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckerner at illinois.edu Mon Jan 15 14:03:26 2018 From: ckerner at illinois.edu (Chad Kerner) Date: Mon, 15 Jan 2018 08:03:26 -0600 Subject: [gpfsug-discuss] temperature disk gss In-Reply-To: References: <4300efd8e3134cf1ba50454cac76bd09@CHIHT3.ad.uillinois.edu> Message-ID: You can get it from the Smart statistics with smartctl -A . smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.10.0-229.el7.x86_64] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === Current Drive Temperature: 30 C Drive Trip Temperature: 65 C Manufactured in week 39 of year 2016 Specified cycle count over device lifetime: 50000 Accumulated start-stop cycles: 6 Specified load-unload count over device lifetime: 600000 Accumulated load-unload cycles: 35 Elements in grown defect list: 0 Vendor (Seagate) cache information Blocks sent to initiator = 418499985932288 Chad -- Chad Kerner, Senior Storage Engineer National Center for Supercomputing Applications University of Illinois, Urbana-Champaign On Jan 15, 2018 4:48 AM, "atmane khiredine" wrote: Dear All, is there a way to display the temperature of the drives in GSS Thanks Everyone! Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From alvise.dorigo at psi.ch Mon Jan 15 13:58:12 2018 From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI)) Date: Mon, 15 Jan 2018 13:58:12 +0000 Subject: [gpfsug-discuss] temperature disk gss In-Reply-To: <4B32CB5C696F2849BDEF7DF9EACE884B85A15F93@SDEB-EXC01.meteo.dz> References: <4B32CB5C696F2849BDEF7DF9EACE884B85A15F93@SDEB-EXC01.meteo.dz> Message-ID: <83A6EEB0EC738F459A39439733AE80451BBC421D@MBX114.d.ethz.ch> Hi Atmane, in my Gl2 system I can do: # mmlsenclosure all -L|grep -B2 tempSensor component type serial number component id failed value unit properties -------------- ------------- ------------ ------ ----- ---- ---------- tempSensor XXXXXXXXX DCM_0A no 41 C tempSensor XXXXXXXXX DCM_0B no 40 C tempSensor XXXXXXXXX DCM_1A no 49 C tempSensor XXXXXXXXX DCM_1B no 42 C tempSensor XXXXXXXXX DCM_2A no 50 C tempSensor XXXXXXXXX DCM_2B no 41 C tempSensor XXXXXXXXX DCM_3A no 48 C tempSensor XXXXXXXXX DCM_3B no 42 C tempSensor XXXXXXXXX DCM_4A no 47 C tempSensor XXXXXXXXX DCM_4B no 42 C tempSensor XXXXXXXXX ESM_A no 43 C tempSensor XXXXXXXXX ESM_B no 44 C tempSensor XXXXXXXXX POWERSUPPLY_BOT no 41 C tempSensor XXXXXXXXX POWERSUPPLY_TOP no 39 C tempSensor XXXXXXXXX DCM_0A no 50 C tempSensor XXXXXXXXX DCM_0B no 41 C tempSensor XXXXXXXXX DCM_1A no 48 C tempSensor XXXXXXXXX DCM_1B no 41 C tempSensor XXXXXXXXX DCM_2A no 47 C tempSensor XXXXXXXXX DCM_2B no 42 C tempSensor XXXXXXXXX DCM_3A no 47 C tempSensor XXXXXXXXX DCM_3B no 41 C tempSensor XXXXXXXXX DCM_4A no 47 C tempSensor XXXXXXXXX DCM_4B no 43 C tempSensor XXXXXXXXX ESM_A no 43 C tempSensor XXXXXXXXX ESM_B no 44 C tempSensor XXXXXXXXX POWERSUPPLY_BOT no 43 C tempSensor SV55261173 POWERSUPPLY_TOP no 39 C ( serials, 2nd column, are intentionally hidden by me). Check your admin guide to map the sensor names (3rd columns) to the particular locations inside your enclosures (which include locations very close to the disk drives) Alvise ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of atmane khiredine [a.khiredine at meteo.dz] Sent: Monday, January 15, 2018 11:41 AM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] temperature disk gss Dear All, is there a way to display the temperature of the drives in GSS Thanks Everyone! Atmane Khiredine HPC System Administrator | Office National de la M?t?orologie T?l : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail : a.khiredine at meteo.dz _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From oehmes at gmail.com Mon Jan 15 15:00:17 2018 From: oehmes at gmail.com (Sven Oehme) Date: Mon, 15 Jan 2018 15:00:17 +0000 Subject: [gpfsug-discuss] mmap performance against Spectrum Scale In-Reply-To: References: <4717a350329f4077bb05d893a4b515a7@jumptrading.com> Message-ID: do you have a repeatable test that runs on just a small number of nodes ? we where working (still are) on some mmap enhancements lest year, but didn't get all of them ready for the DCUT of Scale 5.0, so we integrated only a subset that was ready into Scale 5.0 and left the enhancement turned off by default (you can toggle it on/off via mmchconfig). if you are interested in testing this send me a direct mail and i will provide instructions on how to turn this on. it does help mmap read workloads significant (think factors) in testing with synthetic benchmarks in my lab, but i would be very interested in real application results and also interested to get a trace with the new code to see where we need to make further improvements. sven On Mon, Jan 15, 2018 at 4:26 AM Ray Coetzee wrote: > Hi Sven > yes, it's primarily reads. > > Kind regards > > Ray Coetzee > Mob: +44 759 704 7060 <+44%207597%20047060> > > Skype: ray.coetzee > > Email: coetzee.ray at gmail.com > > > On Fri, Jan 12, 2018 at 8:57 PM, Sven Oehme wrote: > >> is this primary read or write ? >> >> >> On Fri, Jan 12, 2018, 12:51 PM Ray Coetzee wrote: >> >>> Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. >>> (Without the meltdown patch) >>> >>> Hey Bryan, I remember that quote from Yuri, that's why I hoped some >>> "magic" may have been done since then. >>> >>> Other attempts to improve performance I've tried include: >>> >>> - Using LROC to have a larger chance of a cache hit (Unfortunately >>> the entire dataset is multiple TB) >>> - Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for >>> this purpose (Job runs halved in duration but nowhere near what NFS can >>> give) >>> - Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, >>> and some others with little improvement. >>> >>> For those interested, as a performance comparison. The same job when run >>> on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe >>> scratch filesystem and over 60min on spindle based filesystem. >>> >>> Kind regards >>> >>> Ray Coetzee >>> Email: coetzee.ray at gmail.com >>> >>> >>> On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister < >>> bbanister at jumptrading.com> wrote: >>> >>>> You could put all of your data onto SSDs in a RAID1 configuration so >>>> that you don?t have insane read-modify-write penalties on writes (RAID1) >>>> and avoid horrible seek thrashing that spinning rust requires (SSD random >>>> access medium) for your 4K I/O operations. >>>> >>>> >>>> >>>> One of my favorite Yuri quotes, ?The mmap code is like asbestos? best >>>> not to touch it?. He gave many reasons why mmap operations on a >>>> distributed file system is incredibly hard and not recommended. >>>> >>>> -Bryan >>>> >>>> >>>> >>>> *From:* gpfsug-discuss-bounces at spectrumscale.org [mailto: >>>> gpfsug-discuss-bounces at spectrumscale.org] *On Behalf Of *Sven Oehme >>>> *Sent:* Friday, January 12, 2018 8:45 AM >>>> *To:* coetzee.ray at gmail.com; gpfsug main discussion list < >>>> gpfsug-discuss at spectrumscale.org> >>>> *Subject:* Re: [gpfsug-discuss] mmap performance against Spectrum Scale >>>> >>>> >>>> >>>> *Note: External Email* >>>> ------------------------------ >>>> >>>> what version of Scale are you using right now ? >>>> >>>> >>>> >>>> On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee >>>> wrote: >>>> >>>> I'd like to ask the group of their experiences in improving the >>>> performance of applications that use mmap calls against files on Spectrum >>>> Scale. >>>> >>>> >>>> >>>> Besides using an NFS export from CES instead of a native GPFS mount, or >>>> precaching the dataset into the pagepool, what other approaches are there >>>> to offset the performance hit of the 4K IO size? >>>> >>>> >>>> >>>> Kind regards >>>> >>>> Ray Coetzee >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>>> >>>> >>>> ------------------------------ >>>> >>>> Note: This email is for the confidential use of the named addressee(s) >>>> only and may contain proprietary, confidential or privileged information. >>>> If you are not the intended recipient, you are hereby notified that any >>>> review, dissemination or copying of this email is strictly prohibited, and >>>> to please notify the sender immediately and destroy this email and any >>>> attachments. Email transmission cannot be guaranteed to be secure or >>>> error-free. The Company, therefore, does not make any guarantees as to the >>>> completeness or accuracy of this email or any attachments. This email is >>>> for informational purposes only and does not constitute a recommendation, >>>> offer, request or solicitation of any kind to buy, sell, subscribe, redeem >>>> or perform any type of transaction of a financial product. >>>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Christian.Fey at sva.de Mon Jan 15 16:16:05 2018 From: Christian.Fey at sva.de (Fey, Christian) Date: Mon, 15 Jan 2018 16:16:05 +0000 Subject: [gpfsug-discuss] switching on perfileset-quota Message-ID: <05334d4771da49b4bb5f4cb43f1c3558@sva.de> Hi all, does anyone here have experience about a possible impact of switching on "--perfileset-quota" for an existing gpfs filesystem in order to achieve quotas on per-user basis? During my very small tests I did not discover anything, but I would like to be sure. The existing filesystem already has quotas in place that hopefully do not get messed up. Is there anything I need to consider? Mit freundlichen Gr??en / Best Regards Christian Fey SVA System Vertrieb Alexander GmbH Borsigstra?e 14 65205 Wiesbaden Tel.: +49 6122 536-0 Fax: +49 6122 536-399 E-Mail: christian.fey at sva.de http://www.sva.de Gesch?ftsf?hrung: Philipp Alexander, Sven Eichelbaum Sitz der Gesellschaft: Wiesbaden Registergericht: Amtsgericht Wiesbaden, HRB 10315 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5415 bytes Desc: not available URL: From pierre-marie.brunet at cnes.fr Tue Jan 16 10:40:46 2018 From: pierre-marie.brunet at cnes.fr (Brunet Pierre-Marie) Date: Tue, 16 Jan 2018 10:40:46 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint Message-ID: Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I'm talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Tue Jan 16 12:41:28 2018 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 16 Jan 2018 12:41:28 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint Message-ID: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com> Hi PMB This is one of the areas where even the most experienced admins struggle. There is no single answer here. Much of it depends on your particular use case and the file system layout, storage choices, block sizes all go together. IBM has started to provide some templates for this (the one out is for Genomics) but much is left to do. If you could share some details on the overall user environment and data myself and others could offer some ideas. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Brunet Pierre-Marie Reply-To: gpfsug main discussion list Date: Tuesday, January 16, 2018 at 4:51 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] GPFS best practises : end user standpoint Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I?m talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jan 16 13:03:06 2018 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Tue, 16 Jan 2018 13:03:06 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com> References: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com> Message-ID: Aaron Knister liked your message with Boxer. On January 16, 2018 at 07:41:40 EST, Oesterlin, Robert wrote: Hi PMB This is one of the areas where even the most experienced admins struggle. There is no single answer here. Much of it depends on your particular use case and the file system layout, storage choices, block sizes all go together. IBM has started to provide some templates for this (the one out is for Genomics) but much is left to do. If you could share some details on the overall user environment and data myself and others could offer some ideas. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Brunet Pierre-Marie Reply-To: gpfsug main discussion list Date: Tuesday, January 16, 2018 at 4:51 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] GPFS best practises : end user standpoint Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I?m talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Tue Jan 16 13:47:19 2018 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Tue, 16 Jan 2018 13:47:19 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: <4827AC97-D2CE-46EB-922E-B7C8A4E0D92C@nuance.com>, Message-ID: <86593829-3D51-4203-98E0-E22B0A727C87@nasa.gov> Apologies for that. My mobile exchange email client has a like button you can tap in the email action menu. I just discovered it this morning accidentally and thought ?wonder what this does. Better push it to find out.? Nothing happened or so I thought. Apparently all it does is make you look like a moron on public mailing lists. Doh! On January 16, 2018 at 08:03:06 EST, Aaron Knister wrote: Aaron Knister liked your message with Boxer. On January 16, 2018 at 07:41:40 EST, Oesterlin, Robert wrote: Hi PMB This is one of the areas where even the most experienced admins struggle. There is no single answer here. Much of it depends on your particular use case and the file system layout, storage choices, block sizes all go together. IBM has started to provide some templates for this (the one out is for Genomics) but much is left to do. If you could share some details on the overall user environment and data myself and others could offer some ideas. Bob Oesterlin Sr Principal Storage Engineer, Nuance From: on behalf of Brunet Pierre-Marie Reply-To: gpfsug main discussion list Date: Tuesday, January 16, 2018 at 4:51 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] GPFS best practises : end user standpoint Hi all GPFS-gurus ! We (hardly) try to train our users in order to improve how they use storage. I found a lot of best practices on how to configure GPFS from the admin point of view but is there any documentation about how to best user a parallel filesystem like GPFS ? I?m talking about very basic rules such as max number of files / subdir into a directory. I know this is closely related to the FS and storage configuration but there may exist some common rules in order to achieve the best scalabilty, right ? Thanks for your advices, PMB -- HPC architect CNES, French space agency -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlz at us.ibm.com Tue Jan 16 15:47:54 2018 From: carlz at us.ibm.com (Carl Zetie) Date: Tue, 16 Jan 2018 15:47:54 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Tue Jan 16 16:08:15 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 16 Jan 2018 16:08:15 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: Message-ID: <1516118895.3326.25.camel@strath.ac.uk> On Tue, 2018-01-16 at 15:47 +0000, Carl Zetie wrote: > Maybe this would make for a good session at a future user group > meeting -- perhaps as an interactive session? IBM could potentially > provide a facilitator from our Design practice. > ? Most of it in my view is standard best practice regardless of the file system in use. So in our mandatory training for the HPC, we tell our users don't use whacked out characters in your file names and directories. Specifically no backticks, no asterik's, no question marks, no newlines (yes really), no slashes (either forward or backward) and for Mac users don't start the name with a space (forces sorting to the top). We recommend sticking to plain ASCII so no accented characters either (harder if your native language is not English I guess but we are UK based so...). We don't enforce that but if it causes the user problems then they are on their own. We also strongly recommend using ISO 8601 date formats in file names to get date sorting from a directory listing too. Surprisingly not widely known about, but a great "life hack". Then it boils down to don't create zillions of files. I would love to be able to somehow do per directory file number quota's where one could say set a default of a few thousand. Users would then have to justify needing a larger quota. Sure you can set a file number quota but that does not stop them putting them all in one directory. If users really need to have zillions of files then charge them more so you can afford to beef up your metadata disks to SSD. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 16 16:35:05 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 16 Jan 2018 16:35:05 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516118895.3326.25.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> Message-ID: Hi Jonathan, Comments / questions inline. Thanks! Kevin > On Jan 16, 2018, at 10:08 AM, Jonathan Buzzard wrote: > > On Tue, 2018-01-16 at 15:47 +0000, Carl Zetie wrote: >> Maybe this would make for a good session at a future user group >> meeting -- perhaps as an interactive session? IBM could potentially >> provide a facilitator from our Design practice. >> > > Most of it in my view is standard best practice regardless of the file > system in use. > > So in our mandatory training for the HPC, we tell our users don't use > whacked out characters in your file names and directories. Specifically > no backticks, no asterik's, no question marks, no newlines (yes > really), no slashes (either forward or backward) and for Mac users > don't start the name with a space (forces sorting to the top). We > recommend sticking to plain ASCII so no accented characters either > (harder if your native language is not English I guess but we are UK > based so...). We don't enforce that but if it causes the user problems > then they are on their own. We?re in Tennessee, so not only do we not speak English, we barely speak American ? y?all will just have to understand, bless your hearts! ;-). But seriously, like most Universities, we have a ton of users for whom English is not their ?primary? language, so dealing with ?interesting? filenames is pretty hard to avoid. And users? problems are our problems whether or not they?re our problem. > > We also strongly recommend using ISO 8601 date formats in file names to > get date sorting from a directory listing too. Surprisingly not widely > known about, but a great "life hack". > > Then it boils down to don't create zillions of files. I would love to > be able to somehow do per directory file number quota's where one could > say set a default of a few thousand. Users would then have to justify > needing a larger quota. Sure you can set a file number quota but that > does not stop them putting them all in one directory. If you?ve got (bio)medical users using your cluster I don?t see how you avoid this ? they?re using commercial apps that do this kind of stupid stuff (10?s of thousands of files in a directory and the full path to each file is longer than the contents of the files themselves!). This reminds me of way back in 2005 when we moved from an NFS server to GPFS ? I was moving users over by tarring up their home directories on the NFS server, copying the tarball over to GPFS and untarring it there ? worked great for 699 out of 700 users. But there was one user for whom the untar would fail every time I tried ? turned out that back in early versions of GPFS 2.3 IBM hadn?t considered that someone would put 6 million files in one directory! :-O > > If users really need to have zillions of files then charge them more so > you can afford to beef up your metadata disks to SSD. OK, so here?s my main question ? you?re right that SSD?s are the answer ? but how do you charge them more? SSDs are move expensive than hard disks, and enterprise SSDs are stupid expensive ? and users barely want to pay hard drive prices for their storage. If you?ve got the magic answer to how to charge them enough to pay for SSDs I?m sure I?m not the only one who?d love to hear how you do it?!?! > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cdd3310fd309b4986f95c08d55cfb5d10%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636517157039256068&sdata=jZZV718gaMie92MW43qaxlDl6EQcULdk6FONrXpsP7c%3D&reserved=0 From heinz.ernst at id.ethz.ch Tue Jan 16 16:56:52 2018 From: heinz.ernst at id.ethz.ch (Ernst Heinz (ID SD)) Date: Tue, 16 Jan 2018 16:56:52 +0000 Subject: [gpfsug-discuss] GPFS GA 5.0.0.0: mmces commands with inconsistent output Message-ID: <0CE1E9F220AF564DB0114D8539010F9B5229A68C@MBX116.d.ethz.ch> Hello to all peers and gurus Since more or less two weeks we have gpfs GA 5.0.0.0 running on our testenvironment Today I've seen following behavior on our SpectrumScale-testcluster which slighdly surprised me Following: Checking status of the cluster on different ways [root at testnas13ces01 idsd_erh_t1]# mmces state cluster CLUSTER AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13.ethz.ch FAILED DISABLED HEALTHY DISABLED DEPEND DISABLED DEPEND FAILED [root at testnas13ces01 idsd_erh_t1]# mmces state show -a NODE AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13ces01-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY testnas13ces02-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY does anyone of you guys has an explanation therefore? Is there someone else who has seen a behavior like this? By the way we have a similar view on one of our clusters on gpfs 4.2.3.4 (open PMR: 30218.112.848) Any kind of response would be very grateful Kind regards Heinz =============================================================== Heinz Ernst ID-Systemdienste WEC C 16 Weinbergstrasse 11 CH-8092 Zurich heinz.ernst at id.ethz.ch Phone: +41 44 633 84 48 Mobile: +41 79 216 15 50 =============================================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Tue Jan 16 17:25:47 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Tue, 16 Jan 2018 17:25:47 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: <1516118895.3326.25.camel@strath.ac.uk> Message-ID: <1516123547.3326.29.camel@strath.ac.uk> On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: [SNIP] > > We?re in Tennessee, so not only do we not speak English, we barely > speak American ? y?all will just have to understand, bless your > hearts!??;-).? > > But seriously, like most Universities, we have a ton of users for > whom English is not their ?primary? language, so dealing with > ?interesting? filenames is pretty hard to avoid.??And users? problems > are our problems whether or not they?re our problem. > User comes with problem, you investigate find problem is due to "wacky" characters point them to the mandatory training documentation, tell them they need to rename their files to something sane and take no further action. Sure English is not their primary language but *they* have chosen to study in an English speaking country so best to actually embrace that. I do get it, many of our users are not native English speakers as well. Yes it's a tough policy but on the other hand pandering to them does them no favours either. [SNIP] > If you?ve got (bio)medical users using your cluster I don?t see how > you avoid this ? they?re using commercial apps that do this kind of > stupid stuff (10?s of thousands of files in a directory and the full > path to each file is longer than the contents of the files > themselves!). Well then they have justified the use; aka it's not their fault so you up the quota for them. Though they could use different less brain dead software. The idea is to force a bump in the road so the users are aware that what they are doing is considered bad practice. Most users have no idea that putting a million files in a directory is not sensible and worse that trying to access them using a GUI file manager is positively brain dead. [SNIP] > OK, so here?s my main question ? you?re right that SSD?s are the > answer ? but how do you charge them more???SSDs are move expensive > than hard disks, and enterprise SSDs are stupid expensive ? and users > barely want to pay hard drive prices for their storage.??If you?ve > got the magic answer to how to charge them enough to pay for SSDs I?m > sure I?m not the only one who?d love to hear how you do it?!?! > Give every user a one million file number quota. Need to store more than one million files, then you are going to have to pay $X per extra million files. Either they cough up the money to continue using their brain dead software or they switch to less stupid software. If they complain you just say that enterprise SSD's are stupidly expensive and you are using that space up at an above average rate and so have to pay the costs. I am quite sure someone storing 1PB has to pay more than someone storing 1TB, so why should someone storing 20 million files not have to pay more than someone storing 100k files? The only difference is people are used to paying more to store extra bytes and not used to paying more for more files, but that is because most sane people don't store millions and millions of files necessitating the purchase of large amounts of expensive enterprise SSD's. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From valdis.kletnieks at vt.edu Wed Jan 17 11:35:58 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 17 Jan 2018 06:35:58 -0500 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516123547.3326.29.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> Message-ID: <72644.1516188958@turing-police.cc.vt.edu> On Tue, 16 Jan 2018 17:25:47 +0000, Jonathan Buzzard said: > User comes with problem, you investigate find problem is due to "wacky" > characters point them to the mandatory training documentation, tell > them they need to rename their files to something sane and take no > further action. Sure English is not their primary language but *they* > have chosen to study in an English speaking country so best to actually > embrace that. > I do get it, many of our users are not native English speakers as well. > Yes it's a tough policy but on the other hand pandering to them does > them no favours either. Go ahead and try to make that policy stick in Tokyo or Cairo or Berlin or any other city or country where English isn't the primary language. Seriously - this is something that IBM has to make work well across all languages if they want to make this fly globally and not just in places that are English-predominant. Though to be honest, most of the problems I've had have been with embedded blanks in filenames, newline characters in filenames (fortunately only a half dozen or so out of almost 500M files), and esc-[-random from people banging on F8 instead of 8, etc. Have the occasional backspace character creep in too. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From jonathan.buzzard at strath.ac.uk Wed Jan 17 13:48:14 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 17 Jan 2018 13:48:14 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <72644.1516188958@turing-police.cc.vt.edu> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> <72644.1516188958@turing-police.cc.vt.edu> Message-ID: <1516196894.3326.37.camel@strath.ac.uk> On Wed, 2018-01-17 at 06:35 -0500, valdis.kletnieks at vt.edu wrote: > On Tue, 16 Jan 2018 17:25:47 +0000, Jonathan Buzzard said: > > User comes with problem, you investigate find problem is due to > > "wacky" > > characters point them to the mandatory training documentation, tell > > them they need to rename their files to something sane and take no > > further action. Sure English is not their primary language but > > *they* > > have chosen to study in an English speaking country so best to > > actually > > embrace that. > > I do get it, many of our users are not native English speakers as > > well. > > Yes it's a tough policy but on the other hand pandering to them > > does > > them no favours either. > > Go ahead and try to make that policy stick in Tokyo or Cairo or > Berlin or any other city or country where English isn't the primary > language. Sure hard, but not *my* problem :-) Though to be fair missing accent's off is not that bad. Obviously if your language does not use the Roman alphabet then things are somewhat more serious. Note I am in the UK and you just don't use a ? in your file name (if your sensible) so I really do get it. > > Seriously - this is something that IBM has to make work well across > all languages if they want to make this fly globally and not just in > places that are English-predominant. Oh GPFS works fine; well unless it is ancient versions of mmbackup in which case they just cause your backup to be aborted without doing anything! It's all the other end user programs that, especially end user scripts that cause problems. So either we advise users that special characters might cause them problems and best to not use them in the first place, or wait till they have a problem and then tell them they need to rename some thousands of files. In sort it's not IBM's problem other than they are one of the major culprits who helped create this mess back in the day :-) > Though to be honest, most of the problems I've had have been with > embedded blanks in filenames, newline characters in filenames > (fortunately only a half dozen or so out of almost 500M files), and > esc-[-random from people banging on F8 instead of 8, etc.? Have the > occasional backspace character creep in too. The mind boggles how you manage to get a backspace character into a file name. A new line is bad enough, but a backspace!!! To be honest its more this sort of nonsense than none ASCII characters that cause the problems. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From valdis.kletnieks at vt.edu Wed Jan 17 14:43:05 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 17 Jan 2018 09:43:05 -0500 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516196894.3326.37.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> <72644.1516188958@turing-police.cc.vt.edu> <1516196894.3326.37.camel@strath.ac.uk> Message-ID: <8491.1516200185@turing-police.cc.vt.edu> On Wed, 17 Jan 2018 13:48:14 +0000, Jonathan Buzzard said: > The mind boggles how you manage to get a backspace character into a > file name. You can get yourself into all sorts of trouble with stty - in particular, if you're ssh'ed from one system to another, and they disagree on whether the "make the last character go away" key is Delete or Backspace. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jan 17 16:56:42 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 17 Jan 2018 16:56:42 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516123547.3326.29.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> Message-ID: <38F290EB-D2DD-40E8-910E-C0C9C2812E1E@vanderbilt.edu> Inline? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 On Jan 16, 2018, at 11:25 AM, Jonathan Buzzard > wrote: On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: [SNIP] I am quite sure someone storing 1PB has to pay more than someone storing 1TB, so why should someone storing 20 million files not have to pay more than someone storing 100k files? Because they won?t ? they?ll do something more brain dead like put a WD MyBook they bought at Costco on their desk and expect their job to copy data back and forth from it to /tmp on the compute node. We have to offer a service that users are willing to pay for ? we can?t dictate to them the way things WILL be. There?s a big difference between the way things should be and the way things actually are ? trust me, those of us in the United States know that better than most people around the world after the past year! Bigly! Buh-leave ME! :-O JAB. -------------- next part -------------- An HTML attachment was scrubbed... URL: From MDIETZ at de.ibm.com Wed Jan 17 17:08:02 2018 From: MDIETZ at de.ibm.com (Mathias Dietz) Date: Wed, 17 Jan 2018 18:08:02 +0100 Subject: [gpfsug-discuss] GPFS GA 5.0.0.0: mmces commands with inconsistent output In-Reply-To: <0CE1E9F220AF564DB0114D8539010F9B5229A68C@MBX116.d.ethz.ch> References: <0CE1E9F220AF564DB0114D8539010F9B5229A68C@MBX116.d.ethz.ch> Message-ID: Hi, let me start with a recommendation first before I explain how the cluster state is build. Starting with 4.2.1 please use the mmhealth command instead of using the mmces state/events command. The mmces state/event command will be deprecated in future releases. mmhealth node show -> show the node state for all components (incl. CES) mmhealth node show CES -> shows the CES components only. mmhealth cluster show -> show the cluster state Now to your problem: The Spectrum Scale health monitoring is done by a daemon which runs on each cluster node. This daemon is monitoring the state of all Spectrum Scale components on the local system and based on the resulting monitoring events it compiles a local system state (shown by mmhealth node show). By having a decentralized monitoring we reduce the monitoring overhead and increase resiliency against network glitches. In order to show a cluster wide state view we have to consolidate the events from all cluster nodes on a single node. The health monitoring daemon running on the cluster manager is taking the role (CSM) to receive events from all nodes through RPC calls and to compile the cluster state (shown by mmhealth cluster show) There can be cases where the (async) event forwarding to the CSM is delayed or dropped because of network delays, high system load, cluster manager failover or split brain cases. Those cases should resolve automatically after some time when event is resend. Summary: the cluster state might be temporary out of sync (eventually consistent), for getting a current state you should refer to mmhealth node show. If the problem does not resolve automatically, restarting the monitoring daemon will force a re-sync. Please open a PMR for the 5.0 issue too if the problem persist. Mit freundlichen Gr??en / Kind regards Mathias Dietz Spectrum Scale Development - Release Lead Architect (4.2.x) Spectrum Scale RAS Architect --------------------------------------------------------------------------- IBM Deutschland Am Weiher 24 65451 Kelsterbach Phone: +49 70342744105 Mobile: +49-15152801035 E-Mail: mdietz at de.ibm.com ----------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Gesch?ftsf?hrung: Dirk WittkoppSitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Ernst Heinz (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Date: 01/16/2018 06:09 PM Subject: [gpfsug-discuss] GPFS GA 5.0.0.0: mmces commands with inconsistent output Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello to all peers and gurus Since more or less two weeks we have gpfs GA 5.0.0.0 running on our testenvironment Today I?ve seen following behavior on our SpectrumScale-testcluster which slighdly surprised me Following: Checking status of the cluster on different ways [root at testnas13ces01 idsd_erh_t1]# mmces state cluster CLUSTER AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13.ethz.ch FAILED DISABLED HEALTHY DISABLED DEPEND DISABLED DEPEND FAILED [root at testnas13ces01 idsd_erh_t1]# mmces state show -a NODE AUTH BLOCK NETWORK AUTH_OBJ NFS OBJ SMB CES testnas13ces01-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY testnas13ces02-i HEALTHY DISABLED HEALTHY DISABLED HEALTHY DISABLED HEALTHY HEALTHY does anyone of you guys has an explanation therefore? Is there someone else who has seen a behavior like this? By the way we have a similar view on one of our clusters on gpfs 4.2.3.4 (open PMR: 30218.112.848) Any kind of response would be very grateful Kind regards Heinz =============================================================== Heinz Ernst ID-Systemdienste WEC C 16 Weinbergstrasse 11 CH-8092 Zurich heinz.ernst at id.ethz.ch Phone: +41 44 633 84 48 Mobile: +41 79 216 15 50 =============================================================== _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Wed Jan 17 17:31:39 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Wed, 17 Jan 2018 17:31:39 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <38F290EB-D2DD-40E8-910E-C0C9C2812E1E@vanderbilt.edu> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> <38F290EB-D2DD-40E8-910E-C0C9C2812E1E@vanderbilt.edu> Message-ID: <1516210299.3326.43.camel@strath.ac.uk> On Wed, 2018-01-17 at 16:56 +0000, Buterbaugh, Kevin L wrote: > Inline? > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu?- (615)875-9633 > > > On Jan 16, 2018, at 11:25 AM, Jonathan Buzzard > rath.ac.uk> wrote: > > > > On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: > > > > [SNIP] > > > > I am quite sure someone storing 1PB has to pay more than someone > > storing 1TB, so why should someone storing 20 million files not > > have to pay more than someone storing 100k files?? > > > ? > Because they won?t ? they?ll do something more brain dead like put a > WD MyBook they bought at Costco on their desk and expect their job to > copy data back and forth from it to /tmp on the compute node. ?We > have to offer a service that users are willing to pay for ? we can?t > dictate to them the way things WILL be. However users need to be willing to pay for something that works. As they say cheap, fast and resilient pick any two. Getting back to your scenario, their performance will suck assuming it would work in the first place and secondly it's not your problem any more :-) > There?s a big difference between the way things should be and the way > things actually are ? trust me, those of us in the United States know > that better than most people around the world after the past year! > ?Bigly! ?Buh-leave ME! ?:-O > In my experience part of the problem is system administrators *not* pushing back a bit to users when they want something unreasonable. If you don't all you do is raise a generation of spoilt children. Secondly in my experience when you do push back most users are reasonable and understanding when it's explained to them in a manner they understand that what they are trying to do is not sensible. Oh and for the record I have been right their and had to put a file quota on a user that effectively stopped them dead in their tracks because half the files on the file system where from that one user, and everyone else was complaining that performance was going down the tube. I relevant example is a research group in my university where too cheap to by a proper file server so brought a large NAS box instead from a well known vendor. Just before Christmas the NAS box popped, it's still not replaced yet. Said research group have put their hands in their pockets for a server from a tier one vendor with a 4hr response maintenance plan. Funny that :-) JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From makaplan at us.ibm.com Wed Jan 17 20:10:45 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 17 Jan 2018 17:10:45 -0300 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: References: <1516118895.3326.25.camel@strath.ac.uk> Message-ID: Yes, "special" characters in pathnames can lead to trouble... But just for the record... GPFS supports the same liberal file name policy as standard POSIX. Namely any bytestring is valid, except: / delimits directory names The \0 (zero or Null Character) byte value marks the end of the pathname. There are limits on the length of an individual file or directory name. AND there is an OS imposed limit on the total length of a pathname you can pass through the file system APIs. From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/16/2018 01:58 PM Subject: Re: [gpfsug-discuss] GPFS best practises : end user standpoint Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Jonathan, Comments / questions inline. Thanks! Kevin > On Jan 16, 2018, at 10:08 AM, Jonathan Buzzard wrote: > > On Tue, 2018-01-16 at 15:47 +0000, Carl Zetie wrote: >> Maybe this would make for a good session at a future user group >> meeting -- perhaps as an interactive session? IBM could potentially >> provide a facilitator from our Design practice. >> > > Most of it in my view is standard best practice regardless of the file > system in use. > > So in our mandatory training for the HPC, we tell our users don't use > whacked out characters in your file names and directories. Specifically > no backticks, no asterik's, no question marks, no newlines (yes > really), no slashes (either forward or backward) and for Mac users > don't start the name with a space (forces sorting to the top). We > recommend sticking to plain ASCII so no accented characters either > (harder if your native language is not English I guess but we are UK > based so...). We don't enforce that but if it causes the user problems > then they are on their own. We?re in Tennessee, so not only do we not speak English, we barely speak American ? y?all will just have to understand, bless your hearts! ;-). But seriously, like most Universities, we have a ton of users for whom English is not their ?primary? language, so dealing with ?interesting? filenames is pretty hard to avoid. And users? problems are our problems whether or not they?re our problem. > > We also strongly recommend using ISO 8601 date formats in file names to > get date sorting from a directory listing too. Surprisingly not widely > known about, but a great "life hack". > > Then it boils down to don't create zillions of files. I would love to > be able to somehow do per directory file number quota's where one could > say set a default of a few thousand. Users would then have to justify > needing a larger quota. Sure you can set a file number quota but that > does not stop them putting them all in one directory. If you?ve got (bio)medical users using your cluster I don?t see how you avoid this ? they?re using commercial apps that do this kind of stupid stuff (10?s of thousands of files in a directory and the full path to each file is longer than the contents of the files themselves!). This reminds me of way back in 2005 when we moved from an NFS server to GPFS ? I was moving users over by tarring up their home directories on the NFS server, copying the tarball over to GPFS and untarring it there ? worked great for 699 out of 700 users. But there was one user for whom the untar would fail every time I tried ? turned out that back in early versions of GPFS 2.3 IBM hadn?t considered that someone would put 6 million files in one directory! :-O > > If users really need to have zillions of files then charge them more so > you can afford to beef up your metadata disks to SSD. OK, so here?s my main question ? you?re right that SSD?s are the answer ? but how do you charge them more? SSDs are move expensive than hard disks, and enterprise SSDs are stupid expensive ? and users barely want to pay hard drive prices for their storage. If you?ve got the magic answer to how to charge them enough to pay for SSDs I?m sure I?m not the only one who?d love to hear how you do it?!?! > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fgpfsug.org-252Fmailman-252Flistinfo-252Fgpfsug-2Ddiscuss-26data-3D02-257C01-257CKevin.Buterbaugh-2540vanderbilt.edu-257Cdd3310fd309b4986f95c08d55cfb5d10-257Cba5a7f39e3be4ab3b45067fa80faecad-257C0-257C1-257C636517157039256068-26sdata-3DjZZV718gaMie92MW43qaxlDl6EQcULdk6FONrXpsP7c-253D-26reserved-3D0&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=7n72wm4bwHfrK-yGlxSHLkVEIq0FDXA7XrPI_pyQq1M&s=1cYF3dt9odnG5zCHjcxMGl9_LbVAVNFrFu1iuv5585U&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=7n72wm4bwHfrK-yGlxSHLkVEIq0FDXA7XrPI_pyQq1M&s=_puCWAGyopu4m3M7evNjbILg3LkDCmiI9vJN2IG1iBE&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Wed Jan 17 22:12:01 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 17 Jan 2018 22:12:01 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Message-ID: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn't say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to 'generic' not 'dmm'. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with 'generic' vs. 'dmm' vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Wed Jan 17 08:39:35 2018 From: john.hearns at asml.com (John Hearns) Date: Wed, 17 Jan 2018 08:39:35 +0000 Subject: [gpfsug-discuss] GPFS best practises : end user standpoint In-Reply-To: <1516123547.3326.29.camel@strath.ac.uk> References: <1516118895.3326.25.camel@strath.ac.uk> <1516123547.3326.29.camel@strath.ac.uk> Message-ID: My own thoughts on this one are that I believe when software is being developed, the developers use 'small' use cases. At the company which develops the software, the developers will probably have a desktop machine with a modest amount of RAM and disk space. Then the company might have a small to medium sized HPC cluster. I know I am stretching things a bit, but I would imagine a lot of effort goes into verifying the correct operation of software on given data sets. When the software is released to customers, either commercially or internally within a company, it is suddenly applied to larger datasets, and is applied repetitively. Hence the creation of directories filled with thousands of small files. My own example from this here is in a former job, wind tunnel data was captured as thousands of PNG files which were frames from a camera. The data was shipped back to me on a hard drive, and I was asked to store it on an HSM system with tape as the lowest tier. I knew that the PNG files had all bene processed anyway, and significant data had been extracted. The engineers wanted to keep the data 'just in case'. I knew that keeping thousands of files is bad for filesystem performance and also on a tape based system you can have the fiels stored on many tapes, so if you ever do trigger a recall you have a festival of robot tape loading. So what I did was zip up all the directories. -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jonathan Buzzard Sent: Tuesday, January 16, 2018 6:26 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPFS best practises : end user standpoint On Tue, 2018-01-16 at 16:35 +0000, Buterbaugh, Kevin L wrote: [SNIP] > > We?re in Tennessee, so not only do we not speak English, we barely > speak American ? y?all will just have to understand, bless your > hearts! ;-). > > But seriously, like most Universities, we have a ton of users for whom > English is not their ?primary? language, so dealing with ?interesting? > filenames is pretty hard to avoid. And users? problems are our > problems whether or not they?re our problem. > User comes with problem, you investigate find problem is due to "wacky" characters point them to the mandatory training documentation, tell them they need to rename their files to something sane and take no further action. Sure English is not their primary language but *they* have chosen to study in an English speaking country so best to actually embrace that. I do get it, many of our users are not native English speakers as well. Yes it's a tough policy but on the other hand pandering to them does them no favours either. [SNIP] > If you?ve got (bio)medical users using your cluster I don?t see how > you avoid this ? they?re using commercial apps that do this kind of > stupid stuff (10?s of thousands of files in a directory and the full > path to each file is longer than the contents of the files > themselves!). Well then they have justified the use; aka it's not their fault so you up the quota for them. Though they could use different less brain dead software. The idea is to force a bump in the road so the users are aware that what they are doing is considered bad practice. Most users have no idea that putting a million files in a directory is not sensible and worse that trying to access them using a GUI file manager is positively brain dead. [SNIP] > OK, so here?s my main question ? you?re right that SSD?s are the > answer ? but how do you charge them more? SSDs are move expensive > than hard disks, and enterprise SSDs are stupid expensive ? and users > barely want to pay hard drive prices for their storage. If you?ve got > the magic answer to how to charge them enough to pay for SSDs I?m sure > I?m not the only one who?d love to hear how you do it?!?! > Give every user a one million file number quota. Need to store more than one million files, then you are going to have to pay $X per extra million files. Either they cough up the money to continue using their brain dead software or they switch to less stupid software. If they complain you just say that enterprise SSD's are stupidly expensive and you are using that space up at an above average rate and so have to pay the costs. I am quite sure someone storing 1PB has to pay more than someone storing 1TB, so why should someone storing 20 million files not have to pay more than someone storing 100k files? The only difference is people are used to paying more to store extra bytes and not used to paying more for more files, but that is because most sane people don't store millions and millions of files necessitating the purchase of large amounts of expensive enterprise SSD's. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7Cf9b43f106c124ced6a4108d55d063196%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=c5B3JAJZDp3YiCN2uOzTmf%2BlsLMVRw6BsIzacQuORN8%3D&reserved=0 -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. From jjdoherty at yahoo.com Thu Jan 18 00:28:06 2018 From: jjdoherty at yahoo.com (Jim Doherty) Date: Thu, 18 Jan 2018 00:28:06 +0000 (UTC) Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> Message-ID: <1743984571.54681.1516235286972@mail.yahoo.com> Run a mmlsnsd -X ? I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN,?? not the /dev/mapper/ path.?? In our case the device is setup as dmm? [root at service5 ~]# mmlsnsd -X ?Disk name??? NSD volume ID????? Device???????? Devtype? Node name??????????????? Remarks???????? ? --------------------------------------------------------------------------------------------------- ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service5.pok.stglabs.ibm.com server node ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service6.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-4????? dmm????? service5.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-3????? dmm????? service6.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-1????? dmm????? service5.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-2????? dmm????? service6.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-3????? dmm????? service5.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-4????? dmm????? service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s? it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic ? 0972B6CD587CD8E0 /dev/sdk generic ? 0972B6CD587CD8E0 /dev/sdu generic ? 0972B6CD587CD8E0 /dev/sdah generic ? 0972B6CD587CD8E0 /dev/dm-0 dmm ? [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister wrote: Hi all, ? We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. ? The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: ? # Known disk types currently are: # #?? powerdisk? - EMC power path disk #?? vpath????? - IBM virtual path disk #?? dmm??????? - Device-Mapper Multipath (DMM) #?? dlmfdrv??? - Hitachi dlm #?? hdisk????? - AIX hard disk #?? lv???????? - AIX logical volume.? Historical usage only. #??????????????? Not allowed as a new device to mmcrnsd. #?? gpt??????? - GPFS partition on Windows disk #?? generic??? - Device having no unique failover or multipathing #??????????????? characteristic (predominantly Linux devices). #?? dasd?????? - DASD device (for Linux on z Systems) ? We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?.? This is configured in the /var/mmfs/etc/nsddevices file: ? if [[ $osName = Linux ]] then ? : # Add function to discover disks in the Linux environment. ? ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ? ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 "generic"}' ? ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi ? Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? ? Thanks in advance! -Bryan Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Thu Jan 18 15:09:22 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Thu, 18 Jan 2018 15:09:22 +0000 Subject: [gpfsug-discuss] Kerberos NFS Message-ID: Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Jan 18 15:49:15 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 15:49:15 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <1743984571.54681.1516235286972@mail.yahoo.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: <38f69158067c4b7495d525b9aba161ff@jumptrading.com> Hi Jim, Our NSDs are reporting the /dev/mapper/ devices in the `mmlsnsd -X` output, but due to the nsddevices file configuration they are all configured as ?generic? Devtype. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjdoherty at yahoo.com Thu Jan 18 16:32:58 2018 From: jjdoherty at yahoo.com (Jim Doherty) Date: Thu, 18 Jan 2018 16:32:58 +0000 (UTC) Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <38f69158067c4b7495d525b9aba161ff@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <38f69158067c4b7495d525b9aba161ff@jumptrading.com> Message-ID: <244901478.616245.1516293178108@mail.yahoo.com> What does the /var/mmfs/gen/mmsdrfs file show? [root at service5 ~]# grep SG_DISK /var/mmfs/gen/mmsdrfs | cut -f 15 -d ':' dmm dmm dmm dmm On Thursday, January 18, 2018, 10:49:32 AM EST, Bryan Banister wrote: #yiv4648160129 #yiv4648160129 -- _filtered #yiv4648160129 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv4648160129 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv4648160129 {font-family:Verdana;panose-1:2 11 6 4 3 5 4 4 2 4;} _filtered #yiv4648160129 {}#yiv4648160129 #yiv4648160129 p.yiv4648160129MsoNormal, #yiv4648160129 li.yiv4648160129MsoNormal, #yiv4648160129 div.yiv4648160129MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv4648160129 a:link, #yiv4648160129 span.yiv4648160129MsoHyperlink {color:blue;text-decoration:underline;}#yiv4648160129 a:visited, #yiv4648160129 span.yiv4648160129MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv4648160129 p.yiv4648160129msonormal, #yiv4648160129 li.yiv4648160129msonormal, #yiv4648160129 div.yiv4648160129msonormal {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 p.yiv4648160129msochpdefault, #yiv4648160129 li.yiv4648160129msochpdefault, #yiv4648160129 div.yiv4648160129msochpdefault {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 span.yiv4648160129msohyperlink {}#yiv4648160129 span.yiv4648160129msohyperlinkfollowed {}#yiv4648160129 span.yiv4648160129emailstyle17 {}#yiv4648160129 p.yiv4648160129msonormal1, #yiv4648160129 li.yiv4648160129msonormal1, #yiv4648160129 div.yiv4648160129msonormal1 {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;}#yiv4648160129 span.yiv4648160129msohyperlink1 {color:#0563C1;text-decoration:underline;}#yiv4648160129 span.yiv4648160129msohyperlinkfollowed1 {color:#954F72;text-decoration:underline;}#yiv4648160129 span.yiv4648160129emailstyle171 {color:windowtext;}#yiv4648160129 p.yiv4648160129msochpdefault1, #yiv4648160129 li.yiv4648160129msochpdefault1, #yiv4648160129 div.yiv4648160129msochpdefault1 {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv4648160129 span.yiv4648160129EmailStyle28 {color:#1F497D;}#yiv4648160129 .yiv4648160129MsoChpDefault {font-size:10.0pt;} _filtered #yiv4648160129 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv4648160129 div.yiv4648160129WordSection1 {}#yiv4648160129 Hi Jim, ? Our NSDs are reporting the /dev/mapper/ devices in the `mmlsnsd -X` output, but due to the nsddevices file configuration they are all configured as ?generic? Devtype. -Bryan ? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org]On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file ? Note: External Email ? Run a mmlsnsd -X ? I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN,?? not the /dev/mapper/ path.?? In our case the device is setup as dmm? ? [root at service5 ~]# mmlsnsd -X ?Disk name??? NSD volume ID????? Device???????? Devtype? Node name??????????????? Remarks???????? ? --------------------------------------------------------------------------------------------------- ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service5.pok.stglabs.ibm.com server node ?volume1????? 0972B6CD587CD8E0?? /dev/dm-0????? dmm????? service6.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-4????? dmm????? service5.pok.stglabs.ibm.com server node ?volume2????? 0972B6CE587CD8E4?? /dev/dm-3????? dmm????? service6.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-1????? dmm????? service5.pok.stglabs.ibm.com server node ?volume3????? 0972B6CD587CD8E7?? /dev/dm-2????? dmm????? service6.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-3????? dmm????? service5.pok.stglabs.ibm.com server node ?volume4????? 0972B6CE587CF625?? /dev/dm-4????? dmm????? service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# ? If you run an tspreparedisk -s? it will show you all of the paths. ? [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic ? 0972B6CD587CD8E0 /dev/sdk generic ? 0972B6CD587CD8E0 /dev/sdu generic ? 0972B6CD587CD8E0 /dev/sdah generic ? 0972B6CD587CD8E0 /dev/dm-0 dmm ? [root at service5 ~]# ? Jim ? ? ? Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister wrote: ? ? Hi all, ? We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. ? The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: ? # Known disk types currently are: # #?? powerdisk? - EMC power path disk #?? vpath????? - IBM virtual path disk #?? dmm??????? - Device-Mapper Multipath (DMM) #?? dlmfdrv??? - Hitachi dlm #?? hdisk????? - AIX hard disk #?? lv???????? - AIX logical volume.? Historical usage only. #??????????????? Not allowed as a new device to mmcrnsd. #?? gpt??????? - GPFS partition on Windows disk #?? generic??? - Device having no unique failover or multipathing #??????????????? characteristic (predominantly Linux devices). #?? dasd?????? - DASD device (for Linux on z Systems) ? We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?.? This is configured in the /var/mmfs/etc/nsddevices file: ? if [[ $osName = Linux ]] then ? : # Add function to discover disks in the Linux environment. ? ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ? ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 "generic"}' ? ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi ? Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? ? Thanks in advance! -Bryan ? Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Jan 18 16:35:23 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 16:35:23 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <244901478.616245.1516293178108@mail.yahoo.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <38f69158067c4b7495d525b9aba161ff@jumptrading.com> <244901478.616245.1516293178108@mail.yahoo.com> Message-ID: <9715f09cbdc74465ac2ba7cd883c9a8e@jumptrading.com> Generic: # grep SG_DISK /var/mmfs/gen/mmsdrfs | cut -f 15 -d ':' generic generic generic generic generic Cheers, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Thursday, January 18, 2018 10:33 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ What does the /var/mmfs/gen/mmsdrfs file show? [root at service5 ~]# grep SG_DISK /var/mmfs/gen/mmsdrfs | cut -f 15 -d ':' dmm dmm dmm dmm On Thursday, January 18, 2018, 10:49:32 AM EST, Bryan Banister > wrote: Hi Jim, Our NSDs are reporting the /dev/mapper/ devices in the `mmlsnsd -X` output, but due to the nsddevices file configuration they are all configured as ?generic? Devtype. -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From JRLang at uwyo.edu Thu Jan 18 15:20:18 2018 From: JRLang at uwyo.edu (Jeffrey R. Lang) Date: Thu, 18 Jan 2018 15:20:18 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <1743984571.54681.1516235286972@mail.yahoo.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: So is there a way to change it if it?s set incorrectly? jeff From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Thu Jan 18 16:59:15 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 16:59:15 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the Devtype to the ?correct? setting, for example: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi My question is what is the correct setting? And what does GPFS do differently with each setting? Thanks, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. Lang Sent: Thursday, January 18, 2018 9:20 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ So is there a way to change it if it?s set incorrectly? jeff From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty Sent: Wednesday, January 17, 2018 6:28 PM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Run a mmlsnsd -X I suspect you will see that GPFS is using one of the /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our case the device is setup as dmm [root at service5 ~]# mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service5.pok.stglabs.ibm.com server node volume1 0972B6CD587CD8E0 /dev/dm-0 dmm service6.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-4 dmm service5.pok.stglabs.ibm.com server node volume2 0972B6CE587CD8E4 /dev/dm-3 dmm service6.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-1 dmm service5.pok.stglabs.ibm.com server node volume3 0972B6CD587CD8E7 /dev/dm-2 dmm service6.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-3 dmm service5.pok.stglabs.ibm.com server node volume4 0972B6CE587CF625 /dev/dm-4 dmm service6.pok.stglabs.ibm.com server node [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: [root at service5 ~]# If you run an tspreparedisk -s it will show you all of the paths. [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 0972B6CD587CD8E0 /dev/sda generic 0972B6CD587CD8E0 /dev/sdk generic 0972B6CD587CD8E0 /dev/sdu generic 0972B6CD587CD8E0 /dev/sdah generic 0972B6CD587CD8E0 /dev/dm-0 dmm [root at service5 ~]# Jim Jim On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: Hi all, We are reviewing some of our configurations and were not sure what to make of the NSD Device Types that GPFS uses and what, if anything, do they change about how GPFS accesses/recovers/manages/etc the underlying storage based on this setting. The documentation doesn?t say much about it other than to consult the /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this section: # Known disk types currently are: # # powerdisk - EMC power path disk # vpath - IBM virtual path disk # dmm - Device-Mapper Multipath (DMM) # dlmfdrv - Hitachi dlm # hdisk - AIX hard disk # lv - AIX logical volume. Historical usage only. # Not allowed as a new device to mmcrnsd. # gpt - GPFS partition on Windows disk # generic - Device having no unique failover or multipathing # characteristic (predominantly Linux devices). # dasd - DASD device (for Linux on z Systems) We have our storage under Linux Device-Mapper Multipath control (two device paths to all storage, active/passive) and are accessible under /dev/mapper, but the NSD types are current set to ?generic? not ?dmm?. This is configured in the /var/mmfs/etc/nsddevices file: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' fi Can somebody from IBM explain what the correct setting should be and what differences GPFS does with ?generic? vs. ?dmm? vs. others? Thanks in advance! -Bryan ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From er.a.ross at gmail.com Thu Jan 18 20:01:16 2018 From: er.a.ross at gmail.com (Eric Ross) Date: Thu, 18 Jan 2018 14:01:16 -0600 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: Bryan, While waiting on the "official" word as to what each setting does differently, I remembered there was a brief explanation of the differences (at least of dmm vs generic) in the /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit (https://github.com/finley/GPFS-Goodies) I use to use when I was at IBM. //snip dmm vs. generic is used by GPFS to prioritize internal order of searching through available disks, then later GPFS discards other disk device names that it finds that match as the same NSD device by a different path. For this reason, dmm vs. generic is an important distinction if you are not explicitly producing the entire and exclusive set of disks that GPFS should use, as output from this script (nsddevices) _and_ exiting this script with a "return 0". -Brian Finley //end snip If I read that correctly, it would indicate GPFS uses those additional labels (at least dmm vs generic) as a mechanism to determine which device files to prefer when scanning a system and finding the same NSD available via different devices (i.e. /dev/mapper/foo vs /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed device, I guess it would just ignore the /dev/sd${foo} devices when it scans them. Also, the difference only seems to matter if you're not explicitly creating a list a Brian F. indicates. If you simply generate the list and exit (via return 0), GPFS wouldn't continue scanning, and then find the associated /dev/sd${foo} devices to begin with, and therefore wouldn't need a label like dmm to prioritize it over say a generic device. - Eric On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister wrote: > Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > Devtype to the ?correct? setting, for example: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > My question is what is the correct setting? > > > > And what does GPFS do differently with each setting? > > > > Thanks, > > -Bryan > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. > Lang > Sent: Thursday, January 18, 2018 9:20 AM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ________________________________ > > So is there a way to change it if it?s set incorrectly? > > > > jeff > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty > Sent: Wednesday, January 17, 2018 6:28 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > > > Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our > case the device is setup as dmm > > > > [root at service5 ~]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > --------------------------------------------------------------------------------------------------- > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service5.pok.stglabs.ibm.com server node > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service6.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > service5.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > service6.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > service5.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > service6.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-3 dmm > service5.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-4 dmm > service6.pok.stglabs.ibm.com server node > > [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > [root at service5 ~]# > > > > If you run an tspreparedisk -s it will show you all of the paths. > > > > [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > 0972B6CD587CD8E0 /dev/sda generic > 0972B6CD587CD8E0 /dev/sdk generic > 0972B6CD587CD8E0 /dev/sdu generic > 0972B6CD587CD8E0 /dev/sdah generic > 0972B6CD587CD8E0 /dev/dm-0 dmm > [root at service5 ~]# > > > > Jim > > > > > > > > Jim > > On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: > > > > > > Hi all, > > > > We are reviewing some of our configurations and were not sure what to make > of the NSD Device Types that GPFS uses and what, if anything, do they change > about how GPFS accesses/recovers/manages/etc the underlying storage based on > this setting. > > > > The documentation doesn?t say much about it other than to consult the > /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > section: > > > > # Known disk types currently are: > > # > > # powerdisk - EMC power path disk > > # vpath - IBM virtual path disk > > # dmm - Device-Mapper Multipath (DMM) > > # dlmfdrv - Hitachi dlm > > # hdisk - AIX hard disk > > # lv - AIX logical volume. Historical usage only. > > # Not allowed as a new device to mmcrnsd. > > # gpt - GPFS partition on Windows disk > > # generic - Device having no unique failover or multipathing > > # characteristic (predominantly Linux devices). > > # dasd - DASD device (for Linux on z Systems) > > > > We have our storage under Linux Device-Mapper Multipath control (two device > paths to all storage, active/passive) and are accessible under /dev/mapper, > but the NSD types are current set to ?generic? not ?dmm?. This is > configured in the /var/mmfs/etc/nsddevices file: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > Can somebody from IBM explain what the correct setting should be and what > differences GPFS does with ?generic? vs. ?dmm? vs. others? > > > > Thanks in advance! > > -Bryan > > > > ________________________________ > > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From bbanister at jumptrading.com Thu Jan 18 20:09:09 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 20:09:09 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> Message-ID: <149af11d97db41b0a343932cd936eafc@jumptrading.com> Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross Sent: Thursday, January 18, 2018 2:01 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ------------------------------------------------- Bryan, While waiting on the "official" word as to what each setting does differently, I remembered there was a brief explanation of the differences (at least of dmm vs generic) in the /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit (https://github.com/finley/GPFS-Goodies) I use to use when I was at IBM. //snip dmm vs. generic is used by GPFS to prioritize internal order of searching through available disks, then later GPFS discards other disk device names that it finds that match as the same NSD device by a different path. For this reason, dmm vs. generic is an important distinction if you are not explicitly producing the entire and exclusive set of disks that GPFS should use, as output from this script (nsddevices) _and_ exiting this script with a "return 0". -Brian Finley //end snip If I read that correctly, it would indicate GPFS uses those additional labels (at least dmm vs generic) as a mechanism to determine which device files to prefer when scanning a system and finding the same NSD available via different devices (i.e. /dev/mapper/foo vs /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed device, I guess it would just ignore the /dev/sd${foo} devices when it scans them. Also, the difference only seems to matter if you're not explicitly creating a list a Brian F. indicates. If you simply generate the list and exit (via return 0), GPFS wouldn't continue scanning, and then find the associated /dev/sd${foo} devices to begin with, and therefore wouldn't need a label like dmm to prioritize it over say a generic device. - Eric On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister wrote: > Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > Devtype to the ?correct? setting, for example: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > My question is what is the correct setting? > > > > And what does GPFS do differently with each setting? > > > > Thanks, > > -Bryan > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. > Lang > Sent: Thursday, January 18, 2018 9:20 AM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ________________________________ > > So is there a way to change it if it?s set incorrectly? > > > > jeff > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty > Sent: Wednesday, January 17, 2018 6:28 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > > > Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our > case the device is setup as dmm > > > > [root at service5 ~]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > --------------------------------------------------------------------------------------------------- > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service5.pok.stglabs.ibm.com server node > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service6.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > service5.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > service6.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > service5.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > service6.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-3 dmm > service5.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-4 dmm > service6.pok.stglabs.ibm.com server node > > [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > [root at service5 ~]# > > > > If you run an tspreparedisk -s it will show you all of the paths. > > > > [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > 0972B6CD587CD8E0 /dev/sda generic > 0972B6CD587CD8E0 /dev/sdk generic > 0972B6CD587CD8E0 /dev/sdu generic > 0972B6CD587CD8E0 /dev/sdah generic > 0972B6CD587CD8E0 /dev/dm-0 dmm > [root at service5 ~]# > > > > Jim > > > > > > > > Jim > > On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: > > > > > > Hi all, > > > > We are reviewing some of our configurations and were not sure what to make > of the NSD Device Types that GPFS uses and what, if anything, do they change > about how GPFS accesses/recovers/manages/etc the underlying storage based on > this setting. > > > > The documentation doesn?t say much about it other than to consult the > /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > section: > > > > # Known disk types currently are: > > # > > # powerdisk - EMC power path disk > > # vpath - IBM virtual path disk > > # dmm - Device-Mapper Multipath (DMM) > > # dlmfdrv - Hitachi dlm > > # hdisk - AIX hard disk > > # lv - AIX logical volume. Historical usage only. > > # Not allowed as a new device to mmcrnsd. > > # gpt - GPFS partition on Windows disk > > # generic - Device having no unique failover or multipathing > > # characteristic (predominantly Linux devices). > > # dasd - DASD device (for Linux on z Systems) > > > > We have our storage under Linux Device-Mapper Multipath control (two device > paths to all storage, active/passive) and are accessible under /dev/mapper, > but the NSD types are current set to ?generic? not ?dmm?. This is > configured in the /var/mmfs/etc/nsddevices file: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > Can somebody from IBM explain what the correct setting should be and what > differences GPFS does with ?generic? vs. ?dmm? vs. others? > > > > Thanks in advance! > > -Bryan > > > > ________________________________ > > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From bbanister at jumptrading.com Thu Jan 18 20:12:15 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Thu, 18 Jan 2018 20:12:15 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <149af11d97db41b0a343932cd936eafc@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: I should have also stated that the iostat results also report data going through the device-mapper-multipath devices as desired, -B -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Bryan Banister Sent: Thursday, January 18, 2018 2:09 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ------------------------------------------------- Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! -Bryan -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross Sent: Thursday, January 18, 2018 2:01 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ------------------------------------------------- Bryan, While waiting on the "official" word as to what each setting does differently, I remembered there was a brief explanation of the differences (at least of dmm vs generic) in the /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit (https://github.com/finley/GPFS-Goodies) I use to use when I was at IBM. //snip dmm vs. generic is used by GPFS to prioritize internal order of searching through available disks, then later GPFS discards other disk device names that it finds that match as the same NSD device by a different path. For this reason, dmm vs. generic is an important distinction if you are not explicitly producing the entire and exclusive set of disks that GPFS should use, as output from this script (nsddevices) _and_ exiting this script with a "return 0". -Brian Finley //end snip If I read that correctly, it would indicate GPFS uses those additional labels (at least dmm vs generic) as a mechanism to determine which device files to prefer when scanning a system and finding the same NSD available via different devices (i.e. /dev/mapper/foo vs /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed device, I guess it would just ignore the /dev/sd${foo} devices when it scans them. Also, the difference only seems to matter if you're not explicitly creating a list a Brian F. indicates. If you simply generate the list and exit (via return 0), GPFS wouldn't continue scanning, and then find the associated /dev/sd${foo} devices to begin with, and therefore wouldn't need a label like dmm to prioritize it over say a generic device. - Eric On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister wrote: > Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > Devtype to the ?correct? setting, for example: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > My question is what is the correct setting? > > > > And what does GPFS do differently with each setting? > > > > Thanks, > > -Bryan > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. > Lang > Sent: Thursday, January 18, 2018 9:20 AM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ________________________________ > > So is there a way to change it if it?s set incorrectly? > > > > jeff > > > > From: gpfsug-discuss-bounces at spectrumscale.org > [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty > Sent: Wednesday, January 17, 2018 6:28 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > > > Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our > case the device is setup as dmm > > > > [root at service5 ~]# mmlsnsd -X > > Disk name NSD volume ID Device Devtype Node name > Remarks > --------------------------------------------------------------------------------------------------- > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service5.pok.stglabs.ibm.com server node > volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > service6.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > service5.pok.stglabs.ibm.com server node > volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > service6.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > service5.pok.stglabs.ibm.com server node > volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > service6.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-3 dmm > service5.pok.stglabs.ibm.com server node > volume4 0972B6CE587CF625 /dev/dm-4 dmm > service6.pok.stglabs.ibm.com server node > > [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > [root at service5 ~]# > > > > If you run an tspreparedisk -s it will show you all of the paths. > > > > [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > 0972B6CD587CD8E0 /dev/sda generic > 0972B6CD587CD8E0 /dev/sdk generic > 0972B6CD587CD8E0 /dev/sdu generic > 0972B6CD587CD8E0 /dev/sdah generic > 0972B6CD587CD8E0 /dev/dm-0 dmm > [root at service5 ~]# > > > > Jim > > > > > > > > Jim > > On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > wrote: > > > > > > Hi all, > > > > We are reviewing some of our configurations and were not sure what to make > of the NSD Device Types that GPFS uses and what, if anything, do they change > about how GPFS accesses/recovers/manages/etc the underlying storage based on > this setting. > > > > The documentation doesn?t say much about it other than to consult the > /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > section: > > > > # Known disk types currently are: > > # > > # powerdisk - EMC power path disk > > # vpath - IBM virtual path disk > > # dmm - Device-Mapper Multipath (DMM) > > # dlmfdrv - Hitachi dlm > > # hdisk - AIX hard disk > > # lv - AIX logical volume. Historical usage only. > > # Not allowed as a new device to mmcrnsd. > > # gpt - GPFS partition on Windows disk > > # generic - Device having no unique failover or multipathing > > # characteristic (predominantly Linux devices). > > # dasd - DASD device (for Linux on z Systems) > > > > We have our storage under Linux Device-Mapper Multipath control (two device > paths to all storage, active/passive) and are accessible under /dev/mapper, > but the NSD types are current set to ?generic? not ?dmm?. This is > configured in the /var/mmfs/etc/nsddevices file: > > > > if [[ $osName = Linux ]] > > then > > : # Add function to discover disks in the Linux environment. > > ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > > ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > > ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > > fi > > > > Can somebody from IBM explain what the correct setting should be and what > differences GPFS does with ?generic? vs. ?dmm? vs. others? > > > > Thanks in advance! > > -Bryan > > > > ________________________________ > > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only > and may contain proprietary, confidential or privileged information. If you > are not the intended recipient, you are hereby notified that any review, > dissemination or copying of this email is strictly prohibited, and to please > notify the sender immediately and destroy this email and any attachments. > Email transmission cannot be guaranteed to be secure or error-free. The > Company, therefore, does not make any guarantees as to the completeness or > accuracy of this email or any attachments. This email is for informational > purposes only and does not constitute a recommendation, offer, request or > solicitation of any kind to buy, sell, subscribe, redeem or perform any type > of transaction of a financial product. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. From er.a.ross at gmail.com Thu Jan 18 20:22:09 2018 From: er.a.ross at gmail.com (Eric Ross) Date: Thu, 18 Jan 2018 14:22:09 -0600 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: <149af11d97db41b0a343932cd936eafc@jumptrading.com> References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: I would say that's what it *appears* to be doing. The sys admin in me would still want some "official" word from IBM, as I wouldn't my assumption to be the cause of a meltdown at 16.50 on a Friday when everyone is heading out for the weekend ;) Cheers, -Eric On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister wrote: > Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross > Sent: Thursday, January 18, 2018 2:01 PM > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file > > Note: External Email > ------------------------------------------------- > > Bryan, > > While waiting on the "official" word as to what each setting does > differently, I remembered there was a brief explanation of the > differences (at least of dmm vs generic) in the > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit > (https://github.com/finley/GPFS-Goodies) I use to use when I was at > IBM. > > //snip > > dmm vs. generic is used by GPFS to prioritize internal order of > searching through available disks, then later GPFS discards other > disk device names that it finds that match as the same NSD device > by a different path. For this reason, dmm vs. generic is an > important distinction if you are not explicitly producing the > entire and exclusive set of disks that GPFS should use, as output > from this script (nsddevices) _and_ exiting this script with a > "return 0". -Brian Finley > > //end snip > > If I read that correctly, it would indicate GPFS uses those additional > labels (at least dmm vs generic) as a mechanism to determine which > device files to prefer when scanning a system and finding the same NSD > available via different devices (i.e. /dev/mapper/foo vs > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed > device, I guess it would just ignore the /dev/sd${foo} devices when it > scans them. Also, the difference only seems to matter if you're not > explicitly creating a list a Brian F. indicates. If you simply > generate the list and exit (via return 0), GPFS wouldn't continue > scanning, and then find the associated /dev/sd${foo} devices to begin > with, and therefore wouldn't need a label like dmm to prioritize it > over say a generic device. > > > - Eric > > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister > wrote: >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the >> Devtype to the ?correct? setting, for example: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> My question is what is the correct setting? >> >> >> >> And what does GPFS do differently with each setting? >> >> >> >> Thanks, >> >> -Bryan >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. >> Lang >> Sent: Thursday, January 18, 2018 9:20 AM >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> Note: External Email >> >> ________________________________ >> >> So is there a way to change it if it?s set incorrectly? >> >> >> >> jeff >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty >> Sent: Wednesday, January 17, 2018 6:28 PM >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> >> >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our >> case the device is setup as dmm >> >> >> >> [root at service5 ~]# mmlsnsd -X >> >> Disk name NSD volume ID Device Devtype Node name >> Remarks >> --------------------------------------------------------------------------------------------------- >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service5.pok.stglabs.ibm.com server node >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service6.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm >> service5.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm >> service6.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm >> service5.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm >> service6.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-3 dmm >> service5.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-4 dmm >> service6.pok.stglabs.ibm.com server node >> >> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: >> [root at service5 ~]# >> >> >> >> If you run an tspreparedisk -s it will show you all of the paths. >> >> >> >> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 >> 0972B6CD587CD8E0 /dev/sda generic >> 0972B6CD587CD8E0 /dev/sdk generic >> 0972B6CD587CD8E0 /dev/sdu generic >> 0972B6CD587CD8E0 /dev/sdah generic >> 0972B6CD587CD8E0 /dev/dm-0 dmm >> [root at service5 ~]# >> >> >> >> Jim >> >> >> >> >> >> >> >> Jim >> >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister >> wrote: >> >> >> >> >> >> Hi all, >> >> >> >> We are reviewing some of our configurations and were not sure what to make >> of the NSD Device Types that GPFS uses and what, if anything, do they change >> about how GPFS accesses/recovers/manages/etc the underlying storage based on >> this setting. >> >> >> >> The documentation doesn?t say much about it other than to consult the >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this >> section: >> >> >> >> # Known disk types currently are: >> >> # >> >> # powerdisk - EMC power path disk >> >> # vpath - IBM virtual path disk >> >> # dmm - Device-Mapper Multipath (DMM) >> >> # dlmfdrv - Hitachi dlm >> >> # hdisk - AIX hard disk >> >> # lv - AIX logical volume. Historical usage only. >> >> # Not allowed as a new device to mmcrnsd. >> >> # gpt - GPFS partition on Windows disk >> >> # generic - Device having no unique failover or multipathing >> >> # characteristic (predominantly Linux devices). >> >> # dasd - DASD device (for Linux on z Systems) >> >> >> >> We have our storage under Linux Device-Mapper Multipath control (two device >> paths to all storage, active/passive) and are accessible under /dev/mapper, >> but the NSD types are current set to ?generic? not ?dmm?. This is >> configured in the /var/mmfs/etc/nsddevices file: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> Can somebody from IBM explain what the correct setting should be and what >> differences GPFS does with ?generic? vs. ?dmm? vs. others? >> >> >> >> Thanks in advance! >> >> -Bryan >> >> >> >> ________________________________ >> >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ________________________________ >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From yguvvala at cambridgecomputer.com Thu Jan 18 22:10:35 2018 From: yguvvala at cambridgecomputer.com (Yugendra Guvvala) Date: Thu, 18 Jan 2018 17:10:35 -0500 Subject: [gpfsug-discuss] Excluding Filesets on snapshots Message-ID: Hi, We have 5 dependent Filesets linked to root fileset (mount point) and we are creating a new independent Fileset which will be linked to root fileset (mount point). We want to take snapshots of the dependent filesets and independent fileset on different periodic intervals. Is there a way to exclude independent fileset and take snapshot of all dependent filesets ? -- Thanks, Yugi -------------- next part -------------- An HTML attachment was scrubbed... URL: From rohwedder at de.ibm.com Fri Jan 19 08:07:17 2018 From: rohwedder at de.ibm.com (Markus Rohwedder) Date: Fri, 19 Jan 2018 09:07:17 +0100 Subject: [gpfsug-discuss] Excluding Filesets on snapshots In-Reply-To: References: Message-ID: Hello Yugendra, Snapshots are taken per Inode space. Both the root fileset and the independent fileset you created own separate inode spaces, even if the independent fileset is nested in the root fileset. Taking a snapshot of the root fileset should exclude the independent fileset. e.g: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.03.52' -j 'root' Taking a filesystem snapshot however would include all inode spaces of the file system and would therefore include the independent fileset. e.g.: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.04.47' Mit freundlichen Gr??en / Kind regards Dr. Markus Rohwedder Spectrum Scale GUI Development Phone: +49 7034 6430190 IBM Deutschland E-Mail: rohwedder at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Martina K?deritz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Yugendra Guvvala To: gpfsug main discussion list Date: 01/18/2018 11:11 PM Subject: [gpfsug-discuss] Excluding Filesets on snapshots Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, We have 5 dependent Filesets linked to root fileset (mount point) and we are creating a new independent Fileset which will be linked to root fileset (mount point). We want to take snapshots of the dependent filesets and independent fileset on different periodic intervals. Is there a way to exclude independent fileset and take snapshot of all dependent filesets ? -- Thanks, Yugi _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=vZOWTPN66Z-dgh-Xs1VV4BI9dhIVi3BUFi7Zt1gPE2E&s=3HAnqhprHHsQe1cvwJIYK4cFJwG7BcNK6hKxXaDQ7sA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14162451.gif Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From jonbernard at gmail.com Fri Jan 19 15:57:11 2018 From: jonbernard at gmail.com (Jon Bernard) Date: Fri, 19 Jan 2018 10:57:11 -0500 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: A few years ago, Yuri noted that "dmm" has a special meaning to GPFS code. SCSI3 PR code keys off this disk type. https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014924464#77777777-0000-0000-0000-000014924464 https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#scsi3 Jon On Thu, Jan 18, 2018 at 3:22 PM, Eric Ross wrote: > I would say that's what it *appears* to be doing. The sys admin in me > would still want some "official" word from IBM, as I wouldn't my > assumption to be the cause of a meltdown at 16.50 on a Friday when > everyone is heading out for the weekend ;) > > Cheers, > -Eric > > On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister > wrote: > > Great! So this is just for selecting devices and according to my > `mmlsnsd -X` output it's using the correct ones, so I can probably return > to ignoring this parameter! > > -Bryan > > > > -----Original Message----- > > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss- > bounces at spectrumscale.org] On Behalf Of Eric Ross > > Sent: Thursday, January 18, 2018 2:01 PM > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ------------------------------------------------- > > > > Bryan, > > > > While waiting on the "official" word as to what each setting does > > differently, I remembered there was a brief explanation of the > > differences (at least of dmm vs generic) in the > > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit > > (https://github.com/finley/GPFS-Goodies) I use to use when I was at > > IBM. > > > > //snip > > > > dmm vs. generic is used by GPFS to prioritize internal order of > > searching through available disks, then later GPFS discards other > > disk device names that it finds that match as the same NSD device > > by a different path. For this reason, dmm vs. generic is an > > important distinction if you are not explicitly producing the > > entire and exclusive set of disks that GPFS should use, as output > > from this script (nsddevices) _and_ exiting this script with a > > "return 0". -Brian Finley > > > > //end snip > > > > If I read that correctly, it would indicate GPFS uses those additional > > labels (at least dmm vs generic) as a mechanism to determine which > > device files to prefer when scanning a system and finding the same NSD > > available via different devices (i.e. /dev/mapper/foo vs > > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed > > device, I guess it would just ignore the /dev/sd${foo} devices when it > > scans them. Also, the difference only seems to matter if you're not > > explicitly creating a list a Brian F. indicates. If you simply > > generate the list and exit (via return 0), GPFS wouldn't continue > > scanning, and then find the associated /dev/sd${foo} devices to begin > > with, and therefore wouldn't need a label like dmm to prioritize it > > over say a generic device. > > > > > > - Eric > > > > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister > > wrote: > >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > >> Devtype to the ?correct? setting, for example: > >> > >> > >> > >> if [[ $osName = Linux ]] > >> > >> then > >> > >> : # Add function to discover disks in the Linux environment. > >> > >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > >> > >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > >> > >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > >> > >> fi > >> > >> > >> > >> My question is what is the correct setting? > >> > >> > >> > >> And what does GPFS do differently with each setting? > >> > >> > >> > >> Thanks, > >> > >> -Bryan > >> > >> > >> > >> From: gpfsug-discuss-bounces at spectrumscale.org > >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey > R. > >> Lang > >> Sent: Thursday, January 18, 2018 9:20 AM > >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > >> nsddevices file > >> > >> > >> > >> Note: External Email > >> > >> ________________________________ > >> > >> So is there a way to change it if it?s set incorrectly? > >> > >> > >> > >> jeff > >> > >> > >> > >> From: gpfsug-discuss-bounces at spectrumscale.org > >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim > Doherty > >> Sent: Wednesday, January 17, 2018 6:28 PM > >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > >> nsddevices file > >> > >> > >> > >> > >> > >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In > our > >> case the device is setup as dmm > >> > >> > >> > >> [root at service5 ~]# mmlsnsd -X > >> > >> Disk name NSD volume ID Device Devtype Node name > >> Remarks > >> ------------------------------------------------------------ > --------------------------------------- > >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume4 0972B6CE587CF625 /dev/dm-3 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume4 0972B6CE587CF625 /dev/dm-4 dmm > >> service6.pok.stglabs.ibm.com server node > >> > >> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata: > 0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6. > pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system: > service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > >> [root at service5 ~]# > >> > >> > >> > >> If you run an tspreparedisk -s it will show you all of the paths. > >> > >> > >> > >> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > >> 0972B6CD587CD8E0 /dev/sda generic > >> 0972B6CD587CD8E0 /dev/sdk generic > >> 0972B6CD587CD8E0 /dev/sdu generic > >> 0972B6CD587CD8E0 /dev/sdah generic > >> 0972B6CD587CD8E0 /dev/dm-0 dmm > >> [root at service5 ~]# > >> > >> > >> > >> Jim > >> > >> > >> > >> > >> > >> > >> > >> Jim > >> > >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > >> wrote: > >> > >> > >> > >> > >> > >> Hi all, > >> > >> > >> > >> We are reviewing some of our configurations and were not sure what to > make > >> of the NSD Device Types that GPFS uses and what, if anything, do they > change > >> about how GPFS accesses/recovers/manages/etc the underlying storage > based on > >> this setting. > >> > >> > >> > >> The documentation doesn?t say much about it other than to consult the > >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > >> section: > >> > >> > >> > >> # Known disk types currently are: > >> > >> # > >> > >> # powerdisk - EMC power path disk > >> > >> # vpath - IBM virtual path disk > >> > >> # dmm - Device-Mapper Multipath (DMM) > >> > >> # dlmfdrv - Hitachi dlm > >> > >> # hdisk - AIX hard disk > >> > >> # lv - AIX logical volume. Historical usage only. > >> > >> # Not allowed as a new device to mmcrnsd. > >> > >> # gpt - GPFS partition on Windows disk > >> > >> # generic - Device having no unique failover or multipathing > >> > >> # characteristic (predominantly Linux devices). > >> > >> # dasd - DASD device (for Linux on z Systems) > >> > >> > >> > >> We have our storage under Linux Device-Mapper Multipath control (two > device > >> paths to all storage, active/passive) and are accessible under > /dev/mapper, > >> but the NSD types are current set to ?generic? not ?dmm?. This is > >> configured in the /var/mmfs/etc/nsddevices file: > >> > >> > >> > >> if [[ $osName = Linux ]] > >> > >> then > >> > >> : # Add function to discover disks in the Linux environment. > >> > >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > >> > >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > >> > >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > >> > >> fi > >> > >> > >> > >> Can somebody from IBM explain what the correct setting should be and > what > >> differences GPFS does with ?generic? vs. ?dmm? vs. others? > >> > >> > >> > >> Thanks in advance! > >> > >> -Bryan > >> > >> > >> > >> ________________________________ > >> > >> > >> Note: This email is for the confidential use of the named addressee(s) > only > >> and may contain proprietary, confidential or privileged information. If > you > >> are not the intended recipient, you are hereby notified that any review, > >> dissemination or copying of this email is strictly prohibited, and to > please > >> notify the sender immediately and destroy this email and any > attachments. > >> Email transmission cannot be guaranteed to be secure or error-free. The > >> Company, therefore, does not make any guarantees as to the completeness > or > >> accuracy of this email or any attachments. This email is for > informational > >> purposes only and does not constitute a recommendation, offer, request > or > >> solicitation of any kind to buy, sell, subscribe, redeem or perform any > type > >> of transaction of a financial product. > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> > >> ________________________________ > >> > >> Note: This email is for the confidential use of the named addressee(s) > only > >> and may contain proprietary, confidential or privileged information. If > you > >> are not the intended recipient, you are hereby notified that any review, > >> dissemination or copying of this email is strictly prohibited, and to > please > >> notify the sender immediately and destroy this email and any > attachments. > >> Email transmission cannot be guaranteed to be secure or error-free. The > >> Company, therefore, does not make any guarantees as to the completeness > or > >> accuracy of this email or any attachments. This email is for > informational > >> purposes only and does not constitute a recommendation, offer, request > or > >> solicitation of any kind to buy, sell, subscribe, redeem or perform any > type > >> of transaction of a financial product. > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ________________________________ > > > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Fri Jan 19 16:10:14 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Fri, 19 Jan 2018 16:10:14 +0000 Subject: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file In-Reply-To: References: <1bc70e60d4414116aa455d113bbed714@jumptrading.com> <1743984571.54681.1516235286972@mail.yahoo.com> <149af11d97db41b0a343932cd936eafc@jumptrading.com> Message-ID: <1b8e86ededfb41de966d23df47b3439f@jumptrading.com> I was thinking that (SCSI3 Persistent Reserve) might be used with dmm after I saw something related to IBM RDAC software in this posting by Scott Fadden: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+%28GPFS%29/page/Device+Naming+and+Discovery+in+GPFS Thanks Jon, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jon Bernard Sent: Friday, January 19, 2018 9:57 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file Note: External Email ________________________________ A few years ago, Yuri noted that "dmm" has a special meaning to GPFS code. SCSI3 PR code keys off this disk type. https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014924464#77777777-0000-0000-0000-000014924464 https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#scsi3 Jon On Thu, Jan 18, 2018 at 3:22 PM, Eric Ross > wrote: I would say that's what it *appears* to be doing. The sys admin in me would still want some "official" word from IBM, as I wouldn't my assumption to be the cause of a meltdown at 16.50 on a Friday when everyone is heading out for the weekend ;) Cheers, -Eric On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister > wrote: > Great! So this is just for selecting devices and according to my `mmlsnsd -X` output it's using the correct ones, so I can probably return to ignoring this parameter! > -Bryan > > -----Original Message----- > From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Eric Ross > Sent: Thursday, January 18, 2018 2:01 PM > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, nsddevices file > > Note: External Email > ------------------------------------------------- > > Bryan, > > While waiting on the "official" word as to what each setting does > differently, I remembered there was a brief explanation of the > differences (at least of dmm vs generic) in the > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit > (https://github.com/finley/GPFS-Goodies) I use to use when I was at > IBM. > > //snip > > dmm vs. generic is used by GPFS to prioritize internal order of > searching through available disks, then later GPFS discards other > disk device names that it finds that match as the same NSD device > by a different path. For this reason, dmm vs. generic is an > important distinction if you are not explicitly producing the > entire and exclusive set of disks that GPFS should use, as output > from this script (nsddevices) _and_ exiting this script with a > "return 0". -Brian Finley > > //end snip > > If I read that correctly, it would indicate GPFS uses those additional > labels (at least dmm vs generic) as a mechanism to determine which > device files to prefer when scanning a system and finding the same NSD > available via different devices (i.e. /dev/mapper/foo vs > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed > device, I guess it would just ignore the /dev/sd${foo} devices when it > scans them. Also, the difference only seems to matter if you're not > explicitly creating a list a Brian F. indicates. If you simply > generate the list and exit (via return 0), GPFS wouldn't continue > scanning, and then find the associated /dev/sd${foo} devices to begin > with, and therefore wouldn't need a label like dmm to prioritize it > over say a generic device. > > > - Eric > > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister > > wrote: >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the >> Devtype to the ?correct? setting, for example: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> My question is what is the correct setting? >> >> >> >> And what does GPFS do differently with each setting? >> >> >> >> Thanks, >> >> -Bryan >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jeffrey R. >> Lang >> Sent: Thursday, January 18, 2018 9:20 AM >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> Note: External Email >> >> ________________________________ >> >> So is there a way to change it if it?s set incorrectly? >> >> >> >> jeff >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org >> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jim Doherty >> Sent: Wednesday, January 17, 2018 6:28 PM >> To: gpfsug main discussion list > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, >> nsddevices file >> >> >> >> >> >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In our >> case the device is setup as dmm >> >> >> >> [root at service5 ~]# mmlsnsd -X >> >> Disk name NSD volume ID Device Devtype Node name >> Remarks >> --------------------------------------------------------------------------------------------------- >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service5.pok.stglabs.ibm.com server node >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm >> service6.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm >> service5.pok.stglabs.ibm.com server node >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm >> service6.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm >> service5.pok.stglabs.ibm.com server node >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm >> service6.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-3 dmm >> service5.pok.stglabs.ibm.com server node >> volume4 0972B6CE587CF625 /dev/dm-4 dmm >> service6.pok.stglabs.ibm.com server node >> >> [root at service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata:0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system:service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: >> [root at service5 ~]# >> >> >> >> If you run an tspreparedisk -s it will show you all of the paths. >> >> >> >> [root at service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 >> 0972B6CD587CD8E0 /dev/sda generic >> 0972B6CD587CD8E0 /dev/sdk generic >> 0972B6CD587CD8E0 /dev/sdu generic >> 0972B6CD587CD8E0 /dev/sdah generic >> 0972B6CD587CD8E0 /dev/dm-0 dmm >> [root at service5 ~]# >> >> >> >> Jim >> >> >> >> >> >> >> >> Jim >> >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister >> > wrote: >> >> >> >> >> >> Hi all, >> >> >> >> We are reviewing some of our configurations and were not sure what to make >> of the NSD Device Types that GPFS uses and what, if anything, do they change >> about how GPFS accesses/recovers/manages/etc the underlying storage based on >> this setting. >> >> >> >> The documentation doesn?t say much about it other than to consult the >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this >> section: >> >> >> >> # Known disk types currently are: >> >> # >> >> # powerdisk - EMC power path disk >> >> # vpath - IBM virtual path disk >> >> # dmm - Device-Mapper Multipath (DMM) >> >> # dlmfdrv - Hitachi dlm >> >> # hdisk - AIX hard disk >> >> # lv - AIX logical volume. Historical usage only. >> >> # Not allowed as a new device to mmcrnsd. >> >> # gpt - GPFS partition on Windows disk >> >> # generic - Device having no unique failover or multipathing >> >> # characteristic (predominantly Linux devices). >> >> # dasd - DASD device (for Linux on z Systems) >> >> >> >> We have our storage under Linux Device-Mapper Multipath control (two device >> paths to all storage, active/passive) and are accessible under /dev/mapper, >> but the NSD types are current set to ?generic? not ?dmm?. This is >> configured in the /var/mmfs/etc/nsddevices file: >> >> >> >> if [[ $osName = Linux ]] >> >> then >> >> : # Add function to discover disks in the Linux environment. >> >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' >> >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' >> >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' >> >> fi >> >> >> >> Can somebody from IBM explain what the correct setting should be and what >> differences GPFS does with ?generic? vs. ?dmm? vs. others? >> >> >> >> Thanks in advance! >> >> -Bryan >> >> >> >> ________________________________ >> >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> ________________________________ >> >> Note: This email is for the confidential use of the named addressee(s) only >> and may contain proprietary, confidential or privileged information. If you >> are not the intended recipient, you are hereby notified that any review, >> dissemination or copying of this email is strictly prohibited, and to please >> notify the sender immediately and destroy this email and any attachments. >> Email transmission cannot be guaranteed to be secure or error-free. The >> Company, therefore, does not make any guarantees as to the completeness or >> accuracy of this email or any attachments. This email is for informational >> purposes only and does not constitute a recommendation, offer, request or >> solicitation of any kind to buy, sell, subscribe, redeem or perform any type >> of transaction of a financial product. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ________________________________ > > Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ewahl at osc.edu Fri Jan 19 21:38:03 2018 From: ewahl at osc.edu (Edward Wahl) Date: Fri, 19 Jan 2018 16:38:03 -0500 Subject: [gpfsug-discuss] policy ilm features? Message-ID: <20180119163803.79fddbeb@osc.edu> This one has been on my list a long time so I figured I'd ask here first before I open an apar or request an enhancement (most likely). Is there a way using the policy engine to determine the following? -metadata replication total/current -unbalanced file Looking to catch things like this that stand out on my filesystem without having to run several hundred million 'mmlsattr's. metadata replication: 1 max 2 flags: unbalanced Ed -- Ed Wahl Ohio Supercomputer Center 614-292-9302 From makaplan at us.ibm.com Sat Jan 20 16:06:58 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sat, 20 Jan 2018 13:06:58 -0300 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180119163803.79fddbeb@osc.edu> References: <20180119163803.79fddbeb@osc.edu> Message-ID: Hint. RTFineManual, particularly the Admin guide, and look for MISC_ATTRIBUTES, Regarding metadata replication, one first has to ask, which metadata is replicated when and what if anything the mmchattr -m does or changes... https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+(GPFS)/page/Configuring+GPFS+for+Reliability --marc K. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hmorales at optimizeit.co Sun Jan 21 21:19:29 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Sun, 21 Jan 2018 16:19:29 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem Message-ID: Hello, I have a GPFS cluster with two filesystems. The disks associated to one filesystem reside on an old storage and the other filesystem disks reside on a much more modern storage system. I have successfully moved data from one fs to the other but there are questions about data integrity that still need verification so the old filesystem needs somehow to be preserved. My question is: Can I remove the old filesystem LUNs association to the NSDs servers without removing spectrum scale filesystems, so that later on, if necessary, I could associate them back and the old filesystem would be operating as normal? If possible: what would be the general steps to achieve this? ----- Thank you, Harold. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Sun Jan 21 21:35:44 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sun, 21 Jan 2018 21:35:44 +0000 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: <1d625007-9b14-a39b-5b30-027cb03a383c@strath.ac.uk> On 21/01/18 21:19, Harold Morales wrote: > Hello, > > I have a GPFS cluster with two filesystems. The disks associated to one > filesystem reside on an old storage and the other filesystem disks > reside on a much more modern storage system. I have successfully moved > data from one fs to the other but there are questions about data > integrity that still need verification so the old filesystem needs > somehow to be preserved. > > My question is: Can I remove the old filesystem LUNs association to the > NSDs servers without removing spectrum scale filesystems, so that later > on, if necessary, I could associate them back and the old filesystem > would be operating as normal? If possible: what would be the general > steps to achieve this? > I would take a look at mmexportfs/mmimportfs myself to be on the safe side. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From hmorales at optimizeit.co Sun Jan 21 23:35:01 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Sun, 21 Jan 2018 18:35:01 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: <1d625007-9b14-a39b-5b30-027cb03a383c@strath.ac.uk> References: <1d625007-9b14-a39b-5b30-027cb03a383c@strath.ac.uk> Message-ID: Thank you. The filesystems belong both to the same cluster. My question is whether LUNs can be unmapped from the host even though filesystems have not been configured and when they are mapped back the filesystem is going to operate correctly. El 21/01/2018 5:35 p.m., "Jonathan Buzzard" escribi?: On 21/01/18 21:19, Harold Morales wrote: > Hello, > > I have a GPFS cluster with two filesystems. The disks associated to one > filesystem reside on an old storage and the other filesystem disks reside > on a much more modern storage system. I have successfully moved data from > one fs to the other but there are questions about data integrity that still > need verification so the old filesystem needs somehow to be preserved. > > My question is: Can I remove the old filesystem LUNs association to the > NSDs servers without removing spectrum scale filesystems, so that later on, > if necessary, I could associate them back and the old filesystem would be > operating as normal? If possible: what would be the general steps to > achieve this? > > I would take a look at mmexportfs/mmimportfs myself to be on the safe side. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From abeattie at au1.ibm.com Sun Jan 21 23:43:47 2018 From: abeattie at au1.ibm.com (Andrew Beattie) Date: Sun, 21 Jan 2018 23:43:47 +0000 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From ulmer at ulmer.org Mon Jan 22 03:41:58 2018 From: ulmer at ulmer.org (Stephen Ulmer) Date: Sun, 21 Jan 2018 22:41:58 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: Harold, The way I read your question, no one has actually answered it fully: You want to put the old file system in cold storage for forensic purposes ? exactly as it is. You want the NSDs to go away until and unless you need them in the future. QUICK AND DIRTY - YOU SHOULD NOT DO THIS: Set the old file system to NOT mount automatically. Make sure it is unmounted everywhere(!!!!!). Make sure none of those NSDs are being used as quorum tie-breakers, etc. Print your resume. Unmap the LUNs. This will leave all of the information ABOUT the filesystem in the Spectrum Scale configuration, but it won?t be available. You?ll get a bunch of errors that the NSDs are gone. Lots of them. It will make starting up Spectrum Scale take a long time while it looks for and fails to find them. That will be mostly noise. I?m not sure how you could corrupt the file system when the LUNs are no longer accessible and there can?t be any writes pending. I have done this recently to keep an old file system around after a migration (the customer required an "A/B" switch when installing new storage). This was with a very old version of GPFS that could not be upgraded. I would not do this unless the time frame to keep this configuration is very short ? I only did it because I didn?t have a choice. PROBABLY BETTER - READ THIS ONE TWICE: Take a look at mmexportfs (this was already suggested). The point of this command to to be able to carry the Spectrum Scale "configuration" for a file system AND the LUNs full of data around and plug them into a cluster later. I think a misunderstanding here led to your objection about this method: It doesn?t have to be a different cluster ? you can use the same one it came from. The advantage to this approach is that it leaves you with a more hygienic cluster ? no "missing" storage errors. The ONLY disadvantage that I can think of at the moment is that the file system configuration MIGHT have some lint removed during the import/export process. I?m not sure that *any* checking is done during the export, but I?d guess that the import involves at least some validation of the imported file. I hope. So if you think the file system *configuration* is wonky, you should call support and see if they will look at your export, or get a snap while you?ve still got the file system in your cluster. If you think that the *structure* of the file system on disk (or maybe just the copy method you?re using) might be wonky, then learn how to use mmexportfs. Note that the learning can certainly include asking questions here. When you want to look at the old file system again, you WILL have to re-map the LUNs, and use mmimportfs (and the file you made with mmexportfs). Then the file system will be part of your cluster again. STANDARD DISCLAIMERS APPLY: Your mileage may vary. Following this advice could cause nausea, vomiting, sweating, weight gain, sleeplessness, unemployment, weight loss, temporary tooth loss, redundancy, olfactory hallucinations, oozing (generalized), redundancy, uneven tire wear, excessive body oder, scoliosis, halitosis, ring worm, and intermittent acute ignorance. Good luck! :) Liberty, -- Stephen > On Jan 21, 2018, at 6:43 PM, Andrew Beattie > wrote: > > Harold, > > How big is the old file system, Spectrum Scale is going to throw a bunch of errors if you remove the luns from the old file system while attempting to keep the file system "data" on the luns. its likely to cause you all sorts of errors and potential data corruption. Its not something that I would recommend. > > Can you do a backup of the old filesystem so you can do a restore of data if you need to? > > Regards, > Andrew Beattie > Software Defined Storage - IT Specialist > Phone: 614-2133-7927 > E-mail: abeattie at au1.ibm.com > > > ----- Original message ----- > From: Harold Morales > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem > Date: Mon, Jan 22, 2018 7:20 AM > > Hello, > > I have a GPFS cluster with two filesystems. The disks associated to one filesystem reside on an old storage and the other filesystem disks reside on a much more modern storage system. I have successfully moved data from one fs to the other but there are questions about data integrity that still need verification so the old filesystem needs somehow to be preserved. > > My question is: Can I remove the old filesystem LUNs association to the NSDs servers without removing spectrum scale filesystems, so that later on, if necessary, I could associate them back and the old filesystem would be operating as normal? If possible: what would be the general steps to achieve this? > > ----- > Thank you, > > Harold. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=STXkGEO2XATS_s2pRCAAh2wXtuUgwVcx1XjUX7ELNdk&m=deVdfZ4CCXfI09tUiJGGP1c17jRhhwjx2TcB12uunoc&s=y3EUXWX1ecxWLR1HG0Ohwn9xsPKHZ6Pdodxoz44HV7A&e= > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.buzzard at strath.ac.uk Mon Jan 22 09:57:04 2018 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Mon, 22 Jan 2018 09:57:04 +0000 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: References: Message-ID: <73d242a4-ee85-a9af-22c0-5c4fe23e3367@strath.ac.uk> On 22/01/18 03:41, Stephen Ulmer wrote: > Harold, > > The way I read your question, no one has actually answered it fully: > > You want to put the old file system in cold storage for forensic > purposes ? exactly as it is. You want the NSDs to go away until and > unless you need them in the future. > Hum, I would have said I did answer it fully even if I didn't go into detail. The only in my view sensible approach is to do mmexportfs so that should you need access to the data then you can get to it by reimporting it using mmimportfs. In the interim you can unmap all the NSD's from the cluster without causing GPFS to go care in the slightest. Otherwise you are doing something that is in my view at the best fool hardy and more generally down right idiotic. Personally I would outright refuse to do it without someone else higher up signing a written disclaimer that I was not responsible should it all go pear shaped. Note that the mmexportfs takes a few seconds at most to complete, and likewise with the mmimport. I am however somewhat perplexed by the data integrity side of the issue. If you suspect the data is corrupt in the old file system then having it around for reference purposes is surely not going to help. What are you going to do mount it again to find the file is corrupt; how is that going to help? If you suspect that whatever method you used to move the files from the old file system to the new one may have introduced corruption, then I would suggest that the best way out of that is to do an rsync with the -c flag so that it takes an MD5 checksum of the files on both file systems. Once that is complete you can safely ditch the old file system completely knowing that you have recovered it as best as possible. You could probably kick a bunch of rsyncs of in parallel to speed this method up. In fact a "rsync -c" would be a gold standard of look it's absolutely the same on the old and new file systems and remove all doubts about the transfer introducing corruption. At that point if someone comes and says your transfer corrupted my files, you can with justification say they where corrupt in the old file system and I have done my best to transfer them over. Note if any user was deliberately creating MD5 collisions then they get everything they deserve in my view, and accidental collisions in MD5 are about as likely as ?. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From john.hearns at asml.com Mon Jan 22 08:29:42 2018 From: john.hearns at asml.com (John Hearns) Date: Mon, 22 Jan 2018 08:29:42 +0000 Subject: [gpfsug-discuss] policy ilm features? In-Reply-To: <20180119163803.79fddbeb@osc.edu> References: <20180119163803.79fddbeb@osc.edu> Message-ID: Ed, This is not a perfect answer. You need to look at policies for this. I have been doing something similar recently. Something like: RULE 'list_file' EXTERNAL LIST 'all-files' EXEC '/var/mmfs/etc/mmpolicyExec-list' RULE 'listall' list 'all-files' SHOW( varchar(kb_allocated) || ' ' || varchar(file_size) || ' ' || varchar(misc_attributes) || ' ' || name || ' ' || fileset_name ) WHERE REGEX(misc_attributes,'[J]') So this policy shows the kbytes allocates, file size, the miscellaneous attributes, name and fileset name For all files with miscellaneous attributes of 'J' which means 'Some data blocks might be ill replicated' -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Edward Wahl Sent: Friday, January 19, 2018 10:38 PM To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] policy ilm features? This one has been on my list a long time so I figured I'd ask here first before I open an apar or request an enhancement (most likely). Is there a way using the policy engine to determine the following? -metadata replication total/current -unbalanced file Looking to catch things like this that stand out on my filesystem without having to run several hundred million 'mmlsattr's. metadata replication: 1 max 2 flags: unbalanced Ed -- Ed Wahl Ohio Supercomputer Center 614-292-9302 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7C056e34c5a8df4d8f10fd08d55f91e73c%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=dnt7vV4TCd68l7fSJnY35eyNM%2B8pNrZElImSZeZbit8%3D&reserved=0 -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. From yguvvala at cambridgecomputer.com Mon Jan 22 21:13:15 2018 From: yguvvala at cambridgecomputer.com (Yugendra Guvvala) Date: Mon, 22 Jan 2018 16:13:15 -0500 Subject: [gpfsug-discuss] Excluding Filesets on snapshots In-Reply-To: References: Message-ID: Thank you that works, we have to specify root fileset to get a snapshot. not specifiying root:snapshotname will include all all file sets. example : mmcrsnapshot device root:snapshot_name On Fri, Jan 19, 2018 at 3:07 AM, Markus Rohwedder wrote: > Hello Yugendra, > > Snapshots are taken per Inode space. Both the root fileset and the > independent fileset you created own separate inode spaces, > even if the independent fileset is nested in the root fileset. > > Taking a snapshot of the root fileset should exclude the independent > fileset. > e.g: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.03.52' -j 'root' > > Taking a filesystem snapshot however would include all inode spaces of the > file system and would therefore include the independent fileset. > e.g.: mmcrsnapshot 'gpfs0' '@GMT-2018.01.19-08.04.47' > > > Mit freundlichen Gr??en / Kind regards > > *Dr. Markus Rohwedder* > > Spectrum Scale GUI Development > ------------------------------ > Phone: +49 7034 6430190 <+49%207034%206430190> IBM Deutschland > E-Mail: rohwedder at de.ibm.com Am Weiher 24 > 65451 Kelsterbach > Germany > ------------------------------ > IBM Deutschland Research & Development GmbH / Vorsitzender des > Aufsichtsrats: Martina K?deritz > Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, > HRB 243294 > > [image: Inactive hide details for Yugendra Guvvala ---01/18/2018 11:11:55 > PM---Hi, We have 5 dependent Filesets linked to root fileset]Yugendra > Guvvala ---01/18/2018 11:11:55 PM---Hi, We have 5 dependent Filesets linked > to root fileset (mount point) and we > > From: Yugendra Guvvala > To: gpfsug main discussion list > Date: 01/18/2018 11:11 PM > Subject: [gpfsug-discuss] Excluding Filesets on snapshots > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi, > > We have 5 dependent Filesets linked to root fileset (mount point) and we > are creating a new independent Fileset which will be linked to root fileset > (mount point). We want to take snapshots of the dependent filesets and > independent fileset on different periodic intervals. Is there a way to > exclude independent fileset and take snapshot of all dependent filesets ? > > > -- > Thanks, > Yugi > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r= > IbxtjdkPAM2Sbon4Lbbi4w&m=vZOWTPN66Z-dgh-Xs1VV4BI9dhIVi3BUFi7Zt1gPE2E&s= > 3HAnqhprHHsQe1cvwJIYK4cFJwG7BcNK6hKxXaDQ7sA&e= > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -- Thanks, *Yugendra Guvvala | HPC Technologist ** |** Cambridge Computer ** |** "Artists in Data Storage" * Direct: 781-250-3273 | Cell: 806-773-4464 | yguvvala at cambridgecomputer.com | www.cambridgecomputer.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14162451.gif Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From hmorales at optimizeit.co Tue Jan 23 04:23:47 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Mon, 22 Jan 2018 23:23:47 -0500 Subject: [gpfsug-discuss] Removing LUN from host without unconfiguring GPFS filesystem In-Reply-To: <73d242a4-ee85-a9af-22c0-5c4fe23e3367@strath.ac.uk> References: <73d242a4-ee85-a9af-22c0-5c4fe23e3367@strath.ac.uk> Message-ID: Thanks Stephen and Jonathan for pointing this important things out. Now I begin to understand the importance of mmimportfs/mmexportfs for operations like the one I discuss here. Will let you know how it goes. 2018-01-22 4:57 GMT-05:00 Jonathan Buzzard : > On 22/01/18 03:41, Stephen Ulmer wrote: > >> Harold, >> >> The way I read your question, no one has actually answered it fully: >> >> You want to put the old file system in cold storage for forensic purposes >> ? exactly as it is. You want the NSDs to go away until and unless you need >> them in the future. >> >> > Hum, I would have said I did answer it fully even if I didn't go into > detail. The only in my view sensible approach is to do mmexportfs so that > should you need access to the data then you can get to it by reimporting it > using mmimportfs. In the interim you can unmap all the NSD's from the > cluster without causing GPFS to go care in the slightest. > > Otherwise you are doing something that is in my view at the best fool > hardy and more generally down right idiotic. Personally I would outright > refuse to do it without someone else higher up signing a written disclaimer > that I was not responsible should it all go pear shaped. Note that the > mmexportfs takes a few seconds at most to complete, and likewise with the > mmimport. > > I am however somewhat perplexed by the data integrity side of the issue. > If you suspect the data is corrupt in the old file system then having it > around for reference purposes is surely not going to help. What are you > going to do mount it again to find the file is corrupt; how is that going > to help? > > If you suspect that whatever method you used to move the files from the > old file system to the new one may have introduced corruption, then I would > suggest that the best way out of that is to do an rsync with the -c flag so > that it takes an MD5 checksum of the files on both file systems. Once that > is complete you can safely ditch the old file system completely knowing > that you have recovered it as best as possible. You could probably kick a > bunch of rsyncs of in parallel to speed this method up. > > In fact a "rsync -c" would be a gold standard of look it's absolutely the > same on the old and new file systems and remove all doubts about the > transfer introducing corruption. At that point if someone comes and says > your transfer corrupted my files, you can with justification say they where > corrupt in the old file system and I have done my best to transfer them > over. Note if any user was deliberately creating MD5 collisions then they > get everything they deserve in my view, and accidental collisions in MD5 > are about as likely as ?. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chetkulk at in.ibm.com Tue Jan 23 09:30:10 2018 From: chetkulk at in.ibm.com (Chetan R Kulkarni) Date: Tue, 23 Jan 2018 15:00:10 +0530 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: Message-ID: Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Tue Jan 23 11:10:57 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Tue, 23 Jan 2018 11:10:57 +0000 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: Message-ID: Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services to halt/timeout/fail/stop/go slow/cause an earthquake or is it completely transparent? My existing config is: FILE access configuration : AD PARAMETERS VALUES ------------------------------------------------- ENABLE_NFS_KERBEROS true SERVERS server.domain USER_NAME username NETBIOS_NAME store IDMAP_ROLE master IDMAP_RANGE 10000000-299999999 IDMAP_RANGE_SIZE 1000000 UNIXMAP_DOMAINS DOMAIN(500 - 2000000) LDAPMAP_DOMAINS none OBJECT access not configured PARAMETERS VALUES Enabled services: SMB SMB is running Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Chetan R Kulkarni Sent: 23 January 2018 09:30 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. [Inactive hide details for "Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a f]"Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nf From: "Sobey, Richard A" > To: "'gpfsug-discuss at spectrumscale.org'" > Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 105 bytes Desc: image001.gif URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 23 17:16:36 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 23 Jan 2018 17:16:36 +0000 Subject: [gpfsug-discuss] Metadata only system pool Message-ID: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_johnson at brown.edu Tue Jan 23 17:23:59 2018 From: david_johnson at brown.edu (david_johnson at brown.edu) Date: Tue, 23 Jan 2018 12:23:59 -0500 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: If the new files need indirect blocks or extended attributes that don?t fit in the basic inode, additional metadata space would need to be allocated. There might be other reasons but these come to mind immediately. -- ddj Dave Johnson > On Jan 23, 2018, at 12:16 PM, Buterbaugh, Kevin L wrote: > > Hi All, > > I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: > > Inode Information > ----------------- > Number of used inodes: 218635454 > Number of free inodes: 131364674 > Number of allocated inodes: 350000128 > Maximum number of inodes: 350000128 > > I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. > > Now my system pool is almost ?full?: > > (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) > > But again, what - outside of me creating more inodes - would cause that to change?? > > Thanks? > > Kevin > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Tue Jan 23 17:25:11 2018 From: bbanister at jumptrading.com (Bryan Banister) Date: Tue, 23 Jan 2018 17:25:11 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: <00afb095d1314045963e740f624882b8@jumptrading.com> Directory entries are also stored in the system pool and existing directories must grow in size (indirect blocks) to hold additional files/dir references as new files/dirs are created. That would be my initial guess anyways. The ?pool total? output below seems to indicate that you only have 34Mb of free space? that is definitely not good, -B From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L Sent: Tuesday, January 23, 2018 11:17 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] Metadata only system pool Note: External Email ________________________________ Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 ________________________________ Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Tue Jan 23 17:25:48 2018 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 23 Jan 2018 12:25:48 -0500 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at calicolabs.com Tue Jan 23 17:27:57 2018 From: alex at calicolabs.com (Alex Chekholko) Date: Tue, 23 Jan 2018 09:27:57 -0800 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: 2.8TB seems quite high for only 350M inodes. Are you sure you only have metadata in there? On Tue, Jan 23, 2018 at 9:25 AM, Frederick Stock wrote: > One possibility is the creation/expansion of directories or allocation of > indirect blocks for large files. > > Not sure if this is the issue here but at one time inode allocation was > considered slow and so folks may have pre-allocated inodes to avoid that > overhead during file creation. To my understanding inode creation time is > not so slow that users need to pre-allocate inodes. Yes, there are likely > some applications where pre-allocating may be necessary but I expect they > would be the exception. I mention this because you have a lot of free > inodes and of course once they are allocated they cannot be de-allocated. > > Fred > __________________________________________________ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 <(720)%20430-8821> > stockf at us.ibm.com > > > > From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 01/23/2018 12:17 PM > Subject: [gpfsug-discuss] Metadata only system pool > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi All, > > I was under the (possibly false) impression that if you have a filesystem > where the system pool contains metadata only then the only thing that would > cause the amount of free space in that pool to change is the creation of > more inodes ? is that correct? In other words, given that I have a > filesystem with 130 million free (but allocated) inodes: > > Inode Information > ----------------- > Number of used inodes: 218635454 > Number of free inodes: 131364674 > Number of allocated inodes: 350000128 > Maximum number of inodes: 350000128 > > I would not expect that a user creating a few hundred or thousands of > files could cause a ?no space left on device? error (which I?ve got one > user getting). There?s plenty of free data space, BTW. > > Now my system pool is almost ?full?: > > (pool total) 2.878T 34M ( 0%) > 140.9M ( 0%) > > But again, what - outside of me creating more inodes - would cause that to > change?? > > Thanks? > > Kevin > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > *Kevin.Buterbaugh at vanderbilt.edu* - > (615)875-9633 <(615)%20875-9633> > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m= > gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_ > vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cphoffma at uoregon.edu Tue Jan 23 17:27:52 2018 From: cphoffma at uoregon.edu (Chris Hoffman) Date: Tue, 23 Jan 2018 17:27:52 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu>, Message-ID: <1516728472608.73429@uoregon.edu> If you are still running out of space when mmdf shows preallocated inodes left I'd check MaxInodes vs AllocInodes for that fileset. You can run: mmlsfileset -L Thanks, Chris? ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederick Stock Sent: Tuesday, January 23, 2018 9:25 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Metadata only system pool One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ... is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a "no space left on device" error (which I've got one user getting). There's plenty of free data space, BTW. Now my system pool is almost "full": (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks... Kevin - Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 23 17:37:51 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 23 Jan 2018 17:37:51 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: Hi All, I do have metadata replication set to two, so Alex, does that make more sense? And I had forgotten about indirect blocks for large files, which actually makes sense with the user in question ? my apologies for that ? due to a very gravely ill pet and a recovering at home from pneumonia family member I?m way more sleep deprived right now than I?d like. :-( Fred - I think you?ve already answered this ? but mmchfs can only create / allocate more inodes ? it cannot be used to shrink the number of inodes? That would make sense, and if that?s the case then I can allocate more NSDs to the system pool. Thanks? Kevin On Jan 23, 2018, at 11:27 AM, Alex Chekholko > wrote: 2.8TB seems quite high for only 350M inodes. Are you sure you only have metadata in there? On Tue, Jan 23, 2018 at 9:25 AM, Frederick Stock > wrote: One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list > Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C1607a3fe872e4241587b08d56286a746%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636523252830007825&sdata=rIFx3lzbAIH5SZtFxJsVqWMMSo%2F0LssNc4K4tZH3uQc%3D&reserved=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Tue Jan 23 18:18:43 2018 From: stockf at us.ibm.com (Frederick Stock) Date: Tue, 23 Jan 2018 13:18:43 -0500 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: You are correct about mmchfs, you can increase the inode maximum but once an inode is allocated it cannot be de-allocated in the sense that the space can be recovered. You can of course decreased the inode maximum to a value equal to the used and allocated inodes but that would not help you here. Providing more metadata space via additional NSDs seems your most expedient option to address the issue. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 01:10 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I do have metadata replication set to two, so Alex, does that make more sense? And I had forgotten about indirect blocks for large files, which actually makes sense with the user in question ? my apologies for that ? due to a very gravely ill pet and a recovering at home from pneumonia family member I?m way more sleep deprived right now than I?d like. :-( Fred - I think you?ve already answered this ? but mmchfs can only create / allocate more inodes ? it cannot be used to shrink the number of inodes? That would make sense, and if that?s the case then I can allocate more NSDs to the system pool. Thanks? Kevin On Jan 23, 2018, at 11:27 AM, Alex Chekholko wrote: 2.8TB seems quite high for only 350M inodes. Are you sure you only have metadata in there? On Tue, Jan 23, 2018 at 9:25 AM, Frederick Stock wrote: One possibility is the creation/expansion of directories or allocation of indirect blocks for large files. Not sure if this is the issue here but at one time inode allocation was considered slow and so folks may have pre-allocated inodes to avoid that overhead during file creation. To my understanding inode creation time is not so slow that users need to pre-allocate inodes. Yes, there are likely some applications where pre-allocating may be necessary but I expect they would be the exception. I mention this because you have a lot of free inodes and of course once they are allocated they cannot be de-allocated. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 12:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C1607a3fe872e4241587b08d56286a746%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636523252830007825&sdata=rIFx3lzbAIH5SZtFxJsVqWMMSo%2F0LssNc4K4tZH3uQc%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=LN14zBEOYVrP2YRk3of_f08Ok5f256m7SYf1xL0qDvU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From UWEFALKE at de.ibm.com Tue Jan 23 19:03:40 2018 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Tue, 23 Jan 2018 20:03:40 +0100 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Tue Jan 23 19:12:50 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 23 Jan 2018 16:12:50 -0300 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: If one were starting over, it might make sense to use a smaller inode size. I believe we still support 512, 1K, 2K. Tradeoff with the fact that inodes can store data and EAs. From: "Uwe Falke" To: gpfsug main discussion list Date: 01/23/2018 04:04 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jan 23 19:25:54 2018 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 23 Jan 2018 19:25:54 +0000 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> Message-ID: <9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> Hi All, This is all making sense and I appreciate everyone?s responses ? and again I apologize for not thinking about the indirect blocks. Marc - we specifically chose 4K inodes when we created this filesystem a little over a year ago so that small files could fit in the inode and therefore be stored on the metadata SSDs. This is more of a curiosity question ? is it documented somewhere how a 4K inode is used? I understand that for very small files up to 3.5K of that can be for data, but what about for large files? I.e., how much of that 4K is used for block addresses (3.5K plus whatever portion was already allocated to block addresses??) ? or what I?m really asking is, given 4K inodes and a 1M block size how big does a file have to be before it will need to use indirect blocks? Thanks again? Kevin On Jan 23, 2018, at 1:12 PM, Marc A Kaplan > wrote: If one were starting over, it might make sense to use a smaller inode size. I believe we still support 512, 1K, 2K. Tradeoff with the fact that inodes can store data and EAs. From: "Uwe Falke" > To: gpfsug main discussion list > Date: 01/23/2018 04:04 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C77fefde14ec54e04b35708d5629550bd%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636523315820548341&sdata=rbazh5e%2BxgHGvgF65VHTs9Hf4kk9EtUizsb19l5rr7U%3D&reserved=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From UWEFALKE at de.ibm.com Tue Jan 23 20:35:56 2018 From: UWEFALKE at de.ibm.com (Uwe Falke) Date: Tue, 23 Jan 2018 21:35:56 +0100 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: <9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu> <9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> Message-ID: Hi, Kevin, (reproducing mainly what I've learned from Yuri) a 512B inode can hold about 32 Disk Addresses (DAs) addressing 32 physical disk blocks (but mind: if max replication - not necessarily actual repl. -- is set to more than 1, that mans for example 16 logical blocks (R=2). A DA is 12 bytes. With a 4k inode you can expect the other 3.5kiB to contain also DAs throughout (about 298) So with R=2 (even if actual replication is r=1), a 4k inode can hold max about 330 DAs addressing 165 logical blocks The max file size w/out indirect addressing is thus 165 x blocksize (e.g. 1320MiB @8MiB) However, there could be other data structures in an inode (EAs) reducing the space available for DAs Hence, YMMV Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 08:26 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, This is all making sense and I appreciate everyone?s responses ? and again I apologize for not thinking about the indirect blocks. Marc - we specifically chose 4K inodes when we created this filesystem a little over a year ago so that small files could fit in the inode and therefore be stored on the metadata SSDs. This is more of a curiosity question ? is it documented somewhere how a 4K inode is used? I understand that for very small files up to 3.5K of that can be for data, but what about for large files? I.e., how much of that 4K is used for block addresses (3.5K plus whatever portion was already allocated to block addresses??) ? or what I?m really asking is, given 4K inodes and a 1M block size how big does a file have to be before it will need to use indirect blocks? Thanks again? Kevin On Jan 23, 2018, at 1:12 PM, Marc A Kaplan wrote: If one were starting over, it might make sense to use a smaller inode size. I believe we still support 512, 1K, 2K. Tradeoff with the fact that inodes can store data and EAs. From: "Uwe Falke" To: gpfsug main discussion list Date: 01/23/2018 04:04 PM Subject: Re: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org rough calculation (assuming 4k inodes): 350 x 10^6 x 4096 Bytes=1.434TB=1.304TiB. With replication that uses 2.877TB or 2.308TiB As already mentioned here, directory and indirect blocks come on top. Even if you could get rid of a portion of the allocated and unused inodes that metadata pool appears bit small to me. If that is a large filesystem there should be some funding to extend it. If you have such a many-but-small-files system as discussed recently in this theatre, you might still beg for more MD storage but that makes than a larger portion of the total cost (assuming data storage is on HDD and md storage on SSD) and that again reduces your chances. Mit freundlichen Gr??en / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: uwefalke at de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: Thomas Wolter, Sven Schoo? Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 01/23/2018 06:17 PM Subject: [gpfsug-discuss] Metadata only system pool Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I was under the (possibly false) impression that if you have a filesystem where the system pool contains metadata only then the only thing that would cause the amount of free space in that pool to change is the creation of more inodes ? is that correct? In other words, given that I have a filesystem with 130 million free (but allocated) inodes: Inode Information ----------------- Number of used inodes: 218635454 Number of free inodes: 131364674 Number of allocated inodes: 350000128 Maximum number of inodes: 350000128 I would not expect that a user creating a few hundred or thousands of files could cause a ?no space left on device? error (which I?ve got one user getting). There?s plenty of free data space, BTW. Now my system pool is almost ?full?: (pool total) 2.878T 34M ( 0%) 140.9M ( 0%) But again, what - outside of me creating more inodes - would cause that to change?? Thanks? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIFAw&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8WgQlUnhzFycOZf-YvqYpdUkCiyEdRvWukE-KKRuFbE&s=aCywbK-1heVHPR8Fg74z9VxkGbNfCxMdtEKIDMWVIwI&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C77fefde14ec54e04b35708d5629550bd%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636523315820548341&sdata=rbazh5e%2BxgHGvgF65VHTs9Hf4kk9EtUizsb19l5rr7U%3D&reserved=0 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Tue Jan 23 21:13:20 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 23 Jan 2018 18:13:20 -0300 Subject: [gpfsug-discuss] Metadata only system pool In-Reply-To: References: <7F2B1D5E-6449-4265-AF0E-90E2531C7CD7@vanderbilt.edu><9E0802AC-9E7F-4556-AA13-4EB70F43A10E@vanderbilt.edu> Message-ID: One way to know for sure it to do some experiments and dump some inodes with the command tsdbfs filesystem_name inode inode_number Of course improper use of tsdbfs command can destroy or corrupt your filesystems. So I take no responsibility for that. To stay safe, only use it on test filesystems you can afford to lose. (If you are root `dd if=/dev/zero of=/dev/some-disk-you-should-not-wipe` is also something you should not do!) -------------- next part -------------- An HTML attachment was scrubbed... URL: From hmorales at optimizeit.co Wed Jan 24 01:39:52 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Tue, 23 Jan 2018 20:39:52 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale Message-ID: Hello, Is there any proven strategy to replicate disks from one local storage to one storage on another site and have this as a spectrum scale backup strategy. Let me explain: Context: - site A simple three node cluster configured (AIX nodes). two failure groups. two filesystems. No quota, no ILM. - Site B the same cluster config as in site A configured (AIX nodes) same hdisk device names used as in site A for each and every NSDs - NO TCPIP connection between those two sites. Same ip addresses available locally on each one. and same non-scale based filesystems. Same scale version installed. (esentially site B is a copy from site A). My question is, can I replicate the spectrum scale disks from site A to site B storage to another (HP 3PAR) and maintain consistency between scale sites? can this be considered a valid DR configuration? People here don't want a stretch cluster (without any apparent reason, which talks very bad about them). Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Jan 24 06:22:33 2018 From: S.J.Thompson at bham.ac.uk (Simon Thompson (IT Research Support)) Date: Wed, 24 Jan 2018 06:22:33 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: How are you proposing to replicate the data between the sites? You mention no IP connection, so were you planning some other form of data shipping? I would suggest it would be a very high risk strategy to try this. Stretched cluster or ASYNC-DR are more appropriate for this type of DR solution. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of hmorales at optimizeit.co [hmorales at optimizeit.co] Sent: 24 January 2018 01:39 To: gpfsug main discussion list Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale Hello, Is there any proven strategy to replicate disks from one local storage to one storage on another site and have this as a spectrum scale backup strategy. Let me explain: Context: - site A simple three node cluster configured (AIX nodes). two failure groups. two filesystems. No quota, no ILM. - Site B the same cluster config as in site A configured (AIX nodes) same hdisk device names used as in site A for each and every NSDs - NO TCPIP connection between those two sites. Same ip addresses available locally on each one. and same non-scale based filesystems. Same scale version installed. (esentially site B is a copy from site A). My question is, can I replicate the spectrum scale disks from site A to site B storage to another (HP 3PAR) and maintain consistency between scale sites? can this be considered a valid DR configuration? People here don't want a stretch cluster (without any apparent reason, which talks very bad about them). Thanks. From hmorales at optimizeit.co Wed Jan 24 06:33:10 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Wed, 24 Jan 2018 01:33:10 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.bolinches at fi.ibm.com Wed Jan 24 07:03:46 2018 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Wed, 24 Jan 2018 07:03:46 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: Message-ID: Hi They are not even open for a routed L3 network? How they talk between DCs today then? I would really go L3 and AFM here if the sole purpose here is to have a DR -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous > On 24 Jan 2018, at 8.33, Harold Morales wrote: > > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Wed Jan 24 07:07:46 2018 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 24 Jan 2018 07:07:46 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm ? Seems to cover what you?re looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales : > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > IP connection between storage boxes but not between servers on both sites, > there is a dark fiber connecting both sites. Here they dont want to explore > the idea of a scaled-based. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alevin at gmail.com Wed Jan 24 07:30:15 2018 From: alevin at gmail.com (Alex Levin) Date: Wed, 24 Jan 2018 02:30:15 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Hi, We are using a similar type of replication. I assume the site B is the cold site prepared for DR The storage layer is EMC VMAX and the LUNs are replicated with SRDF. All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication group to ensure consistency. The cluster name, IP addresses , hostnames of the cluster nodes are different on another site - it can be a pre-configured cluster without gpfs filesystems or with another filesystem. Same names and addresses shouldn't be a problem. Additionally to the replicated LUNs/NSDs you need to deliver copy of /var/mmfs/gen/mmsdrfs file from A to B site. There is no need to replicate it in real-time, only after the change of the cluster configuration. To activate site B - present replicated LUNs to the nodes in the DR cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" Tested with multiples LUNs and filesystems on various workloads - seems to be working --Alex On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales wrote: > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > IP connection between storage boxes but not between servers on both sites, > there is a dark fiber connecting both sites. Here they dont want to explore > the idea of a scaled-based. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chetkulk at in.ibm.com Wed Jan 24 09:37:01 2018 From: chetkulk at in.ibm.com (Chetan R Kulkarni) Date: Wed, 24 Jan 2018 15:07:01 +0530 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: Message-ID: Hi, One can't enable/disable smb/nfs service if file authentication is already configured. Hence, with your already existing config; you can't enable NFS service directly. You need to re-configure file authentication (i.e. remove file auth, enable nfs service; configure file auth). May be following sequence of commands will help you. mmuserauth service remove --data-access-method file mmces service enable nfs mmces service list -a # check nfs is enabled and running mmuserauth service create --data-access-method file --type ad ..... # please complete this command as per your set up details (the way you already configured) FYI, Enabling NFS service won't affect other services. Thanks, Chetan. From: "Sobey, Richard A" To: gpfsug main discussion list Date: 01/23/2018 04:42 PM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services to halt/timeout/fail/stop/go slow/cause an earthquake or is it completely transparent? My existing config is: FILE access configuration : AD PARAMETERS VALUES ------------------------------------------------- ENABLE_NFS_KERBEROS true SERVERS server.domain USER_NAME username NETBIOS_NAME store IDMAP_ROLE master IDMAP_RANGE 10000000-299999999 IDMAP_RANGE_SIZE 1000000 UNIXMAP_DOMAINS DOMAIN(500 - 2000000) LDAPMAP_DOMAINS none OBJECT access not configured PARAMETERS VALUES Enabled services: SMB SMB is running Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Chetan R Kulkarni Sent: 23 January 2018 09:30 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. Inactive hide details for "Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a f"Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nf From: "Sobey, Richard A" To: "'gpfsug-discuss at spectrumscale.org'" Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=mn6WfAqcjuBRZT4DaqmRBJ7KKacO2ma_baqc0GPm0PU&s=7ItSqugWx9SDxGAB2Odvj6oWUoFCiCzOH7cHdvRszY4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From r.sobey at imperial.ac.uk Wed Jan 24 16:13:01 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 24 Jan 2018 16:13:01 +0000 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , Message-ID: Gosh.. Seriously? I need downtime to enable NFS? Can that change in future releases? I have major issues configuring Ad file auth and I would not be confident getting my SMB working again in a timely manner. Thanks for letting me know anyway. Get Outlook for Android ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Chetan R Kulkarni Sent: Wednesday, January 24, 2018 9:37:01 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, One can't enable/disable smb/nfs service if file authentication is already configured. Hence, with your already existing config; you can't enable NFS service directly. You need to re-configure file authentication (i.e. remove file auth, enable nfs service; configure file auth). May be following sequence of commands will help you. mmuserauth service remove --data-access-method file mmces service enable nfs mmces service list -a # check nfs is enabled and running mmuserauth service create --data-access-method file --type ad ..... # please complete this command as per your set up details (the way you already configured) FYI, Enabling NFS service won't affect other services. Thanks, Chetan. [Inactive hide details for "Sobey, Richard A" ---01/23/2018 04:42:33 PM---Many thanks Chetan. Next question, does enabling the N]"Sobey, Richard A" ---01/23/2018 04:42:33 PM---Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services From: "Sobey, Richard A" To: gpfsug main discussion list Date: 01/23/2018 04:42 PM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Many thanks Chetan. Next question, does enabling the NFS service on CES cause any existing services to halt/timeout/fail/stop/go slow/cause an earthquake or is it completely transparent? My existing config is: FILE access configuration : AD PARAMETERS VALUES ------------------------------------------------- ENABLE_NFS_KERBEROS true SERVERS server.domain USER_NAME username NETBIOS_NAME store IDMAP_ROLE master IDMAP_RANGE 10000000-299999999 IDMAP_RANGE_SIZE 1000000 UNIXMAP_DOMAINS DOMAIN(500 - 2000000) LDAPMAP_DOMAINS none OBJECT access not configured PARAMETERS VALUES Enabled services: SMB SMB is running Thanks Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Chetan R Kulkarni Sent: 23 January 2018 09:30 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Hi, https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#protoreqs Typically it should work on CentOS since we know it works on RHEL. Thanks, Chetan. [Inactive hide details for "Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a f]"Sobey, Richard A" ---01/18/2018 08:41:06 PM---Quick starter for 10: *should* I be able to create a file authentication definition using -enable-nf From: "Sobey, Richard A" > To: "'gpfsug-discuss at spectrumscale.org'" > Date: 01/18/2018 08:41 PM Subject: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Quick starter for 10: *should* I be able to create a file authentication definition using ?enable-nfs-kerberos on CentOS 7.4? Or is this strictly for use with real RHEL nodes? Using SS 4.2.3 and 5. Thanks Richard_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=srqIkj3LKkatqQqyFKDDC2PliL3RusFC6lXPDz2iT3s&s=dwpAnqcDbbmQd8IY2XJJv0CwNAOurVJqyEyq-8q6akY&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=uic-29lyJ5TCiTRi0FyznYhKJx5I7Vzu80WyYuZ4_iM&m=mn6WfAqcjuBRZT4DaqmRBJ7KKacO2ma_baqc0GPm0PU&s=7ItSqugWx9SDxGAB2Odvj6oWUoFCiCzOH7cHdvRszY4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: graycol.gif URL: From valdis.kletnieks at vt.edu Wed Jan 24 16:28:21 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 24 Jan 2018 11:28:21 -0500 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , Message-ID: <17153.1516811301@turing-police.cc.vt.edu> On Wed, 24 Jan 2018 16:13:01 +0000, "Sobey, Richard A" said: > Gosh.. Seriously? I need downtime to enable NFS? Wait till you get to the part where 'mmnfs export change /foo/whatever --nfsadd (whatever)' bounces your nfs-ganesha services on all protocol nodes - at the same time. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From valdis.kletnieks at vt.edu Wed Jan 24 16:31:26 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 24 Jan 2018 11:31:26 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: <17310.1516811486@turing-police.cc.vt.edu> On Tue, 23 Jan 2018 20:39:52 -0500, Harold Morales said: > - site A simple three node cluster configured (AIX nodes). two failure > groups. two filesystems. No quota, no ILM. > - Site B the same cluster config as in site A configured (AIX nodes) same > hdisk device names used as in site A for each and every NSDs > - NO TCPIP connection between those two sites. Same ip addresses available > locally on each one. and same non-scale based filesystems. Same scale > version installed. (esentially site B is a copy from site A). You're going to have a *really* bad time with semantic stability at site B, as while it's reading the replica from site A, it will continually be reading metadata and data blocks that it didn't write there. (Thought experiment - what happens when site B thinks there's 3 files in a directory, and a replication block from A changes that to 2 without B's knowledge?) Be ready to be dealing with filesystem issues at B - it's *always* bad karma to do writes to a filesystem that the operating system doesn't know about.... If B is a *cold* backup site, this should work just fine and you can ignore all this. But if B has the filesystem mounted while A is making changes, it *will* go pear shaped on you. > People here don't want a stretch cluster (without any apparent reason, which > talks very bad about them). For what it's worth, we're running a stretch cluster with about 95 cable miles separating the two halves. Biggest problem we have is that a network hiccup can cause the cluster to partition and down the NSD's at the remote site (yes, it's designed that way - 5 nodes at each site, and on a partition the main site has enough quorum nodes to stay up, but the remote site doesn't. So it downs the NSDs' there. Just means a 'mmchdisk start -a' once the cluster recovers all the nodes at the far end (which is a lot less painful than the alternative, which is a split-brain situation). (And yeah, we 're working on getting reliable alternate network connections. Just a bit hard to find another 10G link to light up that isn't fate-sharing *and* plays nice with the rest of our BGP routing and can be fixed to have failover fast enough for GPFS to not notice the blink. We're in the part of Virginia that doesn't have dark fiber all over the place.) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From stockf at us.ibm.com Wed Jan 24 16:36:55 2018 From: stockf at us.ibm.com (Frederick Stock) Date: Wed, 24 Jan 2018 11:36:55 -0500 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: <17153.1516811301@turing-police.cc.vt.edu> References: , <17153.1516811301@turing-police.cc.vt.edu> Message-ID: Thankfully the issue of Ganesha being restarted across the cluster when an export is changed has been addressed in the 5.0 release of Spectrum Scale. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list Date: 01/24/2018 11:34 AM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org On Wed, 24 Jan 2018 16:13:01 +0000, "Sobey, Richard A" said: > Gosh.. Seriously? I need downtime to enable NFS? Wait till you get to the part where 'mmnfs export change /foo/whatever --nfsadd (whatever)' bounces your nfs-ganesha services on all protocol nodes - at the same time. [attachment "attxn3je.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Wed Jan 24 19:16:44 2018 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Wed, 24 Jan 2018 19:16:44 +0000 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , <17153.1516811301@turing-police.cc.vt.edu>, Message-ID: I saw that yes. Maybe I will wait until I upgrade to v5 and sort out NFS then - one downtime window to rule them all. Richard Get Outlook for Android ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederick Stock Sent: Wednesday, January 24, 2018 4:36:55 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Kerberos NFS Thankfully the issue of Ganesha being restarted across the cluster when an export is changed has been addressed in the 5.0 release of Spectrum Scale. Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 stockf at us.ibm.com From: valdis.kletnieks at vt.edu To: gpfsug main discussion list Date: 01/24/2018 11:34 AM Subject: Re: [gpfsug-discuss] Kerberos NFS Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ On Wed, 24 Jan 2018 16:13:01 +0000, "Sobey, Richard A" said: > Gosh.. Seriously? I need downtime to enable NFS? Wait till you get to the part where 'mmnfs export change /foo/whatever --nfsadd (whatever)' bounces your nfs-ganesha services on all protocol nodes - at the same time. [attachment "attxn3je.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From chair at spectrumscale.org Wed Jan 24 19:31:41 2018 From: chair at spectrumscale.org (Simon Thompson (Spectrum Scale UG Chair)) Date: Wed, 24 Jan 2018 19:31:41 +0000 Subject: [gpfsug-discuss] UK April Meeting Agenda Message-ID: <7dd288e6-679e-4bd8-9803-b5f21607e92a@email.android.com> An HTML attachment was scrubbed... URL: From valdis.kletnieks at vt.edu Wed Jan 24 20:32:00 2018 From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu) Date: Wed, 24 Jan 2018 15:32:00 -0500 Subject: [gpfsug-discuss] Kerberos NFS In-Reply-To: References: , <17153.1516811301@turing-police.cc.vt.edu> Message-ID: <50389.1516825920@turing-police.cc.vt.edu> On Wed, 24 Jan 2018 11:36:55 -0500, "Frederick Stock" said: > Thankfully the issue of Ganesha being restarted across the cluster when an > export is changed has been addressed in the 5.0 release of Spectrum Scale. Thanks for the info. Now all I need is a version of LTFS/EE that supports 5.0. :) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 486 bytes Desc: not available URL: From janfrode at tanso.net Thu Jan 25 08:27:37 2018 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 25 Jan 2018 09:27:37 +0100 Subject: [gpfsug-discuss] AFM and RHEL 7.4 Message-ID: The FAQ has a note stating: 1. AFM, Asynch Disaster Recovery with AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4. Could someone please clarify this sentence ? It can't be right that none of these features are supported with RHEL 7.4, or .. ? -jf -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Thu Jan 25 09:53:06 2018 From: john.hearns at asml.com (John Hearns) Date: Thu, 25 Jan 2018 09:53:06 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Jan Frode, thankyou for that link. I have a general question regarding remote GPFS filesystems. If we have two clusters, in separate locations on separate Infiniband fabrics, we can set up a remote relationship between filesystems. As Valdis discusses, what happens if the IP link between the clusters goes down or is unstable? Can nodes in one cluster vote out nodes in the other cluster? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: Wednesday, January 24, 2018 8:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm ? Seems to cover what you?re looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales >: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Thu Jan 25 10:08:38 2018 From: YARD at il.ibm.com (Yaron Daniel) Date: Thu, 25 Jan 2018 12:08:38 +0200 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Hi You can do remote mount between GPFS clusters. https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_admmcch.htm Where is you daemon communication network ? IP ir IP Over IB ? Regards Yaron Daniel 94 Em Ha'Moshavot Rd Server, Storage and Data Services - Team Leader Petach Tiqva, 49527 Global Technology Services Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com IBM Israel From: John Hearns To: gpfsug main discussion list Date: 01/25/2018 11:53 AM Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Sent by: gpfsug-discuss-bounces at spectrumscale.org Jan Frode, thankyou for that link. I have a general question regarding remote GPFS filesystems. If we have two clusters, in separate locations on separate Infiniband fabrics, we can set up a remote relationship between filesystems. As Valdis discusses, what happens if the IP link between the clusters goes down or is unstable? Can nodes in one cluster vote out nodes in the other cluster? From: gpfsug-discuss-bounces at spectrumscale.org [ mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: Wednesday, January 24, 2018 8:08 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm ? Seems to cover what you?re looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales : Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=BMyrARi4hlfjG-EugDznaiyWSqErGF5FyFpQQ-o4ScU&s=R0w70yvIjZaXpcs4P2mGJNQYBSNlUp3aZcCNks-sveU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1851 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4376 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 5093 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4746 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4557 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 11294 bytes Desc: not available URL: From Achim.Rehor at de.ibm.com Thu Jan 25 10:20:41 2018 From: Achim.Rehor at de.ibm.com (Achim Rehor) Date: Thu, 25 Jan 2018 11:20:41 +0100 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: From john.hearns at asml.com Thu Jan 25 10:44:28 2018 From: john.hearns at asml.com (John Hearns) Date: Thu, 25 Jan 2018 10:44:28 +0000 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Achim, Thankyou for your clear reply. Yes indeed we are using AFM at the moment. From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Achim Rehor Sent: Thursday, January 25, 2018 11:21 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale John, yes, they definitely can! Nodes in a remote cluster are tob viewed just as local nodes in terms of taking part in the mechanisms of access to data. Token management will be done just as with local nodes. So if one node in cluster A recognizes a communication issue with a node in cluster B, it will let the clustermgr know, and that one then decides on whether to expel one or the other. Having a remote cluster connected relies on a stable and low latency network, just as a local cluster does. if your network is not reliable, you woudl go for AFM or other replication mechanisms (as the thread title implies;) ) Mit freundlichen Gr??en / Kind regards Achim Rehor ________________________________ Software Technical Support Specialist AIX/ Emea HPC Support [cid:image001.gif at 01D395D1.DA27C7E0] IBM Certified Advanced Technical Expert - Power Systems with AIX TSCC Software Service, Dept. 7922 Global Technology Services ________________________________ Phone: +49-7034-274-7862 IBM Deutschland E-Mail: Achim.Rehor at de.ibm.com Am Weiher 24 65451 Kelsterbach Germany ________________________________ IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsf?hrung: Martin Hartmann (Vorsitzender), Norbert Janzen, Stefan Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 WEEE-Reg.-Nr. DE 99369940 From: John Hearns > To: gpfsug main discussion list > Date: 25/01/2018 10:53 Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Jan Frode, thankyou for that link. I have a general question regarding remote GPFS filesystems. If we have two clusters, in separate locations on separate Infiniband fabrics, we can set up a remote relationship between filesystems. As Valdis discusses, what happens if the IP link between the clusters goes down or is unstable? Can nodes in one cluster vote out nodes in the other cluster? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jan-Frode Myklebust Sent: Wednesday, January 24, 2018 8:08 AM To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Have you seen https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_dr.htm? Seems to cover what you're looking for.. -jf ons. 24. jan. 2018 kl. 07:33 skrev Harold Morales >: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 7182 bytes Desc: image001.gif URL: From olaf.weiser at de.ibm.com Thu Jan 25 13:17:52 2018 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 25 Jan 2018 14:17:52 +0100 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 14941 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 7182 bytes Desc: not available URL: From hmorales at optimizeit.co Fri Jan 26 18:29:09 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Fri, 26 Jan 2018 13:29:09 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Hi Alex, This set up seems close to what I am trying to achieve. With regards to this kind of replication: any prereqs need to be met in the target environment for this to work? for example, should disk devices naming on AIX be the same as in the source environment? when importing the mmsdrfs file, how is Scale going to know which disks should it assign to the cluster? by their hdisk name alone? Thanks again, 2018-01-24 2:30 GMT-05:00 Alex Levin : > Hi, > > We are using a similar type of replication. > I assume the site B is the cold site prepared for DR > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > group to ensure consistency. > > The cluster name, IP addresses , hostnames of the cluster nodes are > different on another site - it can be a pre-configured cluster without > gpfs filesystems or with another filesystem. > Same names and addresses shouldn't be a problem. > > Additionally to the replicated LUNs/NSDs you need to deliver copy > of /var/mmfs/gen/mmsdrfs file from A to B site. > There is no need to replicate it in real-time, only after the change of > the cluster configuration. > > To activate site B - present replicated LUNs to the nodes in the DR > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > Tested with multiples LUNs and filesystems on various workloads - seems > to be working > > --Alex > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > wrote: > >> Thanks for answering. >> >> Essentially, the idea being explored is to replicate LUNs between >> identical storage hardware (HP 3PAR volumesrein) on both sites. There is an >> IP connection between storage boxes but not between servers on both sites, >> there is a dark fiber connecting both sites. Here they dont want to explore >> the idea of a scaled-based. >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gcorneau at us.ibm.com Fri Jan 26 20:21:15 2018 From: gcorneau at us.ibm.com (Glen Corneau) Date: Fri, 26 Jan 2018 14:21:15 -0600 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Scale will walk across all discovered disks upon start time and attempt to read the NSD identifiers from the disks. Once it finds them, it makes a local map file that correlates the NSD id and the hdiskX identifier. The names do not have to be the same as either the source cluster or even from node-to-node. The main thing to keep in mind is to keep the file system definitions in sync between the source and destination clusters. The "syncFSconfig" user exit is the best way to do it because it's automatic. You generally shouldn't be shuffling the mmsdrfs file between sites, that's what the "mmfsctl syncFSconfig" does for you, on a per-file system basis. GPFS+AIX customers have been using this kind of storage replication for over 10 years, it's business as usual. ------------------ Glen Corneau Power Systems Washington Systems Center gcorneau at us.ibm.com From: Harold Morales To: gpfsug main discussion list Date: 01/26/2018 12:30 PM Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum Scale Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Alex, This set up seems close to what I am trying to achieve. With regards to this kind of replication: any prereqs need to be met in the target environment for this to work? for example, should disk devices naming on AIX be the same as in the source environment? when importing the mmsdrfs file, how is Scale going to know which disks should it assign to the cluster? by their hdisk name alone? Thanks again, 2018-01-24 2:30 GMT-05:00 Alex Levin : Hi, We are using a similar type of replication. I assume the site B is the cold site prepared for DR The storage layer is EMC VMAX and the LUNs are replicated with SRDF. All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication group to ensure consistency. The cluster name, IP addresses , hostnames of the cluster nodes are different on another site - it can be a pre-configured cluster without gpfs filesystems or with another filesystem. Same names and addresses shouldn't be a problem. Additionally to the replicated LUNs/NSDs you need to deliver copy of /var/mmfs/gen/mmsdrfs file from A to B site. There is no need to replicate it in real-time, only after the change of the cluster configuration. To activate site B - present replicated LUNs to the nodes in the DR cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" Tested with multiples LUNs and filesystems on various workloads - seems to be working --Alex On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales wrote: Thanks for answering. Essentially, the idea being explored is to replicate LUNs between identical storage hardware (HP 3PAR volumesrein) on both sites. There is an IP connection between storage boxes but not between servers on both sites, there is a dark fiber connecting both sites. Here they dont want to explore the idea of a scaled-based. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=d-vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m=VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh-GeVJiVS49opBwwCFs-FKQrQ7rc&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 26117 bytes Desc: not available URL: From hmorales at optimizeit.co Fri Jan 26 20:47:03 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Fri, 26 Jan 2018 15:47:03 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Thanks for participating in the discussion. Immediately after replication I am getting the error documented below, I have moved the mmsdrfs file (upon deleting the previous filesystem definitions, because I have managed to configure exactly equal my source and target clusters). Even then, I am getting the following error: GPFS: 6027-419 Failed to read a file system descriptor. There is an input or output error. mmlsfs: 6027-1639 Command failed. Examine previous That's the same error that upon first replicating without taking any other action. I think I am missing something really important but still dont know what it is. 2018-01-26 15:21 GMT-05:00 Glen Corneau : > Scale will walk across all discovered disks upon start time and attempt to > read the NSD identifiers from the disks. Once it finds them, it makes a > local map file that correlates the NSD id and the hdiskX identifier. The > names do not have to be the same as either the source cluster or even from > node-to-node. > > The main thing to keep in mind is to keep the file system definitions in > sync between the source and destination clusters. The "syncFSconfig" user > exit is the best way to do it because it's automatic. You generally > shouldn't be shuffling the mmsdrfs file between sites, that's what the > "mmfsctl syncFSconfig" does for you, on a per-file system basis. > > GPFS+AIX customers have been using this kind of storage replication for > over 10 years, it's business as usual. > > ------------------ > Glen Corneau > Power Systems > Washington Systems Center > gcorneau at us.ibm.com > > > > > > From: Harold Morales > To: gpfsug main discussion list > Date: 01/26/2018 12:30 PM > Subject: Re: [gpfsug-discuss] storage-based replication for > Spectrum Scale > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > With regards to this kind of replication: any prereqs need to be met in > the target environment for this to work? for example, should disk devices > naming on AIX be the same as in the source environment? when importing the > mmsdrfs file, how is Scale going to know which disks should it assign to > the cluster? by their hdisk name alone? > > Thanks again, > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin <*alevin at gmail.com* > >: > Hi, > > We are using a similar type of replication. > I assume the site B is the cold site prepared for DR > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > group to ensure consistency. > > The cluster name, IP addresses , hostnames of the cluster nodes are > different on another site - it can be a pre-configured cluster without > gpfs filesystems or with another filesystem. > Same names and addresses shouldn't be a problem. > > Additionally to the replicated LUNs/NSDs you need to deliver copy > of /var/mmfs/gen/mmsdrfs file from A to B site. > There is no need to replicate it in real-time, only after the change of > the cluster configuration. > > To activate site B - present replicated LUNs to the nodes in the DR > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > Tested with multiples LUNs and filesystems on various workloads - seems > to be working > > --Alex > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales <*hmorales at optimizeit.co* > > wrote: > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > IP connection between storage boxes but not between servers on both sites, > there is a dark fiber connecting both sites. Here they dont want to explore > the idea of a scaled-based. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at *spectrumscale.org* > > *http://gpfsug.org/mailman/listinfo/gpfsug-discuss* > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at *spectrumscale.org* > > *http://gpfsug.org/mailman/listinfo/gpfsug-discuss* > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug. > org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_ > iaSHvJObTbx-siA1ZOg&r=d-vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m= > VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- > GeVJiVS49opBwwCFs-FKQrQ7rc&e= > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 26117 bytes Desc: not available URL: From sxiao at us.ibm.com Fri Jan 26 21:04:03 2018 From: sxiao at us.ibm.com (Steve Xiao) Date: Fri, 26 Jan 2018 16:04:03 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: When using this method of replication, you need to either issue "mmfsctl suspend or suspend-write" command before replication or setup a single consistency group for all LUNs. This is needed to ensure replica contain a consistent copy of GPFS data. Steve Y. Xiao gpfsug-discuss-bounces at spectrumscale.org wrote on 01/26/2018 03:21:23 PM: > From: gpfsug-discuss-request at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Date: 01/26/2018 03:21 PM > Subject: gpfsug-discuss Digest, Vol 72, Issue 69 > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: storage-based replication for Spectrum Scale (Harold Morales) > 2. Re: storage-based replication for Spectrum Scale (Glen Corneau) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 26 Jan 2018 13:29:09 -0500 > From: Harold Morales > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > Scale > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Hi Alex, This set up seems close to what I am trying to achieve. > > With regards to this kind of replication: any prereqs need to be met in the > target environment for this to work? for example, should disk devices > naming on AIX be the same as in the source environment? when importing the > mmsdrfs file, how is Scale going to know which disks should it assign to > the cluster? by their hdisk name alone? > > Thanks again, > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > > > Hi, > > > > We are using a similar type of replication. > > I assume the site B is the cold site prepared for DR > > > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > > group to ensure consistency. > > > > The cluster name, IP addresses , hostnames of the cluster nodes are > > different on another site - it can be a pre-configured cluster without > > gpfs filesystems or with another filesystem. > > Same names and addresses shouldn't be a problem. > > > > Additionally to the replicated LUNs/NSDs you need to deliver copy > > of /var/mmfs/gen/mmsdrfs file from A to B site. > > There is no need to replicate it in real-time, only after the change of > > the cluster configuration. > > > > To activate site B - present replicated LUNs to the nodes in the DR > > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > > > Tested with multiples LUNs and filesystems on various workloads - seems > > to be working > > > > --Alex > > > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > > wrote: > > > >> Thanks for answering. > >> > >> Essentially, the idea being explored is to replicate LUNs between > >> identical storage hardware (HP 3PAR volumesrein) on both sites. There is an > >> IP connection between storage boxes but not between servers on both sites, > >> there is a dark fiber connecting both sites. Here they dont want to explore > >> the idea of a scaled-based. > >> > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > >> > >> > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180126_1503995b_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=SKdMmQae8uzHNWZq3vuRTp5UVwYFeeusLAxtbaposX0&e=> > > ------------------------------ > > Message: 2 > Date: Fri, 26 Jan 2018 14:21:15 -0600 > From: "Glen Corneau" > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > Scale > Message-ID: > 006FCEEE at notes.na.collabserv.com> > > Content-Type: text/plain; charset="us-ascii" > > Scale will walk across all discovered disks upon start time and attempt to > read the NSD identifiers from the disks. Once it finds them, it makes a > local map file that correlates the NSD id and the hdiskX identifier. The > names do not have to be the same as either the source cluster or even from > node-to-node. > > The main thing to keep in mind is to keep the file system definitions in > sync between the source and destination clusters. The "syncFSconfig" user > exit is the best way to do it because it's automatic. You generally > shouldn't be shuffling the mmsdrfs file between sites, that's what the > "mmfsctl syncFSconfig" does for you, on a per-file system basis. > > GPFS+AIX customers have been using this kind of storage replication for > over 10 years, it's business as usual. > > ------------------ > Glen Corneau > Power Systems > Washington Systems Center > gcorneau at us.ibm.com > > > > > > From: Harold Morales > To: gpfsug main discussion list > Date: 01/26/2018 12:30 PM > Subject: Re: [gpfsug-discuss] storage-based replication for > Spectrum Scale > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > With regards to this kind of replication: any prereqs need to be met in > the target environment for this to work? for example, should disk devices > naming on AIX be the same as in the source environment? when importing the > mmsdrfs file, how is Scale going to know which disks should it assign to > the cluster? by their hdisk name alone? > > Thanks again, > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > Hi, > > We are using a similar type of replication. > I assume the site B is the cold site prepared for DR > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX replication > group to ensure consistency. > > The cluster name, IP addresses , hostnames of the cluster nodes are > different on another site - it can be a pre-configured cluster without > gpfs filesystems or with another filesystem. > Same names and addresses shouldn't be a problem. > > Additionally to the replicated LUNs/NSDs you need to deliver copy > of /var/mmfs/gen/mmsdrfs file from A to B site. > There is no need to replicate it in real-time, only after the change of > the cluster configuration. > > To activate site B - present replicated LUNs to the nodes in the DR > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > Tested with multiples LUNs and filesystems on various workloads - seems > to be working > > --Alex > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > wrote: > Thanks for answering. > > Essentially, the idea being explored is to replicate LUNs between > identical storage hardware (HP 3PAR volumesrein) on both sites. There is > an IP connection between storage boxes but not between servers on both > sites, there is a dark fiber connecting both sites. Here they dont want to > explore the idea of a scaled-based. > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=d- > vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m=VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- > GeVJiVS49opBwwCFs-FKQrQ7rc&e= > > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180126_e291af63_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=bYnf-7v0CxYUkGth-QaVeUQdIlG8f1Gro-hwOxok7Qw&e=> > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: not available > Type: image/jpeg > Size: 26117 bytes > Desc: not available > URL: u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180126_e291af63_attachment.jpe&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=jYdnqhQBlnpf58oxunzBcTs9XdcbeOtLDQdgnASidDA&e=> > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url? > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx- > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > End of gpfsug-discuss Digest, Vol 72, Issue 69 > ********************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alevin at gmail.com Sat Jan 27 05:02:40 2018 From: alevin at gmail.com (Alex Levin) Date: Sat, 27 Jan 2018 00:02:40 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Steve, I've read that "mmfsctl suspend or suspend-write" should be executed, but in real life it is impossible for DR scenario. We tested both cases, the graceful one when the failover to another site is planned, applications stopped and i/o suspended and the case when there was no advanced notice about the disaster in the primary site. Both worked and for the second case various loads were simulated including heavy writes and reads/writes. In disaster model as expected some data were lost (due to incomplete writes, replication latency ... ), but mmfsck was always able to repair and the applications databases located on the affected filesystem were in an acceptable state. it is possible that we were just lucky and the test case didn't cover all possible scenarios. Harold, In our case, it is Linux, not AIX, but it shouldn't matter And our DR cluster is fully configured ( different IPs, hostnames and cluster name ) and running without filesystems at all or with a differently named filesystem. Before running mmimport , make sure that all expected LUNs are visible and writable. You can verify if the device is correct by reading first blocks of the device ( for example dd if=/dev/NSD_LUN_device bs=1M count=16 | strings ) not sure where you are " moved the mmsdrfs" you don't need to move/modify mmsdrfs file on the target ( disaster recovery ) cluster Just copy the one from primary to /tmp or /var/tmp and try to run mmimportfs fs_name -i copy_of_mmsdrfs /tmp/mmsdrfs Glen, as Harold has no IP connectivity between the clusters "mmfsctl syncFSconfig" is not an option ... Thanks --Alex On Fri, Jan 26, 2018 at 4:04 PM, Steve Xiao wrote: > When using this method of replication, you need to either issue "mmfsctl > suspend or suspend-write" command before replication or setup a single > consistency group for all LUNs. This is needed to ensure replica contain > a consistent copy of GPFS data. > > Steve Y. Xiao > > gpfsug-discuss-bounces at spectrumscale.org wrote on 01/26/2018 03:21:23 PM: > > > From: gpfsug-discuss-request at spectrumscale.org > > To: gpfsug-discuss at spectrumscale.org > > Date: 01/26/2018 03:21 PM > > Subject: gpfsug-discuss Digest, Vol 72, Issue 69 > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Send gpfsug-discuss mailing list submissions to > > gpfsug-discuss at spectrumscale.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > or, via email, send a message with subject or body 'help' to > > gpfsug-discuss-request at spectrumscale.org > > > > You can reach the person managing the list at > > gpfsug-discuss-owner at spectrumscale.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of gpfsug-discuss digest..." > > > > > > Today's Topics: > > > > 1. Re: storage-based replication for Spectrum Scale (Harold Morales) > > 2. Re: storage-based replication for Spectrum Scale (Glen Corneau) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Fri, 26 Jan 2018 13:29:09 -0500 > > From: Harold Morales > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > > Scale > > Message-ID: > > > > Content-Type: text/plain; charset="utf-8" > > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > > > With regards to this kind of replication: any prereqs need to be met in > the > > target environment for this to work? for example, should disk devices > > naming on AIX be the same as in the source environment? when importing > the > > mmsdrfs file, how is Scale going to know which disks should it assign to > > the cluster? by their hdisk name alone? > > > > Thanks again, > > > > > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > > > > > Hi, > > > > > > We are using a similar type of replication. > > > I assume the site B is the cold site prepared for DR > > > > > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX > replication > > > group to ensure consistency. > > > > > > The cluster name, IP addresses , hostnames of the cluster nodes are > > > different on another site - it can be a pre-configured cluster without > > > gpfs filesystems or with another filesystem. > > > Same names and addresses shouldn't be a problem. > > > > > > Additionally to the replicated LUNs/NSDs you need to deliver copy > > > of /var/mmfs/gen/mmsdrfs file from A to B site. > > > There is no need to replicate it in real-time, only after the change of > > > the cluster configuration. > > > > > > To activate site B - present replicated LUNs to the nodes in the DR > > > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > > > > > Tested with multiples LUNs and filesystems on various workloads - > seems > > > to be working > > > > > > --Alex > > > > > > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales < > hmorales at optimizeit.co> > > > wrote: > > > > > >> Thanks for answering. > > >> > > >> Essentially, the idea being explored is to replicate LUNs between > > >> identical storage hardware (HP 3PAR volumesrein) on both sites. There > is an > > >> IP connection between storage boxes but not between servers on both > sites, > > >> there is a dark fiber connecting both sites. Here they dont want to > explore > > >> the idea of a scaled-based. > > >> > > >> > > >> _______________________________________________ > > >> gpfsug-discuss mailing list > > >> gpfsug-discuss at spectrumscale.org > > >> https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > >> > > >> > > > > > > _______________________________________________ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_ > attachments_20180126_1503995b_attachment-2D0001.html&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=SKdMmQae8uzHNWZq3vuRTp5UVwYFeeusLAxtbaposX0&e=> > > > > ------------------------------ > > > > Message: 2 > > Date: Fri, 26 Jan 2018 14:21:15 -0600 > > From: "Glen Corneau" > > To: gpfsug main discussion list > > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum > > Scale > > Message-ID: > > > 006FCEEE at notes.na.collabserv.com> > > > > Content-Type: text/plain; charset="us-ascii" > > > > Scale will walk across all discovered disks upon start time and attempt > to > > read the NSD identifiers from the disks. Once it finds them, it makes a > > local map file that correlates the NSD id and the hdiskX identifier. > The > > names do not have to be the same as either the source cluster or even > from > > node-to-node. > > > > The main thing to keep in mind is to keep the file system definitions in > > sync between the source and destination clusters. The "syncFSconfig" > user > > exit is the best way to do it because it's automatic. You generally > > shouldn't be shuffling the mmsdrfs file between sites, that's what the > > "mmfsctl syncFSconfig" does for you, on a per-file system basis. > > > > GPFS+AIX customers have been using this kind of storage replication for > > over 10 years, it's business as usual. > > > > ------------------ > > Glen Corneau > > Power Systems > > Washington Systems Center > > gcorneau at us.ibm.com > > > > > > > > > > > > From: Harold Morales > > To: gpfsug main discussion list > > Date: 01/26/2018 12:30 PM > > Subject: Re: [gpfsug-discuss] storage-based replication for > > Spectrum Scale > > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > > > > > Hi Alex, This set up seems close to what I am trying to achieve. > > > > With regards to this kind of replication: any prereqs need to be met in > > the target environment for this to work? for example, should disk > devices > > naming on AIX be the same as in the source environment? when importing > the > > mmsdrfs file, how is Scale going to know which disks should it assign to > > the cluster? by their hdisk name alone? > > > > Thanks again, > > > > > > > > 2018-01-24 2:30 GMT-05:00 Alex Levin : > > Hi, > > > > We are using a similar type of replication. > > I assume the site B is the cold site prepared for DR > > > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX > replication > > group to ensure consistency. > > > > The cluster name, IP addresses , hostnames of the cluster nodes are > > different on another site - it can be a pre-configured cluster without > > gpfs filesystems or with another filesystem. > > Same names and addresses shouldn't be a problem. > > > > Additionally to the replicated LUNs/NSDs you need to deliver copy > > of /var/mmfs/gen/mmsdrfs file from A to B site. > > There is no need to replicate it in real-time, only after the change of > > the cluster configuration. > > > > To activate site B - present replicated LUNs to the nodes in the DR > > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" > > > > Tested with multiples LUNs and filesystems on various workloads - seems > > to be working > > > > --Alex > > > > > > > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales > > > wrote: > > Thanks for answering. > > > > Essentially, the idea being explored is to replicate LUNs between > > identical storage hardware (HP 3PAR volumesrein) on both sites. There is > > an IP connection between storage boxes but not between servers on both > > sites, there is a dark fiber connecting both sites. Here they dont want > to > > explore the idea of a scaled-based. > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=d- > > vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m= > VbfWaftYSVjx8fMb2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- > > GeVJiVS49opBwwCFs-FKQrQ7rc&e= > > > > > > > > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_ > attachments_20180126_e291af63_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=bYnf-7v0CxYUkGth-QaVeUQdIlG8f1Gro-hwOxok7Qw&e=> > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: not available > > Type: image/jpeg > > Size: 26117 bytes > > Desc: not available > > URL: > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_ > attachments_20180126_e291af63_attachment.jpe&d=DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=jYdnqhQBlnpf58oxunzBcTs9XdcbeOtLDQdgnASidDA&e=> > > > > ------------------------------ > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url? > > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d= > DwICAg&c=jf_iaSHvJObTbx- > > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- > > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= > > > > > > End of gpfsug-discuss Digest, Vol 72, Issue 69 > > ********************************************** > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hmorales at optimizeit.co Sat Jan 27 09:11:44 2018 From: hmorales at optimizeit.co (Harold Morales) Date: Sat, 27 Jan 2018 04:11:44 -0500 Subject: [gpfsug-discuss] storage-based replication for Spectrum Scale In-Reply-To: References: Message-ID: Thanks for your insights. Alex, we did as you mentioned but after using mmimportfs there are a lot of errors on all commands having to do with filesystems: GPFS: 6027-419 Failed to read a file system descriptor. There is an input or output error that occurs for: mmlsfs mmlsdisk mmdf obviously, fs won't mount. 2018-01-27 0:02 GMT-05:00 Alex Levin : > Steve, > I've read that "mmfsctl suspend or suspend-write" should be executed, > but in real life it is impossible for DR scenario. > > We tested both cases, > the graceful one when the failover to another site is planned, > applications stopped and i/o suspended > and the case when there was no advanced notice about the disaster in the > primary site. > > Both worked and for the second case various loads were simulated including > heavy writes and reads/writes. > In disaster model as expected some data were lost (due to incomplete > writes, replication latency ... ), > but mmfsck was always able to repair and the applications databases > located on the affected filesystem were in an acceptable state. > > > it is possible that we were just lucky and the test case didn't cover all > possible scenarios. > > Harold, > In our case, it is Linux, not AIX, but it shouldn't matter > And our DR cluster is fully configured ( different IPs, hostnames and > cluster name ) and running without filesystems at all or with > a differently named filesystem. > > Before running mmimport , make sure that all expected LUNs are visible > and writable. > You can verify if the device is correct by reading first blocks of the > device ( for example dd if=/dev/NSD_LUN_device bs=1M count=16 | strings ) > > not sure where you are " moved the mmsdrfs" > you don't need to move/modify mmsdrfs file on the target ( disaster > recovery ) cluster > > Just copy the one from primary to /tmp or /var/tmp and try to run mmimportfs > fs_name -i copy_of_mmsdrfs /tmp/mmsdrfs > > > > Glen, as Harold has no IP connectivity between the clusters "mmfsctl > syncFSconfig" is not an option ... > > Thanks > --Alex > > > > > On Fri, Jan 26, 2018 at 4:04 PM, Steve Xiao wrote: > >> When using this method of replication, you need to either issue "mmfsctl >> suspend or suspend-write" command before replication or setup a single >> consistency group for all LUNs. This is needed to ensure replica contain >> a consistent copy of GPFS data. >> >> Steve Y. Xiao >> >> gpfsug-discuss-bounces at spectrumscale.org wrote on 01/26/2018 03:21:23 PM: >> >> > From: gpfsug-discuss-request at spectrumscale.org >> > To: gpfsug-discuss at spectrumscale.org >> > Date: 01/26/2018 03:21 PM >> > Subject: gpfsug-discuss Digest, Vol 72, Issue 69 >> > Sent by: gpfsug-discuss-bounces at spectrumscale.org >> > >> > Send gpfsug-discuss mailing list submissions to >> > gpfsug-discuss at spectrumscale.org >> > >> > To subscribe or unsubscribe via the World Wide Web, visit >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > or, via email, send a message with subject or body 'help' to >> > gpfsug-discuss-request at spectrumscale.org >> > >> > You can reach the person managing the list at >> > gpfsug-discuss-owner at spectrumscale.org >> > >> > When replying, please edit your Subject line so it is more specific >> > than "Re: Contents of gpfsug-discuss digest..." >> > >> > >> > Today's Topics: >> > >> > 1. Re: storage-based replication for Spectrum Scale (Harold Morales) >> > 2. Re: storage-based replication for Spectrum Scale (Glen Corneau) >> > >> > >> > ---------------------------------------------------------------------- >> > >> > Message: 1 >> > Date: Fri, 26 Jan 2018 13:29:09 -0500 >> > From: Harold Morales >> > To: gpfsug main discussion list >> > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum >> > Scale >> > Message-ID: >> > >> > Content-Type: text/plain; charset="utf-8" >> >> > >> > Hi Alex, This set up seems close to what I am trying to achieve. >> > >> > With regards to this kind of replication: any prereqs need to be met in >> the >> > target environment for this to work? for example, should disk devices >> > naming on AIX be the same as in the source environment? when importing >> the >> > mmsdrfs file, how is Scale going to know which disks should it assign to >> > the cluster? by their hdisk name alone? >> > >> > Thanks again, >> > >> > >> > >> > 2018-01-24 2:30 GMT-05:00 Alex Levin : >> > >> > > Hi, >> > > >> > > We are using a similar type of replication. >> > > I assume the site B is the cold site prepared for DR >> > > >> > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. >> > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX >> replication >> > > group to ensure consistency. >> > > >> > > The cluster name, IP addresses , hostnames of the cluster nodes are >> > > different on another site - it can be a pre-configured cluster without >> > > gpfs filesystems or with another filesystem. >> > > Same names and addresses shouldn't be a problem. >> > > >> > > Additionally to the replicated LUNs/NSDs you need to deliver copy >> > > of /var/mmfs/gen/mmsdrfs file from A to B site. >> > > There is no need to replicate it in real-time, only after the change >> of >> > > the cluster configuration. >> > > >> > > To activate site B - present replicated LUNs to the nodes in the DR >> > > cluster and run mmimportfs as "mmimportfs fs_name -i >> copy_of_mmsdrfs" >> > > >> > > Tested with multiples LUNs and filesystems on various workloads - >> seems >> > > to be working >> > > >> > > --Alex >> > > >> > > >> > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales < >> hmorales at optimizeit.co> >> > > wrote: >> > > >> > >> Thanks for answering. >> > >> >> > >> Essentially, the idea being explored is to replicate LUNs between >> > >> identical storage hardware (HP 3PAR volumesrein) on both sites. >> There is an >> > >> IP connection between storage boxes but not between servers on both >> sites, >> > >> there is a dark fiber connecting both sites. Here they dont want to >> explore >> > >> the idea of a scaled-based. >> > >> >> > >> >> > >> _______________________________________________ >> > >> gpfsug-discuss mailing list >> > >> gpfsug-discuss at spectrumscale.org >> > >> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> >> > >> >> > > >> > > _______________________________________________ >> > > gpfsug-discuss mailing list >> > > gpfsug-discuss at spectrumscale.org >> > > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > > >> > > >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: > > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_1503995b_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=SKdMmQae8uzHNWZq3vuRTp5UVwYFeeusLAxtbaposX0&e=> >> > >> > ------------------------------ >> > >> > Message: 2 >> > Date: Fri, 26 Jan 2018 14:21:15 -0600 >> > From: "Glen Corneau" >> > To: gpfsug main discussion list >> > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum >> > Scale >> > Message-ID: >> > > > 006FCEEE at notes.na.collabserv.com> >> > >> > Content-Type: text/plain; charset="us-ascii" >> > >> > Scale will walk across all discovered disks upon start time and attempt >> to >> > read the NSD identifiers from the disks. Once it finds them, it makes >> a >> > local map file that correlates the NSD id and the hdiskX identifier. >> The >> > names do not have to be the same as either the source cluster or even >> from >> > node-to-node. >> > >> > The main thing to keep in mind is to keep the file system definitions >> in >> > sync between the source and destination clusters. The "syncFSconfig" >> user >> > exit is the best way to do it because it's automatic. You generally >> > shouldn't be shuffling the mmsdrfs file between sites, that's what the >> > "mmfsctl syncFSconfig" does for you, on a per-file system basis. >> > >> > GPFS+AIX customers have been using this kind of storage replication for >> > over 10 years, it's business as usual. >> > >> > ------------------ >> > Glen Corneau >> > Power Systems >> > Washington Systems Center >> > gcorneau at us.ibm.com >> > >> > >> > >> > >> > >> > From: Harold Morales >> > To: gpfsug main discussion list >> > Date: 01/26/2018 12:30 PM >> > Subject: Re: [gpfsug-discuss] storage-based replication for >> > Spectrum Scale >> > Sent by: gpfsug-discuss-bounces at spectrumscale.org >> > >> > >> > >> > Hi Alex, This set up seems close to what I am trying to achieve. >> > >> > With regards to this kind of replication: any prereqs need to be met in >> > the target environment for this to work? for example, should disk >> devices >> > naming on AIX be the same as in the source environment? when importing >> the >> > mmsdrfs file, how is Scale going to know which disks should it assign >> to >> > the cluster? by their hdisk name alone? >> > >> > Thanks again, >> > >> > >> > >> > 2018-01-24 2:30 GMT-05:00 Alex Levin : >> > Hi, >> > >> > We are using a similar type of replication. >> > I assume the site B is the cold site prepared for DR >> > >> > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. >> > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX >> replication >> > group to ensure consistency. >> > >> > The cluster name, IP addresses , hostnames of the cluster nodes are >> > different on another site - it can be a pre-configured cluster without >> > gpfs filesystems or with another filesystem. >> > Same names and addresses shouldn't be a problem. >> > >> > Additionally to the replicated LUNs/NSDs you need to deliver copy >> > of /var/mmfs/gen/mmsdrfs file from A to B site. >> > There is no need to replicate it in real-time, only after the change of >> > the cluster configuration. >> > >> > To activate site B - present replicated LUNs to the nodes in the DR >> > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" >> > >> > Tested with multiples LUNs and filesystems on various workloads - >> seems >> > to be working >> > >> > --Alex >> > >> > >> > >> > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales >> >> > wrote: >> > Thanks for answering. >> > >> > Essentially, the idea being explored is to replicate LUNs between >> > identical storage hardware (HP 3PAR volumesrein) on both sites. There >> is >> > an IP connection between storage boxes but not between servers on both >> > sites, there is a dark fiber connecting both sites. Here they dont want >> to >> > explore the idea of a scaled-based. >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=d- >> > vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m=VbfWaftYSVjx8fMb >> 2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- >> > GeVJiVS49opBwwCFs-FKQrQ7rc&e= >> > >> > >> > >> > >> > >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: > > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_e291af63_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=bYnf-7v0CxYUkGth-QaVeUQdIlG8f1Gro-hwOxok7Qw&e=> >> > -------------- next part -------------- >> > A non-text attachment was scrubbed... >> > Name: not available >> > Type: image/jpeg >> > Size: 26117 bytes >> > Desc: not available >> > URL: > > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_e291af63_attachment.jpe&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=jYdnqhQBlnpf58oxunzBcTs9XdcbeOtLDQdgnASidDA&e=> >> > >> > ------------------------------ >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > >> > End of gpfsug-discuss Digest, Vol 72, Issue 69 >> > ********************************************** >> > >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egonle at aim.com Sat Jan 27 15:18:24 2018 From: egonle at aim.com (Eg. Bo.) Date: Sat, 27 Jan 2018 10:18:24 -0500 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Message-ID: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> Hello, while operating a GPFS filesystem for HPC some workload of few clients is able to eat available storage and NSD performance. Compute Nodes and NSD servers are connected to Infiniband.This question somehow is weird but is there a way to limit performance / I/O / bandwidth consumption for nodes so that other node still get "good" performance? Thanks, Eg. Bo.egonle at aim.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Sun Jan 28 15:53:47 2018 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sun, 28 Jan 2018 12:53:47 -0300 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth In-Reply-To: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> References: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> Message-ID: Depends... While the current version of mmchqos is primarily geared towards controlling "maintenance" tasks. It can be used to limit (cap) the IOP rate for IOs coming from a particular GFPFS client node. You would need a version that supports the -N option on the mmchqos command and then configure each node that will mount the FS. Caution: try some experiments on a test system -- incorrect configuration can severely impact performance to the point where it can appear that things are totally "stuck"! mmchqos FS --disable --reset should get you back to the default config for gpfs/QOS. From: "Eg. Bo." To: gpfsug-discuss at spectrumscale.org Date: 01/27/2018 12:23 PM Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, while operating a GPFS filesystem for HPC some workload of few clients is able to eat available storage and NSD performance. Compute Nodes and NSD servers are connected to Infiniband. This question somehow is weird but is there a way to limit performance / I/O / bandwidth consumption for nodes so that other node still get "good" performance? Thanks, Eg. Bo. egonle at aim.com _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=r0AtLTBmDo_uZhIdntBO8k2-1Alg_2LBLPoJamV-zsY&s=BXGAuET7K4UhA24P72F4YRsAdcdJR3H02gqGOERENlE&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From vpuvvada at in.ibm.com Mon Jan 29 07:04:56 2018 From: vpuvvada at in.ibm.com (Venkateswara R Puvvada) Date: Mon, 29 Jan 2018 12:34:56 +0530 Subject: [gpfsug-discuss] AFM and RHEL 7.4 In-Reply-To: References: Message-ID: This is testing effort, AFM with RHEL 7.4 will be officially supported in next PTF releases 5.0.0.1 and 4.2.3.7. ~Venkat (vpuvvada at in.ibm.com) From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 01/25/2018 01:57 PM Subject: [gpfsug-discuss] AFM and RHEL 7.4 Sent by: gpfsug-discuss-bounces at spectrumscale.org The FAQ has a note stating: 1. AFM, Asynch Disaster Recovery with AFM, Integrated Protocols, and Installation Toolkit are not supported on RHEL 7.4. Could someone please clarify this sentence ? It can't be right that none of these features are supported with RHEL 7.4, or .. ? -jf_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=WsrKUTbd_huFePIa8dbAt4aezWpKHV902b-BlZTwGiA&s=OMRkxSzqA8U8J8mo3ap8I5TKscWunNEA2KZV3Z5_6HA&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From richardb+gpfsUG at ellexus.com Mon Jan 29 10:27:37 2018 From: richardb+gpfsUG at ellexus.com (Richard Booth) Date: Mon, 29 Jan 2018 10:27:37 +0000 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Message-ID: Hi Without wishing to blow our own trumpet, Ellexus tools do this with GPFS, There is a youtube video that shows our tool Mistral managing a fair usage policy, link below, https://youtu.be/LJFcBUkcn3c Richard ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 27 Jan 2018 10:18:24 -0500 > From: "Eg. Bo." > To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth > Message-ID: <1613832bc3c-1721-1396f9 at webjas-vaa235.srv.aolmail.net> > Content-Type: text/plain; charset="utf-8" > > > Hello, > > > while operating a GPFS filesystem for HPC some workload of few clients is > able to eat available storage and NSD performance. Compute Nodes and NSD > servers are connected to Infiniband.This question somehow is weird but is > there a way to limit performance / I/O / bandwidth consumption for nodes so > that other node still get "good" performance? > > > > > Thanks, > > > Eg. Bo.egonle at aim.com > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: 20180127/dcc7cd79/attachment-0001.html> > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 72, Issue 74 > ********************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.hearns at asml.com Mon Jan 29 09:54:46 2018 From: john.hearns at asml.com (John Hearns) Date: Mon, 29 Jan 2018 09:54:46 +0000 Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth In-Reply-To: References: <1613832bc3c-1721-1396f9@webjas-vaa235.srv.aolmail.net> Message-ID: Hello Egonle. I looked at this scenario recently (H Mark!) Maybe you could send me an email off-list? From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Marc A Kaplan Sent: Sunday, January 28, 2018 4:54 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] limit / fair share of GPFS bandwidth Depends... While the current version of mmchqos is primarily geared towards controlling "maintenance" tasks. It can be used to limit (cap) the IOP rate for IOs coming from a particular GFPFS client node. You would need a version that supports the -N option on the mmchqos command and then configure each node that will mount the FS. Caution: try some experiments on a test system -- incorrect configuration can severely impact performance to the point where it can appear that things are totally "stuck"! mmchqos FS --disable --reset should get you back to the default config for gpfs/QOS. From: "Eg. Bo." > To: gpfsug-discuss at spectrumscale.org Date: 01/27/2018 12:23 PM Subject: [gpfsug-discuss] limit / fair share of GPFS bandwidth Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Hello, while operating a GPFS filesystem for HPC some workload of few clients is able to eat available storage and NSD performance. Compute Nodes and NSD servers are connected to Infiniband. This question somehow is weird but is there a way to limit performance / I/O / bandwidth consumption for nodes so that other node still get "good" performance? Thanks, Eg. Bo. egonle at aim.com_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=r0AtLTBmDo_uZhIdntBO8k2-1Alg_2LBLPoJamV-zsY&s=BXGAuET7K4UhA24P72F4YRsAdcdJR3H02gqGOERENlE&e= -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkr at lbl.gov Wed Jan 31 21:20:04 2018 From: kkr at lbl.gov (Kristy Kallback-Rose) Date: Wed, 31 Jan 2018 13:20:04 -0800 Subject: [gpfsug-discuss] US Spectrum Scale/GPFS User Group - Save the Date ~May 16 - ~May 17 - Boston Message-ID: <46BCDF1D-72C8-4FD1-8375-DD4207732F78@lbl.gov> Please mark your calendars, we?re planning for our next US Spectrum Scale/GPFS User Group meeting. We should be able to provide more exact details in a week or so, but loosely the event will be two days in Boston around May 16 - May 17, 2018. Send agenda suggestions now. Feel free to reply to the list to generate discussion. Best, Kristy From chris.schlipalius at pawsey.org.au Wed Jan 31 23:21:02 2018 From: chris.schlipalius at pawsey.org.au (Chris Schlipalius) Date: Thu, 01 Feb 2018 07:21:02 +0800 Subject: [gpfsug-discuss] 2018 March 26th Singapore Spectrum Scale User Group event announced - Final call for user stories - submission deadline 9th Feb Message-ID: Hello, First, let me apologise to all who have responded. This is a follow up call out for user use cases or presentations for those wishing to present at this Usergroup ? namely client use cases or client stories, from those who haven?t yet contacted me to organise to do so. At the inaugural Spectrum Scale Usergroup Singapore on the Monday 26th March 2018, Sentosa, Singapore. This is being held in conjunction with SCA18 https://sc-asia.org/ All current Spectrum Scale User Group event details can be found here: http://goo.gl/dXtqvS Feel free to circulate this event link to all that may need it. Please ensure a speaker?s ticket is reserved, and please email me the title and duration of the talk and speakers name details so I can add this to the agenda on Eventbrite and promote on the discussion list and spectrumscale.org website. Accommodation and conference registration - both are open ? please see https://www.sc-asia.org/accommodation/ for the accommodation booking form, and https://www.sc-asia.org/register/ for the early bird conference registration (early bird rate closes soon 9th Feb). Please reserve a ticket in the Eventbrite link above ASAP and if you wish, email me separately if you have accommodation sorted, or need Sentosa accommodation (in addition to completing the form linked above). As I am in contact with the organisers of the event and I can also ensure there are places for presenters at this group and attendees. The user group location is still located at Resorts World Convention We are looking forwards to a great new Usergroup soon in a fabulous venue. Thanks again to NSCC and IBM for helping to arrange the venue and event booking. Regards, Chris Schlipalius Team Lead, Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO) 12 Burvill Court Kensington WA 6151 Australia