<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div dir="ltr" >Simon,</div>
<div dir="ltr" >Hi.</div>
<div dir="ltr" >Can you explain why you would like a full copy of all blocks on all 4 NSD servers ?</div>
<div dir="ltr" >Is there a particular use case, and hence an interest from product development?</div>
<div dir="ltr" > </div>
<div dir="ltr" >Otherwise remember that with 4 NSD servers, with one failure group per (storage rich) NSD server, then all 4 disk arrays will be loaded equally, as new files will get written to any 3 (or 2 or 1) of the 4 failure groups.</div>
<div dir="ltr" >Also remember that as you add more replication then there is more network load on the gpfs client as it has to perform all the writes itself.</div>
<div dir="ltr" > </div>
<div dir="ltr" >Perhaps someone technical can comment on the logic that determines which '3' out of 4 failure groups, a particular block is written to.</div>
<div dir="ltr" > </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div dir="ltr" style="margin-top: 20px;" ><div style="font-family: sans-serif; font-size: 8pt; margin-top: 10px;" ><div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" ><span style="font-family:Verdana, Geneva, sans-serif" ><span style="font-size:11.5pt;" >Daniel</span></span></div>
<div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" ><img alt="/spectrum_storage-banne" src="" src="http://ausgsa.ibm.com/projects/t/tivoli_visual_design/public/2015/Spectrum-Storage/Email-signatures/Storage/spectrum_storage-banner.png" style="width: 601px; height: 5px;" ></div>
<div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" ><br> </div>
<table cellpadding="0" cellspacing="0" border="0" > <tbody> <tr> <td style="width:201px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" ><img alt="Spectrum Scale Logo" src="" src="http://ausgsa.ibm.com/projects/t/tivoli_visual_design/public/2015/Spectrum-Storage/Email-signatures/Storage/spectrum_scale-logo.png" style="width: 75px; height: 120px; float: left;" ></div>
<div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" > </div> </td> <td style="width:21px;padding:0cm 0cm 0cm 0cm;" > </td> <td style="width:202px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" ><strong><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:10.0pt;" >Dr Daniel Kidger</span></span></strong><br> <span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:7.5pt;" >IBM Technical Sales Specialist<br> Software Defined Solution Sales<br> <br> +</span></span><span style="color:#5F5F5F;" ><span style="font-family:Verdana, Geneva, sans-serif" ><span style="font-size:10.0pt;" >44-07818 522 266 </span></span></span><br> <span style="color:#5F5F5F;" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >daniel.kidger@uk.ibm.com</span></span></span></div> </td> </tr> </tbody></table>
<div> </div>
<div><br><font size="2" face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" > </font></div></div></div></div></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Steve Duersch <duersch@us.ibm.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss@spectrumscale.org<br>Cc:<br>Subject: Re: [gpfsug-discuss] Maximum value for data replication?<br>Date: Wed, Aug 31, 2016 1:45 PM<br>
<p><font size="2" face="sans-serif" >>></font><tt><font size="3" face="" >Is there a maximum value for data replication in Spectrum Scale?</font></tt><br><font size="2" face="sans-serif" >The maximum value for replication is 3. </font><br><br><br><font size="2" face="sans-serif" >Steve Duersch<br>Spectrum Scale RAID<br>845-433-7902<br>IBM Poughkeepsie, New York</font><br><br><br><br><br><img alt="Inactive hide details for gpfsug-discuss-request---08/30/2016 07:25:24 PM---Send gpfsug-discuss mailing list submissions to gp" src="cid:1__=0ABB0AB3DFD67DBA8f9e8a93df938@us.ibm.com" height="16" width="16" border="0" ><font color="#424282" size="2" face="sans-serif" >gpfsug-discuss-request---08/30/2016 07:25:24 PM---Send gpfsug-discuss mailing list submissions to gpfsug-discuss@spectrumscale.org</font><br><br><font color="#5F5F5F" size="1" face="sans-serif" >From: </font><font size="1" face="sans-serif" >gpfsug-discuss-request@spectrumscale.org</font><br><font color="#5F5F5F" size="1" face="sans-serif" >To: </font><font size="1" face="sans-serif" >gpfsug-discuss@spectrumscale.org</font><br><font color="#5F5F5F" size="1" face="sans-serif" >Date: </font><font size="1" face="sans-serif" >08/30/2016 07:25 PM</font><br><font color="#5F5F5F" size="1" face="sans-serif" >Subject: </font><font size="1" face="sans-serif" >gpfsug-discuss Digest, Vol 55, Issue 55</font><br><font color="#5F5F5F" size="1" face="sans-serif" >Sent by: </font><font size="1" face="sans-serif" >gpfsug-discuss-bounces@spectrumscale.org</font></p>
<hr style="color:#8091A5; " size="2" width="100%" align="left" ><br><br><br><tt><font size="3" face="" >Send gpfsug-discuss mailing list submissions to<br>gpfsug-discuss@spectrumscale.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit</font></tt><br><tt><font size="3" face="" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br><tt><font size="3" face="" >or, via email, send a message with subject or body 'help' to<br>gpfsug-discuss-request@spectrumscale.org<br><br>You can reach the person managing the list at<br>gpfsug-discuss-owner@spectrumscale.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of gpfsug-discuss digest..."<br><br><br>Today's Topics:<br><br> 1. Maximum value for data replication?<br> (Simon Thompson (Research Computing - IT Services))<br> 2. greetings (Kevin D Johnson)<br> 3. GPFS 3.5.0 on RHEL 6.8 (Lukas Hejtmanek)<br> 4. Re: GPFS 3.5.0 on RHEL 6.8 (Kevin D Johnson)<br> 5. Re: GPFS 3.5.0 on RHEL 6.8 (mark.bergman@uphs.upenn.edu)<br> 6. Re: *New* IBM Spectrum Protect Whitepaper "Petascale Data<br> Protection" (Lukas Hejtmanek)<br> 7. Re: *New* IBM Spectrum Protect Whitepaper "Petascale Data<br> Protection" (Sven Oehme)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Tue, 30 Aug 2016 19:09:05 +0000<br>From: "Simon Thompson (Research Computing - IT Services)"<br><S.J.Thompson@bham.ac.uk><br>To: "gpfsug-discuss@spectrumscale.org"<br><gpfsug-discuss@spectrumscale.org><br>Subject: [gpfsug-discuss] Maximum value for data replication?<br>Message-ID:<br><CF45EE16DEF2FE4B9AA7FF2B6EE26545F5813FFC@EX13.adf.bham.ac.uk><br>Content-Type: text/plain; charset="us-ascii"<br><br>Is there a maximum value for data replication in Spectrum Scale?<br><br>I have a number of nsd servers which have local storage and Id like each node to have a full copy of all the data in the file-system, say this value is 4, can I set replication to 4 for data and metadata and have each server have a full copy?<br><br>These are protocol nodes and multi cluster mount another file system (yes I know not supported) and the cesroot is in the remote file system. On several occasions where GPFS has wibbled a bit, this has caused issues with ces locks, so I was thinking of moving the cesroot to a local filesysyem which is replicated on the local ssds in the protocol nodes. I.e. Its a generally quiet file system as its only ces cluster config.<br><br>I assume if I stop protocols, rsync the data and then change to the new ces root, I should be able to get this working?<br><br>Thanks<br><br>Simon<br><br>------------------------------<br><br>Message: 2<br>Date: Tue, 30 Aug 2016 19:43:39 +0000<br>From: "Kevin D Johnson" <kevindjo@us.ibm.com><br>To: gpfsug-discuss@spectrumscale.org<br>Subject: [gpfsug-discuss] greetings<br>Message-ID:<br><OFBE10B787.D9EE56E3-ON0025801F.006C39E8-0025801F.006C5E11@notes.na.collabserv.com><br><br>Content-Type: text/plain; charset="us-ascii"<br><br>An HTML attachment was scrubbed...<br>URL: <</font></tt><tt><font size="3" face="" ><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/5a2e22a3/attachment-0001.html" target="_blank" >http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/5a2e22a3/attachment-0001.html</a></font></tt><tt><font size="3" face="" >><br><br>------------------------------<br><br>Message: 3<br>Date: Tue, 30 Aug 2016 22:39:18 +0200<br>From: Lukas Hejtmanek <xhejtman@ics.muni.cz><br>To: gpfsug-discuss@spectrumscale.org<br>Subject: [gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8<br>Message-ID: <20160830203917.qptfgqvlmdbzu6wr@ics.muni.cz><br>Content-Type: text/plain; charset=iso-8859-2<br><br>Hello,<br><br>does it work for anyone? As of kernel 2.6.32-642, GPFS 3.5.0 (including the<br>latest patch 32) does start but does not mount and file system. The internal<br>mount cmd gets stucked.<br><br>--<br>Luk?? Hejtm?nek<br><br><br>------------------------------<br><br>Message: 4<br>Date: Tue, 30 Aug 2016 20:51:39 +0000<br>From: "Kevin D Johnson" <kevindjo@us.ibm.com><br>To: gpfsug-discuss@spectrumscale.org<br>Subject: Re: [gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8<br>Message-ID:<br><OF60CB398A.7B2AF5FF-ON0025801F.007282A3-0025801F.0072979C@notes.na.collabserv.com><br><br>Content-Type: text/plain; charset="us-ascii"<br><br>An HTML attachment was scrubbed...<br>URL: <</font></tt><tt><font size="3" face="" ><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/341d5e11/attachment-0001.html" target="_blank" >http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/341d5e11/attachment-0001.html</a></font></tt><tt><font size="3" face="" >><br><br>------------------------------<br><br>Message: 5<br>Date: Tue, 30 Aug 2016 17:07:21 -0400<br>From: mark.bergman@uphs.upenn.edu<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Subject: Re: [gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8<br>Message-ID: <24437-1472591241.445832@bR6O.TofS.917u><br>Content-Type: text/plain; charset="UTF-8"<br><br>In the message dated: Tue, 30 Aug 2016 22:39:18 +0200,<br>The pithy ruminations from Lukas Hejtmanek on<br><[gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8> were:<br>=> Hello, <br><br>GPFS 3.5.0.[23..3-0] work for me under [CentOS|ScientificLinux] 6.8,<br>but at kernel 2.6.32-573 and lower.<br><br>I've found kernel bugs in blk_cloned_rq_check_limits() in later kernel<br>revs that caused multipath errors, resulting in GPFS being unable to<br>find all NSDs and mount the filesystem.<br><br>I am not updating to a newer kernel until I'm certain this is resolved.<br><br>I opened a bug with CentOS:</font></tt><br><br><tt><font size="3" face="" ><a href="https://bugs.centos.org/view.php?id=10997" target="_blank" >https://bugs.centos.org/view.php?id=10997</a></font></tt><br><br><tt><font size="3" face="" >and began an extended discussion with the (RH & SUSE) developers of that<br>chunk of kernel code. I don't know if an upstream bug has been opened<br>by RH, but see:</font></tt><br><br><tt><font size="3" face="" ><a href="https://patchwork.kernel.org/patch/9140337/" target="_blank" >https://patchwork.kernel.org/patch/9140337/</a></font></tt><br><tt><font size="3" face="" >=><br>=> does it work for anyone? As of kernel 2.6.32-642, GPFS 3.5.0 (including the<br>=> latest patch 32) does start but does not mount and file system. The internal<br>=> mount cmd gets stucked.<br>=><br>=> --<br>=> Luk?? Hejtm?nek <br><br><br>--<br>Mark Bergman voice: 215-746-4061 <br>mark.bergman@uphs.upenn.edu fax: 215-614-0266</font></tt><br><tt><font size="3" face="" ><a href="http://www.cbica.upenn.edu/" target="_blank" >http://www.cbica.upenn.edu/</a></font></tt><br><tt><font size="3" face="" >IT Technical Director, Center for Biomedical Image Computing and Analytics<br>Department of Radiology University of Pennsylvania<br> PGP Key: </font></tt><tt><font size="3" face="" ><a href="http://www.cbica.upenn.edu/sbia/bergman" target="_blank" >http://www.cbica.upenn.edu/sbia/bergman</a></font></tt><tt><font size="3" face="" > <br><br><br>------------------------------<br><br>Message: 6<br>Date: Wed, 31 Aug 2016 00:02:50 +0200<br>From: Lukas Hejtmanek <xhejtman@ics.muni.cz><br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Subject: Re: [gpfsug-discuss] *New* IBM Spectrum Protect Whitepaper<br>"Petascale Data Protection"<br>Message-ID: <20160830220250.yt6r7gvfq7rlvtcs@ics.muni.cz><br>Content-Type: text/plain; charset=iso-8859-2<br><br>Hello,<br><br>On Mon, Aug 29, 2016 at 09:20:46AM +0200, Frank Kraemer wrote:<br>> Find the paper here:<br>><br>> </font></tt><tt><font size="3" face="" ><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection" target="_blank" >https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection</a></font></tt><br><br><tt><font size="3" face="" >thank you for the paper, I appreciate it.<br><br>However, I wonder whether it could be extended a little. As it has the title<br>Petascale Data Protection, I think that in Peta scale, you have to deal with<br>millions (well rather hundreds of millions) of files you store in and this is<br>something where TSM does not scale well.<br><br>Could you give some hints:<br><br>On the backup site:<br>mmbackup takes ages for:<br>a) scan (try to scan 500M files even in parallel)<br>b) backup - what if 10 % of files get changed - backup process can be blocked<br>several days as mmbackup cannot run in several instances on the same file<br>system, so you have to wait until one run of mmbackup finishes. How long could<br>it take at petascale?<br><br>On the restore site:<br>how can I restore e.g. 40 millions of file efficiently? dsmc restore '/path/*'<br>runs into serious troubles after say 20M files (maybe wrong internal<br>structures used), however, scanning 1000 more files takes several minutes<br>resulting the dsmc restore never reaches that 40M files.<br><br>using filelists the situation is even worse. I run dsmc restore -filelist<br>with a filelist consisting of 2.4M files. Running for *two* days without<br>restoring even a single file. dsmc is consuming 100 % CPU.<br><br>So any hints addressing these issues with really large number of files would<br>be even more appreciated.<br><br>--<br>Luk?? Hejtm?nek<br><br><br>------------------------------<br><br>Message: 7<br>Date: Tue, 30 Aug 2016 16:24:59 -0700<br>From: Sven Oehme <oehmes@gmail.com><br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Subject: Re: [gpfsug-discuss] *New* IBM Spectrum Protect Whitepaper<br>"Petascale Data Protection"<br>Message-ID:<br><CALssuR1qWo2Y5adUUZJtLgkNPUYztWpYP0XhNLTjBOG5352qjg@mail.gmail.com><br>Content-Type: text/plain; charset="utf-8"<br><br>so lets start with some simple questions.<br><br>when you say mmbackup takes ages, what version of gpfs code are you running<br>?<br>how do you execute the mmbackup command ? exact parameters would be useful<br>.<br>what HW are you using for the metadata disks ?<br>how much capacity (df -h) and how many inodes (df -i) do you have in the<br>filesystem you try to backup ?<br><br>sven<br><br><br>On Tue, Aug 30, 2016 at 3:02 PM, Lukas Hejtmanek <xhejtman@ics.muni.cz><br>wrote:<br><br>> Hello,<br>><br>> On Mon, Aug 29, 2016 at 09:20:46AM +0200, Frank Kraemer wrote:<br>> > Find the paper here:<br>> ><br>> > </font></tt><tt><font size="3" face="" ><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/" target="_blank" >https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/</a></font></tt><br><tt><font size="3" face="" >> Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection<br>><br>> thank you for the paper, I appreciate it.<br>><br>> However, I wonder whether it could be extended a little. As it has the<br>> title<br>> Petascale Data Protection, I think that in Peta scale, you have to deal<br>> with<br>> millions (well rather hundreds of millions) of files you store in and this<br>> is<br>> something where TSM does not scale well.<br>><br>> Could you give some hints:<br>><br>> On the backup site:<br>> mmbackup takes ages for:<br>> a) scan (try to scan 500M files even in parallel)<br>> b) backup - what if 10 % of files get changed - backup process can be<br>> blocked<br>> several days as mmbackup cannot run in several instances on the same file<br>> system, so you have to wait until one run of mmbackup finishes. How long<br>> could<br>> it take at petascale?<br>><br>> On the restore site:<br>> how can I restore e.g. 40 millions of file efficiently? dsmc restore<br>> '/path/*'<br>> runs into serious troubles after say 20M files (maybe wrong internal<br>> structures used), however, scanning 1000 more files takes several minutes<br>> resulting the dsmc restore never reaches that 40M files.<br>><br>> using filelists the situation is even worse. I run dsmc restore -filelist<br>> with a filelist consisting of 2.4M files. Running for *two* days without<br>> restoring even a single file. dsmc is consuming 100 % CPU.<br>><br>> So any hints addressing these issues with really large number of files<br>> would<br>> be even more appreciated.<br>><br>> --<br>> Luk?? Hejtm?nek<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><tt><font size="3" face="" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br><tt><font size="3" face="" >><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><tt><font size="3" face="" ><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/d9b3fb68/attachment.html" target="_blank" >http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/d9b3fb68/attachment.html</a></font></tt><tt><font size="3" face="" >><br><br>------------------------------<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><br><tt><font size="3" face="" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br><br><br><tt><font size="3" face="" >End of gpfsug-discuss Digest, Vol 55, Issue 55<br>**********************************************</font></tt><br><br><br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr" > </div></div>
Unless stated otherwise above:<BR>
IBM United Kingdom Limited - Registered in England and Wales with number 741598. <BR>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU<BR>
<BR>