<html><body><p>Thanks for reading the paper. I agree that the restore of a large number of files is a challenge today. The restore is the focus area for future enhancements for the integration between IBM Spectrum Scale and IBM Spectrum Protect. If something will be available that helps to improve the restore capabilities the paper will be updated with this information.<br><br>Greetings, Dominic.<br><br><br><img width="16" height="16" src="cid:1__=4EBB0AB3DF8C478D8f9e8a93df938690918c4EB@" border="0" alt="Inactive hide details for gpfsug-discuss-request---31.08.2016 01:25:17---Send gpfsug-discuss mailing list submissions to gpfsu"><font color="#424282">gpfsug-discuss-request---31.08.2016 01:25:17---Send gpfsug-discuss mailing list submissions to gpfsug-discuss@spectrumscale.org</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">gpfsug-discuss-request@spectrumscale.org</font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug-discuss@spectrumscale.org</font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">31.08.2016 01:25</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">gpfsug-discuss Digest, Vol 55, Issue 55</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt>Send gpfsug-discuss mailing list submissions to<br> gpfsug-discuss@spectrumscale.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> </tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br>or, via email, send a message with subject or body 'help' to<br> gpfsug-discuss-request@spectrumscale.org<br><br>You can reach the person managing the list at<br> gpfsug-discuss-owner@spectrumscale.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of gpfsug-discuss digest..."<br>Today's Topics:<br><br> 1. Maximum value for data replication?<br> (Simon Thompson (Research Computing - IT Services))<br> 2. greetings (Kevin D Johnson)<br> 3. GPFS 3.5.0 on RHEL 6.8 (Lukas Hejtmanek)<br> 4. Re: GPFS 3.5.0 on RHEL 6.8 (Kevin D Johnson)<br> 5. Re: GPFS 3.5.0 on RHEL 6.8 (mark.bergman@uphs.upenn.edu)<br> 6. Re: *New* IBM Spectrum Protect Whitepaper "Petascale Data<br> Protection" (Lukas Hejtmanek)<br> 7. Re: *New* IBM Spectrum Protect Whitepaper "Petascale Data<br> Protection" (Sven Oehme)<br></tt><font color="#800080"><br>----- Message from "Simon Thompson (Research Computing - IT Services)" <S.J.Thompson@bham.ac.uk> on Tue, 30 Aug 2016 19:09:05 +0000 -----</font><table border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">To:</font></b></div></td><td width="457" valign="middle"><font size="4">"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org></font></td></tr>
<tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">Subject:</font></b></div></td><td width="457" valign="middle"><font size="4">[gpfsug-discuss] Maximum value for data replication?</font></td></tr></table><tt>Is there a maximum value for data replication in Spectrum Scale?<br><br>I have a number of nsd servers which have local storage and Id like each node to have a full copy of all the data in the file-system, say this value is 4, can I set replication to 4 for data and metadata and have each server have a full copy?<br><br>These are protocol nodes and multi cluster mount another file system (yes I know not supported) and the cesroot is in the remote file system. On several occasions where GPFS has wibbled a bit, this has caused issues with ces locks, so I was thinking of moving the cesroot to a local filesysyem which is replicated on the local ssds in the protocol nodes. I.e. Its a generally quiet file system as its only ces cluster config.<br><br>I assume if I stop protocols, rsync the data and then change to the new ces root, I should be able to get this working?<br><br>Thanks <br><br>Simon<br></tt><font color="#800080"><br>----- Message from "Kevin D Johnson" <kevindjo@us.ibm.com> on Tue, 30 Aug 2016 19:43:39 +0000 -----</font><table border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">To:</font></b></div></td><td width="213" valign="middle"><font size="4">gpfsug-discuss@spectrumscale.org</font></td></tr>
<tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">Subject:</font></b></div></td><td width="213" valign="middle"><font size="4">[gpfsug-discuss] greetings</font></td></tr></table><font size="4" face="Arial">I'm in Lab Services at IBM - just joining and happy to help any way I can.</font><br><font size="4" face="Arial"> </font><br><b><font face="Arial">Kevin D. Johnson, MBA, MAFM</font></b><br>Spectrum Computing, Senior Managing Consultant<br><font color="#696969"><br>IBM Certified Deployment Professional - Spectrum Scale V4.1.1<br>IBM Certified Deployment Professional - Cloud Object Storage V3.8<br>720.349.6199 - kevindjo@us.ibm.com</font><br><font size="2" face="Arial"> </font><br><font color="#800080"><br>----- Message from Lukas Hejtmanek <xhejtman@ics.muni.cz> on Tue, 30 Aug 2016 22:39:18 +0200 -----</font><table border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">To:</font></b></div></td><td width="263" valign="middle"><font size="4">gpfsug-discuss@spectrumscale.org</font></td></tr>
<tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">Subject:</font></b></div></td><td width="263" valign="middle"><font size="4">[gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8</font></td></tr></table><tt>Hello,<br><br>does it work for anyone? As of kernel 2.6.32-642, GPFS 3.5.0 (including the<br>latest patch 32) does start but does not mount and file system. The internal<br>mount cmd gets stucked. <br><br>-- <br>Lukáš Hejtmánek<br><br></tt><font color="#800080"><br>----- Message from "Kevin D Johnson" <kevindjo@us.ibm.com> on Tue, 30 Aug 2016 20:51:39 +0000 -----</font><table border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">To:</font></b></div></td><td width="287" valign="middle"><font size="4">gpfsug-discuss@spectrumscale.org</font></td></tr>
<tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">Subject:</font></b></div></td><td width="287" valign="middle"><font size="4">Re: [gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8</font></td></tr></table><font size="4" face="Arial">RHEL 6.8/2.6.32-642 requires 4.1.1.8 or 4.2.1. You can either go to 6.7 for GPFS 3.5 or bump it up to 7.0/7.1.<br> </font><br><font size="4" face="Arial">See Table 13, here:</font><br><a href="http://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html?view=kc#linuxq"><u><font size="4" color="#0000FF" face="Arial">http://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html?view=kc#linuxq</font></u></a><font size="4" face="Arial"><br> </font><br><b><font face="Arial">Kevin D. Johnson, MBA, MAFM</font></b><br>Spectrum Computing, Senior Managing Consultant<br><font color="#696969"><br>IBM Certified Deployment Professional - Spectrum Scale V4.1.1<br>IBM Certified Deployment Professional - Cloud Object Storage V3.8<br>720.349.6199 - kevindjo@us.ibm.com</font><br><font size="2" face="Arial"> </font><br><font size="4" face="Arial"> </font><br><font size="4" face="Arial"> </font><br><font size="4" face="Arial">----- Original message -----<br>From: Lukas Hejtmanek <xhejtman@ics.muni.cz><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss@spectrumscale.org<br>Cc:<br>Subject: [gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8<br>Date: Tue, Aug 30, 2016 4:39 PM<br> </font><br><tt>Hello,<br><br>does it work for anyone? As of kernel 2.6.32-642, GPFS 3.5.0 (including the<br>latest patch 32) does start but does not mount and file system. The internal<br>mount cmd gets stucked.<br><br>--<br>Lukáš Hejtmánek<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</tt><tt><u><font color="#0000FF"><br></font></u></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><u><font color="#0000FF">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></tt></a><font size="4" face="Arial"><br> </font><br><font size="4" face="Arial"> </font><br><font color="#800080"><br>----- Message from mark.bergman@uphs.upenn.edu on Tue, 30 Aug 2016 17:07:21 -0400 -----</font><table border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">To:</font></b></div></td><td width="391" valign="middle"><font size="4">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font></td></tr>
<tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">Subject:</font></b></div></td><td width="391" valign="middle"><font size="4">Re: [gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8</font></td></tr></table><tt>In the message dated: Tue, 30 Aug 2016 22:39:18 +0200,<br>The pithy ruminations from Lukas Hejtmanek on <br><[gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8> were:<br>=> Hello, <br><br>GPFS 3.5.0.[23..3-0] work for me under [CentOS|ScientificLinux] 6.8,<br>but at kernel 2.6.32-573 and lower.<br><br>I've found kernel bugs in blk_cloned_rq_check_limits() in later kernel<br>revs that caused multipath errors, resulting in GPFS being unable to<br>find all NSDs and mount the filesystem.<br><br>I am not updating to a newer kernel until I'm certain this is resolved.<br><br>I opened a bug with CentOS:<br><br> </tt><tt><a href="https://bugs.centos.org/view.php?id=10997">https://bugs.centos.org/view.php?id=10997</a></tt><tt><br><br>and began an extended discussion with the (RH & SUSE) developers of that<br>chunk of kernel code. I don't know if an upstream bug has been opened<br>by RH, but see:<br><br> </tt><tt><a href="https://patchwork.kernel.org/patch/9140337/">https://patchwork.kernel.org/patch/9140337/</a></tt><tt><br>=> <br>=> does it work for anyone? As of kernel 2.6.32-642, GPFS 3.5.0 (including the<br>=> latest patch 32) does start but does not mount and file system. The internal<br>=> mount cmd gets stucked. <br>=> <br>=> -- <br>=> Lukáš Hejtmánek <br><br><br>-- <br>Mark Bergman voice: 215-746-4061 <br>mark.bergman@uphs.upenn.edu fax: 215-614-0266<br></tt><tt><a href="http://www.cbica.upenn.edu/">http://www.cbica.upenn.edu/</a></tt><tt><br>IT Technical Director, Center for Biomedical Image Computing and Analytics<br>Department of Radiology University of Pennsylvania<br> PGP Key: </tt><tt><a href="http://www.cbica.upenn.edu/sbia/bergman">http://www.cbica.upenn.edu/sbia/bergman</a></tt><tt> <br><br></tt><font color="#800080"><br>----- Message from Lukas Hejtmanek <xhejtman@ics.muni.cz> on Wed, 31 Aug 2016 00:02:50 +0200 -----</font><table border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">To:</font></b></div></td><td width="561" valign="middle"><font size="4">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font></td></tr>
<tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">Subject:</font></b></div></td><td width="561" valign="middle"><font size="4">Re: [gpfsug-discuss] *New* IBM Spectrum Protect Whitepaper "Petascale Data Protection"</font></td></tr></table><tt>Hello,<br><br>On Mon, Aug 29, 2016 at 09:20:46AM +0200, Frank Kraemer wrote:<br>> Find the paper here:<br>> <br>> </tt><tt><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection</a></tt><tt><br><br>thank you for the paper, I appreciate it. <br><br>However, I wonder whether it could be extended a little. As it has the title<br>Petascale Data Protection, I think that in Peta scale, you have to deal with<br>millions (well rather hundreds of millions) of files you store in and this is<br>something where TSM does not scale well.<br><br>Could you give some hints:<br><br>On the backup site:<br>mmbackup takes ages for:<br>a) scan (try to scan 500M files even in parallel)<br>b) backup - what if 10 % of files get changed - backup process can be blocked<br>several days as mmbackup cannot run in several instances on the same file<br>system, so you have to wait until one run of mmbackup finishes. How long could<br>it take at petascale?<br><br>On the restore site:<br>how can I restore e.g. 40 millions of file efficiently? dsmc restore '/path/*'<br>runs into serious troubles after say 20M files (maybe wrong internal<br>structures used), however, scanning 1000 more files takes several minutes<br>resulting the dsmc restore never reaches that 40M files. <br><br>using filelists the situation is even worse. I run dsmc restore -filelist<br>with a filelist consisting of 2.4M files. Running for *two* days without<br>restoring even a single file. dsmc is consuming 100 % CPU. <br><br>So any hints addressing these issues with really large number of files would<br>be even more appreciated.<br><br>-- <br>Lukáš Hejtmánek<br><br></tt><font color="#800080"><br>----- Message from Sven Oehme <oehmes@gmail.com> on Tue, 30 Aug 2016 16:24:59 -0700 -----</font><table border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">To:</font></b></div></td><td width="561" valign="middle"><font size="4">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font></td></tr>
<tr valign="top"><td width="55" valign="middle"><div align="right"><b><font size="4">Subject:</font></b></div></td><td width="561" valign="middle"><font size="4">Re: [gpfsug-discuss] *New* IBM Spectrum Protect Whitepaper "Petascale Data Protection"</font></td></tr></table><font size="4">so lets start with some simple questions. </font><br><br><font size="4">when you say mmbackup takes ages, what version of gpfs code are you running ? </font><br><font size="4">how do you execute the mmbackup command ? exact parameters would be useful . </font><br><font size="4">what HW are you using for the metadata disks ? </font><br><font size="4">how much capacity (df -h) and how many inodes (df -i) do you have in the filesystem you try to backup ?</font><br><br><font size="4">sven</font><br><br><br><font size="4">On Tue, Aug 30, 2016 at 3:02 PM, Lukas Hejtmanek <</font><a href="mailto:xhejtman@ics.muni.cz" target="_blank"><u><font size="4" color="#0000FF">xhejtman@ics.muni.cz</font></u></a><font size="4">> wrote:</font><ul><ul><font size="4">Hello,<br><br>On Mon, Aug 29, 2016 at 09:20:46AM +0200, Frank Kraemer wrote:<br>> Find the paper here:<br>><br>> </font><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection" target="_blank"><u><font size="4" color="#0000FF">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection</font></u></a><font size="4"><br><br>thank you for the paper, I appreciate it.<br><br>However, I wonder whether it could be extended a little. As it has the title<br>Petascale Data Protection, I think that in Peta scale, you have to deal with<br>millions (well rather hundreds of millions) of files you store in and this is<br>something where TSM does not scale well.<br><br>Could you give some hints:<br><br>On the backup site:<br>mmbackup takes ages for:<br>a) scan (try to scan 500M files even in parallel)<br>b) backup - what if 10 % of files get changed - backup process can be blocked<br>several days as mmbackup cannot run in several instances on the same file<br>system, so you have to wait until one run of mmbackup finishes. How long could<br>it take at petascale?<br><br>On the restore site:<br>how can I restore e.g. 40 millions of file efficiently? dsmc restore '/path/*'<br>runs into serious troubles after say 20M files (maybe wrong internal<br>structures used), however, scanning 1000 more files takes several minutes<br>resulting the dsmc restore never reaches that 40M files.<br><br>using filelists the situation is even worse. I run dsmc restore -filelist<br>with a filelist consisting of 2.4M files. Running for *two* days without<br>restoring even a single file. dsmc is consuming 100 % CPU.<br><br>So any hints addressing these issues with really large number of files would<br>be even more appreciated.</font><font size="4" color="#888888"><br><br>--<br>Lukáš Hejtmánek<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org/" target="_blank"><u><font size="4" color="#0000FF">spectrumscale.org</font></u></a><u><font size="4" color="#0000FF"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><u><font size="4" color="#0000FF">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a></ul></ul><tt>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br></tt><br><br><BR>
</body></html>