<html><body>
<p><font size="2" face="sans-serif">>></font><tt><font size="2">Is there a maximum value for data replication in Spectrum Scale?</font></tt><br>
<font size="2" face="sans-serif">The maximum value for replication is 3. </font><br>
<font size="2" face="sans-serif"><br>
<br>
Steve Duersch<br>
Spectrum Scale RAID<br>
845-433-7902<br>
IBM Poughkeepsie, New York<br>
<br>
<br>
</font><br>
<br>
<img width="16" height="16" src="cid:1__=0ABB0AB3DFD67DBA8f9e8a93df938@us.ibm.com" border="0" alt="Inactive hide details for gpfsug-discuss-request---08/30/2016 07:25:24 PM---Send gpfsug-discuss mailing list submissions to gp"><font size="2" color="#424282" face="sans-serif">gpfsug-discuss-request---08/30/2016 07:25:24 PM---Send gpfsug-discuss mailing list submissions to gpfsug-discuss@spectrumscale.org</font><br>
<br>
<font size="1" color="#5F5F5F" face="sans-serif">From: </font><font size="1" face="sans-serif">gpfsug-discuss-request@spectrumscale.org</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">To: </font><font size="1" face="sans-serif">gpfsug-discuss@spectrumscale.org</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Date: </font><font size="1" face="sans-serif">08/30/2016 07:25 PM</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Subject: </font><font size="1" face="sans-serif">gpfsug-discuss Digest, Vol 55, Issue 55</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Sent by: </font><font size="1" face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<tt><font size="2">Send gpfsug-discuss mailing list submissions to<br>
gpfsug-discuss@spectrumscale.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
</font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br>
or, via email, send a message with subject or body 'help' to<br>
gpfsug-discuss-request@spectrumscale.org<br>
<br>
You can reach the person managing the list at<br>
gpfsug-discuss-owner@spectrumscale.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Maximum value for data replication?<br>
(Simon Thompson (Research Computing - IT Services))<br>
2. greetings (Kevin D Johnson)<br>
3. GPFS 3.5.0 on RHEL 6.8 (Lukas Hejtmanek)<br>
4. Re: GPFS 3.5.0 on RHEL 6.8 (Kevin D Johnson)<br>
5. Re: GPFS 3.5.0 on RHEL 6.8 (mark.bergman@uphs.upenn.edu)<br>
6. Re: *New* IBM Spectrum Protect Whitepaper "Petascale Data<br>
Protection" (Lukas Hejtmanek)<br>
7. Re: *New* IBM Spectrum Protect Whitepaper "Petascale Data<br>
Protection" (Sven Oehme)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 30 Aug 2016 19:09:05 +0000<br>
From: "Simon Thompson (Research Computing - IT Services)"<br>
<S.J.Thompson@bham.ac.uk><br>
To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br>
Subject: [gpfsug-discuss] Maximum value for data replication?<br>
Message-ID:<br>
<CF45EE16DEF2FE4B9AA7FF2B6EE26545F5813FFC@EX13.adf.bham.ac.uk><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Is there a maximum value for data replication in Spectrum Scale?<br>
<br>
I have a number of nsd servers which have local storage and Id like each node to have a full copy of all the data in the file-system, say this value is 4, can I set replication to 4 for data and metadata and have each server have a full copy?<br>
<br>
These are protocol nodes and multi cluster mount another file system (yes I know not supported) and the cesroot is in the remote file system. On several occasions where GPFS has wibbled a bit, this has caused issues with ces locks, so I was thinking of moving the cesroot to a local filesysyem which is replicated on the local ssds in the protocol nodes. I.e. Its a generally quiet file system as its only ces cluster config.<br>
<br>
I assume if I stop protocols, rsync the data and then change to the new ces root, I should be able to get this working?<br>
<br>
Thanks <br>
<br>
Simon<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Tue, 30 Aug 2016 19:43:39 +0000<br>
From: "Kevin D Johnson" <kevindjo@us.ibm.com><br>
To: gpfsug-discuss@spectrumscale.org<br>
Subject: [gpfsug-discuss] greetings<br>
Message-ID:<br>
<OFBE10B787.D9EE56E3-ON0025801F.006C39E8-0025801F.006C5E11@notes.na.collabserv.com><br>
<br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
An HTML attachment was scrubbed...<br>
URL: <</font></tt><tt><font size="2"><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/5a2e22a3/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/5a2e22a3/attachment-0001.html</a></font></tt><tt><font size="2">><br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Tue, 30 Aug 2016 22:39:18 +0200<br>
From: Lukas Hejtmanek <xhejtman@ics.muni.cz><br>
To: gpfsug-discuss@spectrumscale.org<br>
Subject: [gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8<br>
Message-ID: <20160830203917.qptfgqvlmdbzu6wr@ics.muni.cz><br>
Content-Type: text/plain; charset=iso-8859-2<br>
<br>
Hello,<br>
<br>
does it work for anyone? As of kernel 2.6.32-642, GPFS 3.5.0 (including the<br>
latest patch 32) does start but does not mount and file system. The internal<br>
mount cmd gets stucked. <br>
<br>
-- <br>
Luk?? Hejtm?nek<br>
<br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Tue, 30 Aug 2016 20:51:39 +0000<br>
From: "Kevin D Johnson" <kevindjo@us.ibm.com><br>
To: gpfsug-discuss@spectrumscale.org<br>
Subject: Re: [gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8<br>
Message-ID:<br>
<OF60CB398A.7B2AF5FF-ON0025801F.007282A3-0025801F.0072979C@notes.na.collabserv.com><br>
<br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
An HTML attachment was scrubbed...<br>
URL: <</font></tt><tt><font size="2"><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/341d5e11/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/341d5e11/attachment-0001.html</a></font></tt><tt><font size="2">><br>
<br>
------------------------------<br>
<br>
Message: 5<br>
Date: Tue, 30 Aug 2016 17:07:21 -0400<br>
From: mark.bergman@uphs.upenn.edu<br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8<br>
Message-ID: <24437-1472591241.445832@bR6O.TofS.917u><br>
Content-Type: text/plain; charset="UTF-8"<br>
<br>
In the message dated: Tue, 30 Aug 2016 22:39:18 +0200,<br>
The pithy ruminations from Lukas Hejtmanek on <br>
<[gpfsug-discuss] GPFS 3.5.0 on RHEL 6.8> were:<br>
=> Hello, <br>
<br>
GPFS 3.5.0.[23..3-0] work for me under [CentOS|ScientificLinux] 6.8,<br>
but at kernel 2.6.32-573 and lower.<br>
<br>
I've found kernel bugs in blk_cloned_rq_check_limits() in later kernel<br>
revs that caused multipath errors, resulting in GPFS being unable to<br>
find all NSDs and mount the filesystem.<br>
<br>
I am not updating to a newer kernel until I'm certain this is resolved.<br>
<br>
I opened a bug with CentOS:<br>
<br>
</font></tt><tt><font size="2"><a href="https://bugs.centos.org/view.php?id=10997">https://bugs.centos.org/view.php?id=10997</a></font></tt><tt><font size="2"><br>
<br>
and began an extended discussion with the (RH & SUSE) developers of that<br>
chunk of kernel code. I don't know if an upstream bug has been opened<br>
by RH, but see:<br>
<br>
</font></tt><tt><font size="2"><a href="https://patchwork.kernel.org/patch/9140337/">https://patchwork.kernel.org/patch/9140337/</a></font></tt><tt><font size="2"><br>
=> <br>
=> does it work for anyone? As of kernel 2.6.32-642, GPFS 3.5.0 (including the<br>
=> latest patch 32) does start but does not mount and file system. The internal<br>
=> mount cmd gets stucked. <br>
=> <br>
=> -- <br>
=> Luk?? Hejtm?nek <br>
<br>
<br>
-- <br>
Mark Bergman voice: 215-746-4061 <br>
mark.bergman@uphs.upenn.edu fax: 215-614-0266<br>
</font></tt><tt><font size="2"><a href="http://www.cbica.upenn.edu/">http://www.cbica.upenn.edu/</a></font></tt><tt><font size="2"><br>
IT Technical Director, Center for Biomedical Image Computing and Analytics<br>
Department of Radiology University of Pennsylvania<br>
PGP Key: </font></tt><tt><font size="2"><a href="http://www.cbica.upenn.edu/sbia/bergman">http://www.cbica.upenn.edu/sbia/bergman</a></font></tt><tt><font size="2"> <br>
<br>
<br>
------------------------------<br>
<br>
Message: 6<br>
Date: Wed, 31 Aug 2016 00:02:50 +0200<br>
From: Lukas Hejtmanek <xhejtman@ics.muni.cz><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] *New* IBM Spectrum Protect Whitepaper<br>
"Petascale Data Protection"<br>
Message-ID: <20160830220250.yt6r7gvfq7rlvtcs@ics.muni.cz><br>
Content-Type: text/plain; charset=iso-8859-2<br>
<br>
Hello,<br>
<br>
On Mon, Aug 29, 2016 at 09:20:46AM +0200, Frank Kraemer wrote:<br>
> Find the paper here:<br>
> <br>
> </font></tt><tt><font size="2"><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection</a></font></tt><tt><font size="2"><br>
<br>
thank you for the paper, I appreciate it. <br>
<br>
However, I wonder whether it could be extended a little. As it has the title<br>
Petascale Data Protection, I think that in Peta scale, you have to deal with<br>
millions (well rather hundreds of millions) of files you store in and this is<br>
something where TSM does not scale well.<br>
<br>
Could you give some hints:<br>
<br>
On the backup site:<br>
mmbackup takes ages for:<br>
a) scan (try to scan 500M files even in parallel)<br>
b) backup - what if 10 % of files get changed - backup process can be blocked<br>
several days as mmbackup cannot run in several instances on the same file<br>
system, so you have to wait until one run of mmbackup finishes. How long could<br>
it take at petascale?<br>
<br>
On the restore site:<br>
how can I restore e.g. 40 millions of file efficiently? dsmc restore '/path/*'<br>
runs into serious troubles after say 20M files (maybe wrong internal<br>
structures used), however, scanning 1000 more files takes several minutes<br>
resulting the dsmc restore never reaches that 40M files. <br>
<br>
using filelists the situation is even worse. I run dsmc restore -filelist<br>
with a filelist consisting of 2.4M files. Running for *two* days without<br>
restoring even a single file. dsmc is consuming 100 % CPU. <br>
<br>
So any hints addressing these issues with really large number of files would<br>
be even more appreciated.<br>
<br>
-- <br>
Luk?? Hejtm?nek<br>
<br>
<br>
------------------------------<br>
<br>
Message: 7<br>
Date: Tue, 30 Aug 2016 16:24:59 -0700<br>
From: Sven Oehme <oehmes@gmail.com><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] *New* IBM Spectrum Protect Whitepaper<br>
"Petascale Data Protection"<br>
Message-ID:<br>
<CALssuR1qWo2Y5adUUZJtLgkNPUYztWpYP0XhNLTjBOG5352qjg@mail.gmail.com><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
so lets start with some simple questions.<br>
<br>
when you say mmbackup takes ages, what version of gpfs code are you running<br>
?<br>
how do you execute the mmbackup command ? exact parameters would be useful<br>
.<br>
what HW are you using for the metadata disks ?<br>
how much capacity (df -h) and how many inodes (df -i) do you have in the<br>
filesystem you try to backup ?<br>
<br>
sven<br>
<br>
<br>
On Tue, Aug 30, 2016 at 3:02 PM, Lukas Hejtmanek <xhejtman@ics.muni.cz><br>
wrote:<br>
<br>
> Hello,<br>
><br>
> On Mon, Aug 29, 2016 at 09:20:46AM +0200, Frank Kraemer wrote:<br>
> > Find the paper here:<br>
> ><br>
> > </font></tt><tt><font size="2"><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/</a></font></tt><tt><font size="2"><br>
> Tivoli%20Storage%20Manager/page/Petascale%20Data%20Protection<br>
><br>
> thank you for the paper, I appreciate it.<br>
><br>
> However, I wonder whether it could be extended a little. As it has the<br>
> title<br>
> Petascale Data Protection, I think that in Peta scale, you have to deal<br>
> with<br>
> millions (well rather hundreds of millions) of files you store in and this<br>
> is<br>
> something where TSM does not scale well.<br>
><br>
> Could you give some hints:<br>
><br>
> On the backup site:<br>
> mmbackup takes ages for:<br>
> a) scan (try to scan 500M files even in parallel)<br>
> b) backup - what if 10 % of files get changed - backup process can be<br>
> blocked<br>
> several days as mmbackup cannot run in several instances on the same file<br>
> system, so you have to wait until one run of mmbackup finishes. How long<br>
> could<br>
> it take at petascale?<br>
><br>
> On the restore site:<br>
> how can I restore e.g. 40 millions of file efficiently? dsmc restore<br>
> '/path/*'<br>
> runs into serious troubles after say 20M files (maybe wrong internal<br>
> structures used), however, scanning 1000 more files takes several minutes<br>
> resulting the dsmc restore never reaches that 40M files.<br>
><br>
> using filelists the situation is even worse. I run dsmc restore -filelist<br>
> with a filelist consisting of 2.4M files. Running for *two* days without<br>
> restoring even a single file. dsmc is consuming 100 % CPU.<br>
><br>
> So any hints addressing these issues with really large number of files<br>
> would<br>
> be even more appreciated.<br>
><br>
> --<br>
> Luk?? Hejtm?nek<br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at spectrumscale.org<br>
> </font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <</font></tt><tt><font size="2"><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/d9b3fb68/attachment.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20160830/d9b3fb68/attachment.html</a></font></tt><tt><font size="2">><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
</font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 55, Issue 55<br>
**********************************************<br>
<br>
</font></tt><br>
<br>
</body></html>