<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Years back I did run a trial (to buy) software solution on OSX to address this issue. It worked! It was not cheap and they probably no longer support it anyway. It might have been from a company called Group Logic.
<br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">I would suggest not exposing HSM enabled file systems (in particular ones using tape on the back end) to your general CIFS (or even) GPFS/NFS clients. It produced years (2011-2015 of frustration with recall storms that
made everyone mad. If someone else had success, I think we'd all like to know how they did it....but we gave up on that. In the end I would suggest setting up an explicit archive location using/HSM tape (or low cost, high densisty disk) that is not pointing
to your traditional GPFS/CIFS/NFS clients that users must deliberately access (think portal) to check in/out cold data that they can stage to their primary workspace. It is possible you considered this idea or some variation of it anyway and rejected it for
good reason (e.g. more pain for the users to stage data over from cold storage to primary workspacec).<br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255); font-family:Calibri,Arial,Helvetica,sans-serif,Helvetica,EmojiFont,'Apple Color Emoji','Segoe UI Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols,Helvetica,EmojiFont,'Apple Color Emoji','Segoe UI Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols">
<p>Bill Pappas</p>
<p>901-619-0585</p>
<p>bpappas@dstonline.com</p>
<p><b style="color:rgb(33,33,33); font-family:Calibri,sans-serif; font-size:11pt"></b><img class="EmojiInsert" alt="1466780990050_DSTlogo.png" data-outlook-trace="F:1|T:1" src="cid:bd967507-931a-4b34-aad9-e399483ceeeb"><br>
</p>
<p><br>
</p>
<p></p>
<p><br>
</p>
<p class="x_MsoNormal" style="margin:0in 0in 0.0001pt; font-size:11pt; font-family:Calibri,sans-serif; color:rgb(33,33,33)">
<b></b></p>
<div><b><br>
</b></div>
</div>
</div>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of gpfsug-discuss-request@spectrumscale.org
<gpfsug-discuss-request@spectrumscale.org><br>
<b>Sent:</b> Tuesday, July 10, 2018 9:50 AM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org<br>
<b>Subject:</b> gpfsug-discuss Digest, Vol 78, Issue 32</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Send gpfsug-discuss mailing list submissions to<br>
gpfsug-discuss@spectrumscale.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" id="LPlnk319065" class="OWAAutoLink" previewremoved="true">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
gpfsug-discuss-request@spectrumscale.org<br>
<br>
You can reach the person managing the list at<br>
gpfsug-discuss-owner@spectrumscale.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: preventing HSM tape recall storms (Jonathan Buzzard)<br>
2. Re: What NSDs does a file have blocks on? (Marc A Kaplan)<br>
3. Re: High I/O wait times (Jonathan Buzzard)<br>
4. Re: Allocation map limits - any way around this? (Uwe Falke)<br>
5. Same file opened by many nodes / processes (Peter Childs)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 10 Jul 2018 14:00:48 +0100<br>
From: Jonathan Buzzard <jonathan.buzzard@strath.ac.uk><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] preventing HSM tape recall storms<br>
Message-ID: <1531227648.26036.139.camel@strath.ac.uk><br>
Content-Type: text/plain; charset="UTF-8"<br>
<br>
On Mon, 2018-07-09 at 14:57 -0400, Frederick Stock wrote:<br>
> Another option is to request Apple to support the OFFLINE flag in the<br>
> SMB protocol. ?The more Mac customers making such a request (I have<br>
> asked others to do likewise) might convince Apple to add this<br>
> checking to their SMB client.<br>
> <br>
<br>
And we have a winner. The only workable solution is to get Apple to<br>
Finder to support the OFFLINE flag. However good luck getting Apple to<br>
actually do anything.<br>
<br>
An alternative approach might be to somehow detect the client<br>
connecting is running MacOS and prohibit recalls for them. However I am<br>
not sure the Samba team would be keen on accepting such patches unless<br>
it could be done in say VFS module.<br>
<br>
JAB.<br>
<br>
-- <br>
Jonathan A. Buzzard?????????????????????????Tel: +44141-5483420<br>
HPC System Administrator, ARCHIE-WeSt.<br>
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG<br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Tue, 10 Jul 2018 09:08:45 -0400<br>
From: "Marc A Kaplan" <makaplan@us.ibm.com><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] What NSDs does a file have blocks on?<br>
Message-ID:<br>
<OF5287542C.D657BE67-ON852582C6.00479067-852582C6.004836AF@notes.na.collabserv.com><br>
<br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
As long as we're giving hints...<br>
Seems tsdbfs has several subcommands that might be helpful.<br>
I like "inode"<br>
But there's also "listda"<br>
Subcommand "desc" will show you the structure of the file system under <br>
"disks:" you will see which disk numbers are which NSDs.<br>
<br>
Have fun, but DO NOT use the any of the *patch* subcommands!<br>
<br>
<br>
<br>
From: Simon Thompson <S.J.Thompson@bham.ac.uk><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Date: 07/09/2018 05:21 PM<br>
Subject: Re: [gpfsug-discuss] What NSDs does a file have blocks on?<br>
Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>
<br>
<br>
<br>
I was going to say something like that ? e.g.<br>
<br>
blockaddr 563148261<br>
Inode 563148261 snap 0 offset 0 N=2 1:45255923200 13:59403784320<br>
1: and 13: in the output are the NSD disk devices for inode 563148261<br>
<br>
Simon<br>
<br>
From: <gpfsug-discuss-bounces@spectrumscale.org> on behalf of <br>
"makaplan@us.ibm.com" <makaplan@us.ibm.com><br>
Reply-To: "gpfsug-discuss@spectrumscale.org" <br>
<gpfsug-discuss@spectrumscale.org><br>
Date: Monday, 9 July 2018 at 22:04<br>
To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] What NSDs does a file have blocks on?<br>
<br>
(psss... ) tsdbfs <br>
<br>
Not responsible for anything bad that happens...!<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" id="LPlnk987372" class="OWAAutoLink" previewremoved="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
<br>
<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20180710/0ebdef35/attachment-0001.html" id="LPlnk890167" class="OWAAutoLink" previewremoved="true">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20180710/0ebdef35/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Tue, 10 Jul 2018 14:12:02 +0100<br>
From: Jonathan Buzzard <jonathan.buzzard@strath.ac.uk><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] High I/O wait times<br>
Message-ID: <1531228322.26036.143.camel@strath.ac.uk><br>
Content-Type: text/plain; charset="UTF-8"<br>
<br>
On Mon, 2018-07-09 at 21:44 +0000, Buterbaugh, Kevin L wrote:<br>
<br>
[SNIP]<br>
<br>
> Interestingly enough, one user showed up waaaayyyyyy more often than<br>
> anybody else. ?And many times she was on a node with only one other<br>
> user who we know doesn?t access the GPFS filesystem and other times<br>
> she was the only user on the node. ?<br>
> <br>
<br>
I have seen on our old HPC system which had been running fine for three<br>
years a particular user with a particular piece of software with<br>
presumably a particular access pattern trigger a firmware bug in a SAS<br>
drive (local disk to the node) that caused it to go offline (dead to<br>
the world and power/presence LED off) and only a power cycle of the<br>
node would bring it back.<br>
<br>
At first we through the drives where failing, because what the hell,<br>
but in the end a firmware update to the drives and they where fine.<br>
<br>
The moral of the story is don't rule out wacky access patterns from a<br>
single user causing problems.<br>
<br>
JAB.<br>
<br>
-- <br>
Jonathan A. Buzzard?????????????????????????Tel: +44141-5483420<br>
HPC System Administrator, ARCHIE-WeSt.<br>
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG<br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Tue, 10 Jul 2018 16:28:57 +0200<br>
From: "Uwe Falke" <UWEFALKE@de.ibm.com><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] Allocation map limits - any way around<br>
this?<br>
Message-ID:<br>
<OF9FBA4DCC.1C355D67-ONC12582C6.004F6441-C12582C6.004F8DB8@notes.na.collabserv.com><br>
<br>
Content-Type: text/plain; charset="ISO-8859-1"<br>
<br>
Hi Bob, <br>
you sure the first added NSD was 1 TB? As often as i created a FS, the max <br>
NSD size was way larger than the one I added initially , not just the <br>
fourfold. <br>
<br>
Mit freundlichen Gr??en / Kind regards<br>
<br>
<br>
Dr. Uwe Falke<br>
<br>
IT Specialist<br>
High Performance Computing Services / Integrated Technology Services / <br>
Data Center Services<br>
-------------------------------------------------------------------------------------------------------------------------------------------<br>
IBM Deutschland<br>
Rathausstr. 7<br>
09111 Chemnitz<br>
Phone: +49 371 6978 2165<br>
Mobile: +49 175 575 2877<br>
E-Mail: uwefalke@de.ibm.com<br>
-------------------------------------------------------------------------------------------------------------------------------------------<br>
IBM Deutschland Business & Technology Services GmbH / Gesch?ftsf?hrung: <br>
Thomas Wolter, Sven Schoo?<br>
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, <br>
HRB 17122 <br>
<br>
<br>
<br>
<br>
From: "Oesterlin, Robert" <Robert.Oesterlin@nuance.com><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Date: 10/07/2018 13:59<br>
Subject: [gpfsug-discuss] Allocation map limits - any way around <br>
this?<br>
Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>
<br>
<br>
<br>
File system was originally created with 1TB NSDs (4) and I want to move it <br>
to one 5TB NSD. Any way around this error?<br>
<br>
mmadddisk fs1 -F new.nsd <br>
<br>
The following disks of proserv will be formatted on node srv-gpfs06:<br>
stor1v5tb85: size 5242880 MB<br>
Extending Allocation Map<br>
Disk stor1v5tb85 cannot be added to storage pool Plevel1.<br>
Allocation map cannot accommodate disks larger than 4194555 MB.<br>
Checking Allocation Map for storage pool Plevel1<br>
mmadddisk: tsadddisk failed.<br>
Verifying file system configuration information ...<br>
mmadddisk: Propagating the cluster configuration data to all<br>
affected nodes. This is an asynchronous process.<br>
mmadddisk: Command failed. Examine previous error messages to determine <br>
cause.<br>
<br>
<br>
Bob Oesterlin<br>
Sr Principal Storage Engineer, Nuance<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" id="LPlnk991178" class="OWAAutoLink" previewremoved="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 5<br>
Date: Tue, 10 Jul 2018 14:50:54 +0000<br>
From: Peter Childs <p.childs@qmul.ac.uk><br>
To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br>
Subject: [gpfsug-discuss] Same file opened by many nodes / processes<br>
Message-ID:<br>
<4e038c492713f418242be208532e112f8ea50a9f.camel@qmul.ac.uk><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
We have an situation where the same file is being read by around 5000<br>
"jobs" this is an array job in uge with a tc set, so the file in<br>
question is being opened by about 100 processes/jobs at the same time.<br>
<br>
Its a ~200GB file so copying the file locally first is not an easy<br>
answer, and these jobs are causing issues with mmbackup scanning the<br>
file system, in that the scan is taking 3 hours instead of the normal<br>
40-60 minutes.<br>
<br>
This is read only access to the file, I don't know the specifics about<br>
the job.<br>
<br>
It looks like the metanode is moving around a fair amount (given what I<br>
can see from mmfsadm saferdump file)<br>
<br>
I'm wondering if we there is anything we can do to improve things or<br>
that can be tuned within GPFS, I'm don't think we have an issue with<br>
token management, but would increasing maxFileToCache on our token<br>
manager node help say?<br>
<br>
Is there anything else I should look at, to try and attempt to allow<br>
GPFS to share this file better.<br>
<br>
Thanks in advance<br>
<br>
Peter Childs<br>
<br>
-- <br>
Peter Childs<br>
ITS Research Storage<br>
Queen Mary, University of London<br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" id="LPlnk266295" class="OWAAutoLink" previewremoved="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 78, Issue 32<br>
**********************************************<br>
</div>
</span></font></div>
</div>
</div>
</body>
</html>