<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Dear Radhika,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">In the early days of AFM and at two separate GPFS UK User Group meetings, I discussed AFM cache chaining with IBM technical people
plus at least one developer. My distinct recollection of the outcome was that cache chaining was supported.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Nevertheless, the difference between what my memory tells me and what is being reported now is irrelevant. We are stuck with large
volumes of data being migrated in this fashion, so there is clearly a customer use case for chaining AFM caches.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">It would be much more helpful if IBM could take on this case and look at the suspected bug that’s been chased out here.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Real world observation in the field is that queuing large numbers of metadata updates on the MDS itself causes this crash, whereas
issuing the updates from another node in the cache cluster adds to the MDS queue and the crash does not happen. My guess is that there is a bug whereby daemon-local additions to the MDS queue aren’t handled correctly (further speculation is that there is a
memory leak for local MDS operations, but that needs more testing which I don’t have time for – perhaps IBM could try it out?); however, when a metadata update operation is sent through an RPC from another node, it is added to the queue and handled correctly.
A workaround, if you will.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Other minor observations here are that the further down the chain of caches you are, the larger you should set afmDisconnectTimeout
as any intermediate cache recovery time needs to be taken into account following a disconnect event. Initially, this was slightly counterintuitive because caches B and C as described below are connected over multiple IB interfaces and shouldn’t disconnect
except when there’s some other failure. Conversely, the connection between cache A and B is over a very flaky wide area network and although we’ve managed to tune out a lot of the problems introduced by high and variable latency, the view of cache A from cache
B’s perspective still sometimes gets suspended.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">The failure observed above doesn’t really feel like it’s an artefact of cascading caches, but a bug in MDS code as described. Sharing
background information about the cascading cache setup was in the spirit of the mailing list and might have led IBM or other customers attempting this kind of setup to share some of their experiences.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hope you can help.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Luke.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>Radhika A Parameswaran<br>
<b>Sent:</b> 28 July 2016 06:43<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org<br>
<b>Subject:</b> [gpfsug-discuss] Re. AFM Crashing the MDS<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Luke,</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">AFM is not tested for cascading configurations, this is getting added into the documentation for 4.2.1:</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">"</span>Cascading of AFM caches is not tested.<span style="font-size:10.0pt;font-family:"Arial",sans-serif">"</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
<br>
<br>
Thanks and Regards<br>
Radhika<br>
<br>
</span><br>
<br>
<br>
<br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">From: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif"><a href="mailto:gpfsug-discuss-request@spectrumscale.org">gpfsug-discuss-request@spectrumscale.org</a></span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">To: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif"><a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a></span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Date: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">07/27/2016 04:30 PM</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Subject: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">gpfsug-discuss Digest, Vol 54, Issue 59</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Sent by: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></span><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="3" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<br>
<tt><span style="font-size:10.0pt">Send gpfsug-discuss mailing list submissions to</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt> <a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a></tt><br>
<br>
<tt>To subscribe or unsubscribe via the World Wide Web, visit</tt><br>
<tt> </tt></span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>or, via email, send a message with subject or body 'help' to</tt><br>
<tt> <a href="mailto:gpfsug-discuss-request@spectrumscale.org">gpfsug-discuss-request@spectrumscale.org</a></tt><br>
<br>
<tt>You can reach the person managing the list at</tt><br>
<tt> <a href="mailto:gpfsug-discuss-owner@spectrumscale.org">gpfsug-discuss-owner@spectrumscale.org</a></tt><br>
<br>
<tt>When replying, please edit your Subject line so it is more specific</tt><br>
<tt>than "Re: Contents of gpfsug-discuss digest..."</tt><br>
<br>
<br>
<tt>Today's Topics:</tt><br>
<br>
<tt> 1. AFM Crashing the MDS (Luke Raimbach)</tt><br>
<br>
<br>
<tt>----------------------------------------------------------------------</tt><br>
<br>
<tt>Message: 1</tt><br>
<tt>Date: Tue, 26 Jul 2016 14:17:35 +0000</tt><br>
<tt>From: Luke Raimbach <<a href="mailto:Luke.Raimbach@crick.ac.uk">Luke.Raimbach@crick.ac.uk</a>></tt><br>
<tt>To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>></tt><br>
<tt>Subject: [gpfsug-discuss] AFM Crashing the MDS</tt><br>
<tt>Message-ID:</tt><br>
<tt> <<a href="mailto:AMSPR03MB27605D717C5500D86F6ADEFB00E0@AMSPR03MB276.eurprd03.prod.outlook.com">AMSPR03MB27605D717C5500D86F6ADEFB00E0@AMSPR03MB276.eurprd03.prod.outlook.com</a>></tt><br>
<tt> </tt><br>
<tt>Content-Type: text/plain; charset="utf-8"</tt><br>
<br>
<tt>Hi All,</tt><br>
<br>
<tt>Anyone seen GPFS barf like this before? I'll explain the setup:</tt><br>
<br>
<tt>RO AFM cache on remote site (cache A) for reading remote datasets quickly,</tt><br>
<tt>LU AFM cache at destination site (cache B) for caching data from cache A (has a local compute cluster mounting this over multi-cluster),</tt><br>
<tt>IW AFM cache at destination site (cache C) for presenting cache B over NAS protocols,</tt><br>
<br>
<tt>Reading files in cache C should pull data from the remote source through cache A->B->C</tt><br>
<br>
<tt>Modifying files in cache C should pull data into cache B and then break the cache relationship for that file, converting it to a local copy. Those modifications should include metadata updates (e.g. chown).</tt><br>
<br>
<tt>To speed things up we prefetch files into cache B for datasets which are undergoing migration and have entered a read-only state at the source.</tt><br>
<br>
<tt>When issuing chown on a directory in cache C containing ~4.5million files, the MDS for the AFM cache C crashes badly:</tt><br>
<br>
<br>
<tt>Tue Jul 26 13:28:52.487 2016: [X] logAssertFailed: addr.isReserved() || addr.getClusterIdx() == clusterIdx</tt><br>
<tt>Tue Jul 26 13:28:52.488 2016: [X] return code 0, reason code 1, log record tag 0</tt><br>
<tt>Tue Jul 26 13:28:53.392 2016: [X] *** Assert exp(addr.isReserved() || addr.getClusterIdx() == clusterIdx) in line 1936 of file /project/sprelbmd0/build/rbmd0s003a/src/avs/fs/mmfs/ts/cfgmgr/cfgmgr.h</tt><br>
<tt>Tue Jul 26 13:28:53.393 2016: [E] *** Traceback:</tt><br>
<tt>Tue Jul 26 13:28:53.394 2016: [E] 2:0x7F6DC95444A6 logAssertFailed + 0x2D6 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:53.395 2016: [E] 3:0x7F6DC95C7EF4 ClusterConfiguration::getGatewayNewHash(DiskUID, unsigned int, NodeAddr*) + 0x4B4 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:53.396 2016: [E] 4:0x7F6DC95C8031 ClusterConfiguration::getGatewayNode(DiskUID, unsigned int, NodeAddr, NodeAddr*, unsigned int) + 0x91 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:53.397 2016: [E] 5:0x7F6DC9DC7126 SFSPcache(StripeGroup*, FileUID, int, int, void*, int, voidXPtr*, int*) + 0x346 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:53.398 2016: [E] 6:0x7F6DC9332494 HandleMBPcache(MBPcacheParms*) + 0xB4 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:53.399 2016: [E] 7:0x7F6DC90A4A53 Mailbox::msgHandlerBody(void*) + 0x3C3 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:53.400 2016: [E] 8:0x7F6DC908BC06 Thread::callBody(Thread*) + 0x46 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:53.401 2016: [E] 9:0x7F6DC907A0D2 Thread::callBodyWrapper(Thread*) + 0xA2 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:53.402 2016: [E] 10:0x7F6DC87A3AA1 start_thread + 0xD1 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:53.403 2016: [E] 11:0x7F6DC794A93D clone + 0x6D at ??:0</tt><br>
<tt>mmfsd: /project/sprelbmd0/build/rbmd0s003a/src/avs/fs/mmfs/ts/cfgmgr/cfgmgr.h:1936: void logAssertFailed(UInt32, const char*, UInt32, Int32, Int32, UInt32, const char*, const char*): Assertion `addr.isReserved() || addr.getClusterIdx() == clusterIdx' failed.</tt><br>
<tt>Tue Jul 26 13:28:53.404 2016: [N] Signal 6 at location 0x7F6DC7894625 in process 6262, link reg 0xFFFFFFFFFFFFFFFF.</tt><br>
<tt>Tue Jul 26 13:28:53.405 2016: [I] rax 0x0000000000000000 rbx 0x00007F6DC8DCB000</tt><br>
<tt>Tue Jul 26 13:28:53.406 2016: [I] rcx 0xFFFFFFFFFFFFFFFF rdx 0x0000000000000006</tt><br>
<tt>Tue Jul 26 13:28:53.407 2016: [I] rsp 0x00007F6DAAEA01F8 rbp 0x00007F6DCA05C8B0</tt><br>
<tt>Tue Jul 26 13:28:53.408 2016: [I] rsi 0x00000000000018F8 rdi 0x0000000000001876</tt><br>
<tt>Tue Jul 26 13:28:53.409 2016: [I] r8 0xFEFEFEFEFEFEFEFF r9 0xFEFEFEFEFF092D63</tt><br>
<tt>Tue Jul 26 13:28:53.410 2016: [I] r10 0x0000000000000008 r11 0x0000000000000202</tt><br>
<tt>Tue Jul 26 13:28:53.411 2016: [I] r12 0x00007F6DC9FC5540 r13 0x00007F6DCA05C1C0</tt><br>
<tt>Tue Jul 26 13:28:53.412 2016: [I] r14 0x0000000000000000 r15 0x0000000000000000</tt><br>
<tt>Tue Jul 26 13:28:53.413 2016: [I] rip 0x00007F6DC7894625 eflags 0x0000000000000202</tt><br>
<tt>Tue Jul 26 13:28:53.414 2016: [I] csgsfs 0x0000000000000033 err 0x0000000000000000</tt><br>
<tt>Tue Jul 26 13:28:53.415 2016: [I] trapno 0x0000000000000000 oldmsk 0x0000000010017807</tt><br>
<tt>Tue Jul 26 13:28:53.416 2016: [I] cr2 0x0000000000000000</tt><br>
<tt>Tue Jul 26 13:28:54.225 2016: [D] Traceback:</tt><br>
<tt>Tue Jul 26 13:28:54.226 2016: [D] 0:00007F6DC7894625 raise + 35 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.227 2016: [D] 1:00007F6DC7895E05 abort + 175 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.228 2016: [D] 2:00007F6DC788D74E __assert_fail_base + 11E at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.229 2016: [D] 3:00007F6DC788D810 __assert_fail + 50 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.230 2016: [D] 4:00007F6DC95444CA logAssertFailed + 2FA at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.231 2016: [D] 5:00007F6DC95C7EF4 ClusterConfiguration::getGatewayNewHash(DiskUID, unsigned int, NodeAddr*) + 4B4 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.232 2016: [D] 6:00007F6DC95C8031 ClusterConfiguration::getGatewayNode(DiskUID, unsigned int, NodeAddr, NodeAddr*, unsigned int) + 91 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.233 2016: [D] 7:00007F6DC9DC7126 SFSPcache(StripeGroup*, FileUID, int, int, void*, int, voidXPtr*, int*) + 346 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.234 2016: [D] 8:00007F6DC9332494 HandleMBPcache(MBPcacheParms*) + B4 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.235 2016: [D] 9:00007F6DC90A4A53 Mailbox::msgHandlerBody(void*) + 3C3 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.236 2016: [D] 10:00007F6DC908BC06 Thread::callBody(Thread*) + 46 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.237 2016: [D] 11:00007F6DC907A0D2 Thread::callBodyWrapper(Thread*) + A2 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.238 2016: [D] 12:00007F6DC87A3AA1 start_thread + D1 at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.239 2016: [D] 13:00007F6DC794A93D clone + 6D at ??:0</tt><br>
<tt>Tue Jul 26 13:28:54.240 2016: [N] Restarting mmsdrserv</tt><br>
<tt>Tue Jul 26 13:28:55.535 2016: [N] Signal 6 at location 0x7F6DC790EA7D in process 6262, link reg 0xFFFFFFFFFFFFFFFF.</tt><br>
<tt>Tue Jul 26 13:28:55.536 2016: [N] mmfsd is shutting down.</tt><br>
<tt>Tue Jul 26 13:28:55.537 2016: [N] Reason for shutdown: Signal handler entered</tt><br>
<tt>Tue Jul 26 13:28:55 BST 2016: mmcommon mmfsdown invoked. Subsystem: mmfs Status: active</tt><br>
<tt>Tue Jul 26 13:28:55 BST 2016: /var/mmfs/etc/mmfsdown invoked</tt><br>
<tt>umount2: Device or resource busy</tt><br>
<tt>umount: /camp: device is busy.</tt><br>
<tt> (In some cases useful info about processes that use</tt><br>
<tt> the device is found by lsof(8) or fuser(1))</tt><br>
<tt>umount2: Device or resource busy</tt><br>
<tt>umount: /ingest: device is busy.</tt><br>
<tt> (In some cases useful info about processes that use</tt><br>
<tt> the device is found by lsof(8) or fuser(1))</tt><br>
<tt>Shutting down NFS daemon: [ OK ]</tt><br>
<tt>Shutting down NFS mountd: [ OK ]</tt><br>
<tt>Shutting down NFS quotas: [ OK ]</tt><br>
<tt>Shutting down NFS services: [ OK ]</tt><br>
<tt>Shutting down RPC idmapd: [ OK ]</tt><br>
<tt>Stopping NFS statd: [ OK ]</tt><br>
<br>
<br>
<br>
<tt>Ugly, right?</tt><br>
<br>
<tt>Cheers,</tt><br>
<tt>Luke.</tt><br>
<br>
<br>
<tt>Luke Raimbach?</tt><br>
<tt>Senior HPC Data and Storage Systems Engineer,</tt><br>
<tt>The Francis Crick Institute,</tt><br>
<tt>Gibbs Building,</tt><br>
<tt>215 Euston Road,</tt><br>
<tt>London NW1 2BE.</tt><br>
<br>
<tt>E: <a href="mailto:luke.raimbach@crick.ac.uk">luke.raimbach@crick.ac.uk</a></tt><br>
<tt>W: </tt></span><a href="www.crick.ac.uk"><tt><span style="font-size:10.0pt">www.crick.ac.uk</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<br>
<tt>The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.</tt><br>
<br>
<tt>------------------------------</tt><br>
<br>
<tt>_______________________________________________</tt><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at spectrumscale.org</tt><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<br>
<tt>End of gpfsug-discuss Digest, Vol 54, Issue 59</tt><br>
<tt>**********************************************</tt><br>
<br>
</span><br>
<br>
<o:p></o:p></p>
</div>
</div>
<p style="color:rgb(112,113,115);font-family: 'Trebuchet MS', 'Lucida Grande'; font-style: italic; font-size: 10pt;">
The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.
</p>
</body>
</html>