<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Trebuchet MS";
panose-1:2 11 6 3 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi Marc,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Thanks again for the suggestions. An interesting report in the log while the another node took over managing the filesystem:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Mon Jul 4 10:24:08.616 2016: [W] Inode space 10 in file system gpfs is approaching the limit for the maximum number of inodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Inode space 10 was the independent fileset that the snapshot creation/deletion managed to remove. Still getting negative inode numbers
reported after migrating manager functions and suspending/resuming the file system:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Inode Information<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">-----------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Total number of used inodes in all Inode spaces: -103900000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Total number of free inodes in all Inode spaces: -24797856<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Total number of allocated inodes in all Inode spaces: -128697856<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Total of Maximum number of inodes in all Inode spaces: -103900000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I’ll have to wait until later today to try unmounting, daemon recycle or mmfsck.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Luke.<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>Marc A Kaplan<br>
<b>Sent:</b> 03 July 2016 19:43<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] Trapped Inodes - releases, now count them properly!<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">mmdf statistics are not real-time accurate, there is a trade off in accuracy vs the cost of polling each node that might have the file system mounted.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">That said, here are some possibilities, in increasing order of impact on users and your possible desperation ;-)</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">A1. Wait a while (at most a few minutes) and see if the mmdf stats are updated.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">A2. </span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">mmchmgr fs another-node
</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">may force new stats to be sent to the new fs manager. (Not sure but I expect it will.)</span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif"> </span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">B. Briefly quiesce the file system with:
</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">mmfsctl fs suspend; mmfsctl fs resume; </span><br>
<br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">C. If you have no users active ... I'm pretty sure
</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">mmumount fs -a ; mmmount fs -a; </span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">will clear the problem ... but there's always</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">D. </span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">mmshutdown -a ; mmstartup -a
</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">E.</span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">If none of those resolve the situation something is hosed --
</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">F.</span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">hope that </span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">mmfsck </span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">can fix it.<br>
</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">--marc</span><br>
<br>
<br>
<br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">From: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">Luke Raimbach <<a href="mailto:Luke.Raimbach@crick.ac.uk">Luke.Raimbach@crick.ac.uk</a>></span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">To: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>></span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Date: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">07/03/2016 10:55 AM</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Subject: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">Re: [gpfsug-discuss] Trapped Inodes</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Sent by: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></span><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="3" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal"><br>
<br>
<br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">Hi Marc,</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080"> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">Thanks for that suggestions. This seems to have removed the NULL fileset from the list, however mmdf now shows even more strange statistics:</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080"> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">Inode Information</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">-----------------</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">Total number of used inodes in all Inode spaces: -103900000</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">Total number of free inodes in all Inode spaces: -24797856</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">Total number of allocated inodes in all Inode spaces: -128697856</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">Total of Maximum number of inodes in all Inode spaces: -103900000</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080"> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080"> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">Any ideas why these negative numbers are being reported?</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080"> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">Cheers,</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080">Luke.</span><br>
<span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#004080"> </span><br>
<b><span style="font-size:10.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:10.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a> [</span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif">mailto:gpfsug-discuss-bounces@spectrumscale.org</span></a><span style="font-size:10.0pt;font-family:"Calibri",sans-serif">]
<b>On Behalf Of </b>Marc A Kaplan<b><br>
Sent:</b> 02 July 2016 20:17<b><br>
To:</b> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>><b><br>
Subject:</b> Re: [gpfsug-discuss] Trapped Inodes</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">I have been informed that it is possible that a glitch can occur (for example an abrupt shutdown) which can leave you in a situation where it looks like all snapshots are deleted, but there is still
a hidden snapshot that must be cleaned up... </span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
The workaround is to create a snapshot `mmcrsnapshot fs dummy` and then delete it `mmdelsnapshot fs dummy` and see if that clears up the situation...</span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
--marc</span><br>
<br>
<br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F"><br>
From: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">Luke Raimbach <</span><a href="mailto:Luke.Raimbach@crick.ac.uk"><span style="font-size:7.5pt;font-family:"Arial",sans-serif">Luke.Raimbach@crick.ac.uk</span></a><span style="font-size:7.5pt;font-family:"Arial",sans-serif">><span style="color:#5F5F5F"><br>
To: </span>gpfsug main discussion list <</span><a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="font-size:7.5pt;font-family:"Arial",sans-serif">gpfsug-discuss@spectrumscale.org</span></a><span style="font-size:7.5pt;font-family:"Arial",sans-serif">><span style="color:#5F5F5F"><br>
Date: </span>07/02/2016 06:05 AM<span style="color:#5F5F5F"><br>
Subject: </span>Re: [gpfsug-discuss] Trapped Inodes<span style="color:#5F5F5F"><br>
Sent by: </span></span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><span style="font-size:7.5pt;font-family:"Arial",sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span></a><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="3" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
Hi Marc,<o:p></o:p></p>
<p>Thanks for the suggestion.<o:p></o:p></p>
<p>Snapshots were my first suspect but there are none anywhere on the filesystem.<o:p></o:p></p>
<p>Cheers,<br>
Luke.<o:p></o:p></p>
<p>On 1 Jul 2016 5:30 pm, Marc A Kaplan <<a href="mailto:makaplan@us.ibm.com">makaplan@us.ibm.com</a>> wrote:<span style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
Question and Suggestion: Do you have any snapshots that might include files that were in the fileset you are attempting to delete? Deleting those snapshots will allow the fileset deletion to complete. The snapshots are kinda intertwined with what was the
"live" copy of the inodes. In the GPFS "ditto" implementation of snapshotting, for a file that has not changed since the snapshot operation, the snapshot copy is not really a copy but just a pointer to the "live" file. So even after you have logically deleted
the "live" files, the snapshot still points to those inodes you thought you deleted. Rather than invalidate the snapshot, (you wouldn't want that, would you?!) GPFS holds onto the inodes, until they are no longer referenced by any snapshot.<br>
<br>
--marc </span><br>
<br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F"><br>
<br>
From: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">Luke Raimbach <</span><a href="mailto:Luke.Raimbach@crick.ac.uk"><span style="font-size:7.5pt;font-family:"Arial",sans-serif">Luke.Raimbach@crick.ac.uk</span></a><span style="font-size:7.5pt;font-family:"Arial",sans-serif">><span style="color:#5F5F5F"><br>
To: </span>gpfsug main discussion list <</span><a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="font-size:7.5pt;font-family:"Arial",sans-serif">gpfsug-discuss@spectrumscale.org</span></a><span style="font-size:7.5pt;font-family:"Arial",sans-serif">><span style="color:#5F5F5F"><br>
Date: </span>07/01/2016 06:32 AM<span style="color:#5F5F5F"><br>
Subject: </span>[gpfsug-discuss] Trapped Inodes<span style="color:#5F5F5F"><br>
Sent by: </span></span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><span style="font-size:7.5pt;font-family:"Arial",sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span></a><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="3" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal"><br>
<br>
<span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
Hi All,<br>
<br>
I've run out of inodes on a relatively small filesystem. The total metadata capacity allows for a maximum of 188,743,680 inodes.<br>
<br>
A fileset containing 158,000,000 inodes was force deleted and has gone into a bad state, where it is reported as (NULL) and has state "deleted":<br>
<br>
Attributes for fileset (NULL):<br>
===============================<br>
Status Deleted<br>
Path --<br>
Id 15<br>
Root inode latest:<br>
Parent Id <none><br>
Created Wed Jun 15 14:07:51 2016<br>
Comment<br>
Inode space 8<br>
Maximum number of inodes 158000000<br>
Allocated inodes 158000000<br>
Permission change flag chmodAndSetacl<br>
afm-associated No<br>
<br>
Offline mmfsck fixed a few problems, but didn't free these poor, trapped inodes. Now I've run out and mmdf is telling me crazy things like this:<br>
<br>
Inode Information<br>
-----------------<br>
Total number of used inodes in all Inode spaces: 0<br>
Total number of free inodes in all Inode spaces: 27895680<br>
Total number of allocated inodes in all Inode spaces: 27895680<br>
Total of Maximum number of inodes in all Inode spaces: 34100000<br>
<br>
<br>
Current GPFS build: "4.2.0.3".<br>
<br>
Who will help me rescue these inodes?<br>
<br>
Cheers,<br>
Luke.<br>
<br>
Luke Raimbach</span><span style="font-size:10.0pt;font-family:"Cambria Math",serif"></span><span style="font-size:10.0pt;font-family:"Courier New""><br>
Senior HPC Data and Storage Systems Engineer,<br>
The Francis Crick Institute,<br>
Gibbs Building,<br>
215 Euston Road,<br>
London NW1 2BE.<br>
<br>
E: </span><a href="mailto:luke.raimbach@crick.ac.uk"><span style="font-size:10.0pt;font-family:"Courier New"">luke.raimbach@crick.ac.uk</span></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
W: </span><a href="www.crick.ac.uk"><span style="font-size:10.0pt;font-family:"Courier New"">www.crick.ac.uk</span></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org</span><u><span style="color:blue"><br>
</span></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><span style="font-size:10.0pt;font-family:"Courier New"">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p>
<p><i><span style="font-size:10.0pt;font-family:"Trebuchet MS",sans-serif;color:gray">The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered
office at 215 Euston Road, London NW1 2BE. </span></i><span style="font-size:10.0pt;font-family:"Courier New"">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org</span><u><span style="color:blue"><br>
</span></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><span style="font-size:10.0pt;font-family:"Courier New"">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><i><span style="font-size:10.0pt;font-family:"Trebuchet MS",sans-serif;color:gray">The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered
office at 215 Euston Road, London NW1 2BE. </span></i><tt><span style="font-size:10.0pt">_______________________________________________</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at spectrumscale.org</tt><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><o:p></o:p></p>
<p><o:p> </o:p></p>
</div>
</div>
<p style="color:rgb(112,113,115);font-family: 'Trebuchet MS', 'Lucida Grande'; font-style: italic; font-size: 10pt;">
The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.
</p>
</body>
</html>