<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:Consolas;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1376271249;
mso-list-type:hybrid;
mso-list-template-ids:-802671178 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoPlainText">It's worth sharing that we have seen two problems with CES providing NFS via ganesha in a similar deployment:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>multicluster cache invalidation: ganesha's FSAL upcall for invalidation of its file descriptor cache by GPFS doesn't appear to work for remote GPFS filesystems. As mentioned by Simon, this is unsupported, though the problem can be worked
around with some effort though by disabling ganesha’s FD cache entirely.<br>
<br>
<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Readdir bad cookie bug: an interaction we’re still providing info to IBM about between certain linux NFS clients and ganesha in which readdir calls may sporadically return empty results for directories containing files, without any corresponding
error result code.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Given our multicluster requirements and the problems associated with the readdir bug, we’ve reverted to using CNFS for now.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Thx<br>
Paul<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<br>
From: gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org] On Behalf Of Simon Thompson (Research Computing - IT Services)<br>
Sent: Tuesday, March 22, 2016 6:05 AM<br>
To: gpfsug main discussion list<br>
Subject: Re: [gpfsug-discuss] HAWC/LROC in Ganesha server</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Hi Martin,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">We have LROC enabled on our CES protocol nodes for SMB:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"># mmdiag --lroc<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">=== mmdiag: lroc ===<o:p></o:p></p>
<p class="MsoPlainText">LROC Device(s): '0A0A001755E9634D#/dev/sdb;0A0A001755E96350#/dev/sdc;'<o:p></o:p></p>
<p class="MsoPlainText">status Running<o:p></o:p></p>
<p class="MsoPlainText">Cache inodes 1 dirs 1 data 1 Config: maxFile 0 stubFile 0 Max capacity: 486370 MB, currently in use: 1323 MB Statistics from: Thu Feb 25 11:18:25 2016<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Total objects stored 338690236 (2953113 MB) recalled 336905443 (1326912 MB)<o:p></o:p></p>
<p class="MsoPlainText"> objects failed to store 0 failed to recall 94 failed to inval 0<o:p></o:p></p>
<p class="MsoPlainText"> objects queried 0 (0 MB) not found 0 = 0.00 %<o:p></o:p></p>
<p class="MsoPlainText"> objects invalidated 338719563 (3114191 MB)<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Inode objects stored 336876572 (1315923 MB) recalled 336884262<o:p></o:p></p>
<p class="MsoPlainText">(1315948 MB) = 100.00 %<o:p></o:p></p>
<p class="MsoPlainText"> Inode objects queried 0 (0 MB) = 0.00 % invalidated 336910469<o:p></o:p></p>
<p class="MsoPlainText">(1316052 MB)<o:p></o:p></p>
<p class="MsoPlainText"> Inode objects failed to store 0 failed to recall 0 failed to query 0 failed to inval 0<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Directory objects stored 2896 (115 MB) recalled 564 (29 MB) = 19.48 %<o:p></o:p></p>
<p class="MsoPlainText"> Directory objects queried 0 (0 MB) = 0.00 % invalidated 2857 (725 MB)<o:p></o:p></p>
<p class="MsoPlainText"> Directory objects failed to store 0 failed to recall 2 failed to query 0 failed to inval 0<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> Data objects stored 1797127 (1636968 MB) recalled 16057 (10907 MB) =<o:p></o:p></p>
<p class="MsoPlainText">0.89 %<o:p></o:p></p>
<p class="MsoPlainText"> Data objects queried 0 (0 MB) = 0.00 % invalidated 1805234 (1797405<o:p></o:p></p>
<p class="MsoPlainText">MB)<o:p></o:p></p>
<p class="MsoPlainText"> Data objects failed to store 0 failed to recall 92 failed to query 0 failed to inval 0<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> agent inserts=389305528, reads=337261110<o:p></o:p></p>
<p class="MsoPlainText"> response times (usec):<o:p></o:p></p>
<p class="MsoPlainText"> insert min/max/avg=1/47705/11<o:p></o:p></p>
<p class="MsoPlainText"> read min/max/avg=1/3145728/54<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> ssd writeIOs=5906506, writePages=756033024<o:p></o:p></p>
<p class="MsoPlainText"> readIOs=44692016, readPages=44692610<o:p></o:p></p>
<p class="MsoPlainText"> response times (usec):<o:p></o:p></p>
<p class="MsoPlainText"> write min/max/avg=3072/1117534/3253<o:p></o:p></p>
<p class="MsoPlainText"> read min/max/avg=56/3145728/364<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">So mostly it is inode objects being used form the cache. Whether this is small data-in-inode or plain inode (stat) type operations, pass.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">We don't use HAWC on our protocol nodes, the HAWC pool needs to exist in the cluster where the NSD data is written and we multi-cluster to the protocol nodes (technically this isn't supported, but works fine for us).<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">On HAWC, we did test it out in another of our clusters using SSDs in the nodes, but we er, had a few issues when we should a rack of kit down which included all the HAWC devices which were in nodes. You probably want to think a bit carefully
about how HAWC is implemented in your environment.<o:p></o:p></p>
<p class="MsoPlainText">We are about to implement in one of our clusters, but that will be HAWC devices available to the NSD servers rather than on client nodes.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Simon<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">On 22/03/2016, 09:45, "<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org%20on%20behalf%20of%20Martin%20Gasthuber"><span style="color:windowtext;text-decoration:none">gpfsug-discuss-bounces@spectrumscale.org on behalf of Martin
Gasthuber</span></a>" <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org%20on%20behalf%20of%20martin.gasthuber@desy.de"><span style="color:windowtext;text-decoration:none">gpfsug-discuss-bounces@spectrumscale.org on behalf of martin.gasthuber@desy.de</span></a>>
wrote:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">>Hi,<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">> we're looking for a powerful (and cost efficient) machine config to
<o:p></o:p></p>
<p class="MsoPlainText">>optimally support the new CES services, especially Ganesha. In more
<o:p></o:p></p>
<p class="MsoPlainText">>detail, we're wondering if somebody has already got some experience
<o:p></o:p></p>
<p class="MsoPlainText">>running these services on machines with HAWC and/or LROC enabled HW,
<o:p></o:p></p>
<p class="MsoPlainText">>resulting in a clearer understanding of the benefits of that config. We
<o:p></o:p></p>
<p class="MsoPlainText">>will have ~300 client boxes accessing GPFS via NFS and planning for 2
<o:p></o:p></p>
<p class="MsoPlainText">>nodes initially.<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">>best regards,<o:p></o:p></p>
<p class="MsoPlainText">> Martin<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">>_______________________________________________<o:p></o:p></p>
<p class="MsoPlainText">>gpfsug-discuss mailing list<o:p></o:p></p>
<p class="MsoPlainText">>gpfsug-discuss at spectrumscale.org<o:p></o:p></p>
<p class="MsoPlainText">><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><span style="color:windowtext;text-decoration:none">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">_______________________________________________<o:p></o:p></p>
<p class="MsoPlainText">gpfsug-discuss mailing list<o:p></o:p></p>
<p class="MsoPlainText">gpfsug-discuss at spectrumscale.org<o:p></o:p></p>
<p class="MsoPlainText"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><span style="color:windowtext;text-decoration:none">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p>
</div>
</body>
</html>