<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;
mso-fareast-language:EN-US;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1172451203;
mso-list-type:hybrid;
mso-list-template-ids:70401918 134807567 134807577 134807579 134807567 134807577 134807579 134807567 134807577 134807579;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body bgcolor="white" lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">Hi all.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">After some help from IBM, we’ve concluded (and been told) that AFM over the NSD protocol when latency is greater than around 50ms on the RTT is effectively unusable. We’ve proven that now, so
it is time to move on from the NSD protocol being an effective option in those conditions (unless IBM can consider it something worthy of an RFE and can fix it!).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">The problem we face now, is one of parallelism and filling that 10GbE/40GbE/100GbE pipe efficiently, when using NFS as the transport provider for AFM.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">On my test cluster at “Cache” side I’ve got two or three gateways:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@mc-5 ~]# mmlscluster <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">GPFS cluster information<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">========================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> GPFS cluster name: sdx-gpfs.xxxxxxxxxxxxxxxx<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> GPFS cluster id: 12880500218013865782<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> GPFS UID domain: sdx-gpfs. xxxxxxxxxxxxxxxx<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Remote shell command: /usr/bin/ssh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Remote file copy command: /usr/bin/scp<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Repository type: CCR<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Node Daemon node name IP address Admin node name Designation<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---------------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 1 mc-5. xxxxxxxxxxxxxxxx.net ip.addresses.hidden mc-5.hidden.net quorum-manager<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 2 mc-6. xxxxxxxxxxxxxxxx.net ip.addresses.hidden mc-6. hidden.net quorum-manager-gateway<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 3 mc-7. xxxxxxxxxxxxxxxx.net ip.addresses.hidden mc-7. hidden.net quorum-manager-gateway<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 4 mc-8. xxxxxxxxxxxxxxxx.net ip.addresses.hidden mc-8. hidden.net quorum-manager-gateway<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">The bit I really don’t get is:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span lang="EN-AU" style="font-size:11.0pt"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-AU" style="font-size:11.0pt">Why no traffic ever seems to go through mc-6 or mc-8 back to my “home” directly and<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span lang="EN-AU" style="font-size:11.0pt"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-AU" style="font-size:11.0pt">Why it only ever lists my AFM-cache fileset being associated with one gateway (mc-7).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">I can see traffic flowing through mc-6 sometimes…but when it does, it all seems to channel back through mc-7 THEN back to the AFM-home. Am I missing something?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">This is where I see one of the gateway’s listed (but never the others?).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@mc-5 ~]# mmafmctl afmcachefs getstate<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">------------ -------------- ------------- ------------ ------------ ------------- <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">afm-home nfs://omnipath2/gpfs-flash/afm-home Active mc-7 0 746636 <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">I got told I needed to setup “explicit maps” back to my home cluster to achieve parallelism:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@mc-5 ~]# mmafmconfig show<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Map name: omnipath1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Export server map: address.is.hidden.100/mc-6.ip.address.hidden<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Map name: omnipath2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Export server map: address.is.hidden.101/mc-7.ip.address.hidden<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">But – I have never seen any traffic come back from mc-6 to omnipath1.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">What am I missing, and how do I actually achieve significant enough parallelism over an NFS transport to fill my 10GbE pipe?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">I’ve seen maybe a couple of gigabits per second from the mc-7 host writing back to the omnipath2 host – and that was really trying my level best to put as many files onto the afm-cache at this
side and hoping that enough threads pick up enough different files to start transferring files down the AFM simultaneously – but what I’d really like is those large files (or small, up to the thresholds set) to break into parallel chunks and ALL transfer as
fast as possible, utilising as much of the 10GbE as they can.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">Maybe I am missing fundamental principles in the way AFM works?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">Thanks.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">-jc<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:11.0pt">PS: NB The link is easily capable of 10GbE. We’ve tested it all the way up to about 9.67Gbit/sec transferring data from these sets of hosts using other protocols such as fDT and Globus Grid FTP
Et al.<o:p></o:p></span></p>
</div>
</body>
</html>