<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:windowtext;
letter-spacing:0pt;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1735158137;
mso-list-type:hybrid;
mso-list-template-ids:-1785011348 1568707246 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;
font-family:"Arial",sans-serif;
mso-fareast-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style></head><body lang=en-CH link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>Hello,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>We currently move data to a new AFM fileset and I see poor performance and ask for advice and insight:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>The migration to afm home seems slow. I note:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><ul style='margin-top:0cm' type=disc><li class=MsoListParagraph style='margin-left:-18.0pt;mso-list:l0 level1 lfo1'><span lang=EN-US style='font-family:"Arial",sans-serif'>Afm writes a whole file of ~100MB in much too many small chunks <o:p></o:p></span></li></ul><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>My assumption: The many small writes reduce performance as we have 100km between the sites and a higher latency. The writes are not fully sequentially, but they aren’t done heavily parallel, either (like 10-100 outstanding writes at each time).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>I the afm queue I see<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>8100214 Write [563636091.563636091] inflight (0 @ 0) chunks 2938 bytes 170872410 vIdx 1 thread_id 67862<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>I guess this means afm will write 170’872’410 bytes in 2’938chunks resulting in an average write size of 58k to inode 563636091.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>So if I’m right my question is: <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>What can I change to make afm write less and larger chunks per file? <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>Does it depend on how we copy data? We write through ganesha/nfs, hence even if we write sequentially ganesha may still do it differently?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>Another question – is there a way to dump the afm in-memory queue for a fileset? That would make it easier to see what’s going on when we do changes. I could grep for the inode of a testfile …<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>We don’t do parallel writes across afm gateways, the files are too small, our limit is 1GB.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>We configured two mounts from two ces servers at home for each filesets. Hence AFM could do writes in parallel to both mounts on the single gateway? <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>A short tcpdump suggests: afm writes to a single ces server only and writes to a single inode at a time. But at each time a few writes (2-5) may overlap.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>Kind regards,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>Heiner<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>Just to illustrate – what I see on the afm gateway – too many reads and writes. There are almost no open/close hence its all to the same few files<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'>------------nfs3-client------------ --------gpfs-file-operations------- --gpfs-i/o- -net/total-<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'> read writ rdir inod fs cmmt| open clos read writ rdir inod| read write| recv send<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'> 0 1295 0 0 0 0 | 0 0 1294 0 0 0 |89.8M 0 | 451k 94M<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'> 0 1248 0 0 0 0 | 0 0 1248 0 0 8 |86.2M 0 | 432k 91M<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'> 0 1394 0 0 0 0 | 0 0 1394 0 0 0 |96.8M 0 | 498k 101M<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'> 0 1583 0 0 0 0 | 0 0 1582 0 0 1 | 110M 0 | 560k 115M<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Arial",sans-serif'> 0 1543 0 1 0 0 | 0 0 1544 0 0 0 | 107M 0 | 540k 112M<o:p></o:p></span></p></div></body></html>