<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Hello Anderson,</div><div class=""><br class=""></div><div class="">This application requires minimum throughput of about 10-13MB/s initially and almost no IOPS during first phase where it opens all the files and reads the headers and about 30MB/s throughput during the second phase.</div><div class="">The issue that I face is during the second phase where it tries to randomly read about 4K of block size from random files from 20000 to about 100000.</div><div class="">In this phase - I see a big difference in maxblocksize parameter changing the performance of the reads, with almost no throughput and may be around 2-4K IOPS.</div><div class=""><br class=""></div><div class="">This issue is a follow up to the previous issue that I had mentioned about an year ago - where I see differences in performance - “though there is practically no IO to the storage”</div><div class="">I mean - I see a difference in performance between different FS block-sizes even if all data is cached in pagepool.</div><div class="">Sven had replied to that thread mentioning that it could be because of buffer locking issue.</div><div class=""> </div><div class="">The info requested is as below: </div><div class=""><br class=""></div><div class="">4 Storage clusters:</div><div class=""><br class=""></div><div class=""><u class="">Storage cluster for compute:</u></div><div class="">5.0.3-2 GPFS version</div><div class="">FS version: 19.01 (5.0.1.0)</div><div class="">Subblock size: 16384</div><div class="">Blocksize : 16M </div><div class=""><br class=""></div><div class=""><u class="">Flash Storage Cluster for compute:</u></div><div class="">5.0.4-2 GPFS version</div><div class="">FS version: 18.00 (5.0.0.0)</div><div class="">Subblock size: 8192</div><div class="">Blocksize: 512K</div><div class=""><br class=""></div><div class=""><u class="">Storage cluster for admin tools:</u></div><div class="">5.0.4-2 GPFS version</div><div class="">FS version: 16.00 (4.2.2.0)</div><div class="">Subblock size: 131072</div><div class="">Blocksize: 4M</div><div class=""><br class=""></div><div class="">Storage cluster for archival:</div><div class="">5.0.3-2 GPFS version</div><div class="">FS version: 16.00 (4.2.2.0)</div><div class="">Subblock size: 32K</div><div class="">Blocksize: 1M </div><div class=""><br class=""></div><div class="">The only two clusters that users do/will do compute on is the 16M filesystem and the 512K Filesystem.</div><div class=""><br class=""></div><div class="">When you ask what is the throughput/IOPS and block size - it varies a lot and has not been recorded.</div><div class="">The 16M FS is capable of doing about 27GB/s seq read for about 1.8 PB of storage.</div><div class="">The 512K FS is capable of doing about 10-12GB/s seq read for about 100T of storage.</div><div class=""><br class=""></div><div class="">Now as I mentioned previously - the issue that I am seeing has been related to different FS block sizes on the same storage.</div><div class="">For example: </div><div class="">On the Flash Storage cluster: </div><div class="">Block size of 512K with maxblocksize of 16M gives worse performance than Block size of 512K with maxblocksize of 512K.</div><div class="">It is the maxblocksize that is affecting the performance, on the same storage with same block size and everything else being the same.</div><div class="">I am thinking the above is because of the number of buffers involved, but would like to learn if it happens to be anything else.</div><div class="">I have debugged the same with IBM GPFS techs and it has been found that there is no issue with the storage itself or any of the other GPFS tuning parameters.</div><div class=""><br class=""></div><div class="">Now since we do know that maxblocksize is making a big difference.</div><div class="">I would like to keep it as low as possible but still be able to mount other remote GPFS filesystems with higher block sizes.</div><div class="">Or since it is required to keep the maxblocksize the same across all storage - I would like to know if there is any other parameters that could do the same change as maxblocksize.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Thank you,</div><div class="">Lohit </div><div class=""><br class=""></div><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Feb 28, 2020, at 12:58 PM, Anderson Ferreira Nobre <<a href="mailto:anobre@br.ibm.com" class="">anobre@br.ibm.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt"><div dir="ltr" class="">Hi Lohit,</div>
<div dir="ltr" class=""> </div>
<div dir="ltr" class="">First, a few questions to understand better your problem:</div>
<div dir="ltr" class="">- What is the minimum release level of both clusters?</div>
<div dir="ltr" class="">- What is the version of filesystem layout for 16MB, 1MB and 512KB?</div>
<div dir="ltr" class="">- What is the subblocksize of each filesystem?</div>
<div dir="ltr" class="">- How many IOPS, block size and throughput are you doing on each filesystem?</div>
<div dir="ltr" class=""><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt"><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt"><div dir="ltr" class=""><div class=""> </div>
<table cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;white-space:normal;color:#5f5f5f;width:650px;font-size:7.2pt" width="650px" class=""> <tbody class=""> <tr class=""> <td style="vertical-align: bottom;" class=""> <div style="font-size:9pt" class=""><span style="font-size:1.000em;" class="">Abraços / Regards / Saludos,</span></div><div class=""> <br class="webkit-block-placeholder"></div>
<div class="vcard" style="margin:0 0 8px 0"><span style="font-size:1.000em;" class=""><b class="n fn" style="color:#888888;font-size:12pt"><span class="given-name">Anderson</span> <span class="family-name">Nobre</span></b><br class=""> Power and Storage Consultant<br class=""> IBM Systems Hardware Client Technical Team – IBM Systems Lab Services<br class=""> <br class=""> <img alt="community_general_lab_services" class="org" height="120" src="http://esign.webmaster.ibm.com/img/signature_lab_services.jpg" width="333"></span></div> </td> <td class=""> </td> </tr> </tbody></table>
<table cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;white-space:normal;color:#5f5f5f;width:650px;font-size:7.2pt" width="650px" class=""> <tbody class=""> <tr class=""> <td colspan="3" class=""> <hr style="border:0;color:#696969;background-color:#696969;height:1px;" class=""></td> </tr> <tr class=""> <td style="vertical-align: top;font-family:arial,helvetica,sans-serif" width="400px" class=""><span style="font-size:1.000em;" class=""><span class="tel"><span style="color:#466BB0;font-weight:bold" class=""><abbr class="type" title="work">Phone:</abbr></span> <span class="value">55-19-2132-4317</span></span><br class=""> <span style="color:#466BB0;font-weight:bold;font-style:7.2pt" class="">E-mail:</span> <a class="email" href="mailto:anobre@br.ibm.com" target="_blank"><span class="value" style="color:#5f5f5f">anobre@br.ibm.com</span></a></span></td> <td align="right" style="vertical-align: top;font-family:arial,helvetica,sans-serif" width="150px" class=""><span style="font-size:1.000em;" class=""><img alt="IBM" class="org" src="http://esign.webmaster.ibm.com/img/ibm3.gif"></span></td> </tr> </tbody></table></div></div></div></div>
<div dir="ltr" class=""> </div>
<div dir="ltr" class=""> </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" class="">----- Original message -----<br class="">From: "Valleru, Lohit/Information Systems" <<a href="mailto:valleru@cbio.mskcc.org" class="">valleru@cbio.mskcc.org</a>><br class="">Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a><br class="">To: <a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a><br class="">Cc:<br class="">Subject: [EXTERNAL] [gpfsug-discuss] Maxblocksize tuning alternatives/max number of buffers<br class="">Date: Fri, Feb 28, 2020 12:30<br class="">
<div class=""><font face="Default Monospace,Courier New,Courier,monospace" size="2" class="">Hello Everyone,<br class=""><br class="">I am looking for alternative tuning parameters that could do the same job as tuning the maxblocksize parameter.<br class=""><br class="">One of our users run a deep learning application on GPUs, that does the following IO pattern:<br class=""><br class="">It needs to read random small sections about 4K in size from about 20,000 to 100,000 files of each 100M to 200M size.<br class=""><br class="">When performance tuning for the above application on a 16M filesystem and comparing it to various other file system block sizes - I realized that the performance degradation that I see might be related to the number of buffers.<br class=""><br class="">I observed that the performance varies widely depending on what maxblocksize parameter I use.<br class="">For example, using a 16M maxblocksize for a 512K or a 1M block size filesystem differs widely from using a 512K or 1M maxblocksize for a 512K or a 1M block size filesystem.<br class=""><br class="">The reason I believe might be related to the number of buffers that I could keep on the client side, but I am not sure if that is the all that the maxblocksize is affecting.<br class=""><br class="">We have different file system block sizes in our environment ranging from 512K, 1M and 16M.<br class=""><br class="">We also use storage clusters and compute clusters design.<br class=""><br class="">Now in order to mount the 16M filesystem along with the other filesystems on compute clusters - we had to keep the maxblocksize to be 16M - no matter what the file system block size.<br class=""><br class="">I see that I get maximum performance for this application from a 512K block size filesystem and a 512K maxblocksize.<br class="">However, I will not be able to mount this filesystem along with the other filesystems because I will need to change the maxblocksize to 16M in order to mount the other filesystems of 16M block size.<br class=""><br class="">I am thinking if there is anything else that can do the same job as maxblocksize parameter.<br class=""><br class="">I was thinking about the parameters like maxBufferDescs for a 16M maxblocksize, but I believe it would need a lot more pagepool to keep the same number of buffers as would be needed for a 512k maxblocksize.<br class=""><br class="">May I know if there is any other parameter that could help me the same as maxblocksize, and the side effects of the same?<br class=""><br class="">Thank you,<br class="">Lohit<br class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font><br class=""> </div></blockquote>
<div dir="ltr" class=""> </div></div><br class="">
_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class=""></div></blockquote></div><br class=""></body></html>