<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >Let me add just one more item to Sven's detailed reply: HAWC is especially helpful to decrease the latencies of small synchronous I/Os that come in *bursts*. If your workload contains a sustained high rate of writes, the recovery log will get full very quickly, and HAWC won't help much (or can even decrease performance). Making the recovery log larger allows to adsorb longer I/O bursts. The specific amount of improvements depends on the workload (how long/high are bursts, e.g.) and hardware.</div>
<div dir="ltr" > </div>
<div dir="ltr" >Best,</div>
<div dir="ltr" >Vasily</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><div>--</div>
<div>Vasily Tarasov,</div>
<div>Research Staff Member,</div>
<div>Storage Systems Research,</div>
<div>IBM Research - Almaden</div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Sven Oehme <oehmes@gmail.com><br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc: Vasily Tarasov <vtarasov@us.ibm.com><br>Subject: Re: [gpfsug-discuss] system.log pool on client nodes for HAWC<br>Date: Mon, Sep 3, 2018 8:32 AM<br>
<div dir="ltr" >Hi Ken,
<div> </div>
<div>what the documents is saying (or try to) is that the behavior of data in inode or metadata operations are not changed if HAWC is enabled, means if the data fits into the inode it will be placed there directly instead of writing the data i/o into a data recovery log record (which is what HAWC uses) and then later destage it where ever the data blocks of a given file eventually will be written. that also means if all your application does is creating small files that fit into the inode, HAWC will not be able to improve performance.</div>
<div>its unfortunate not so simple to say if HAWC will help or not, but here are a couple of thoughts where HAWC will not help and help : </div>
<div>on the where it won't help : </div>
<div>1. if you have storage device which has very large or even better are log structured write cache.</div>
<div>2. if majority of your files are very small</div>
<div>3. if your files will almost always be accesses sequentially </div>
<div>4. your storage is primarily flash based</div>
<div>where it most likely will help : </div>
<div>1. your majority of storage is direct attached HDD (e.g. FPO) with a small SSD pool for metadata and HAWC</div>
<div>2. your ratio of clients to storage devices is very high (think hundreds of clients and only 1 storage array) </div>
<div>3. your workload is primarily virtual machines or databases </div>
<div> </div>
<div>as always there are lots of exceptions and corner cases, but is the best list i could come up with. </div>
<div> </div>
<div>on how to find out if HAWC could help, there are 2 ways of doing this </div>
<div>first, look at mmfsadm dump iocounters , you see the average size of i/os and you could check if there is a lot of small write operations done. </div>
<div>a more involved but more accurate way would be to take a trace with trace level trace=io , that will generate a very lightweight trace of only the most relevant io layers of GPFS, you could then post process the operations performance, but the data is not the simplest to understand for somebody with low knowledge of filesystems, but if you stare at it for a while it might make some sense to you. </div>
<div> </div>
<div>Sven</div>
<div> </div>
<div> </div></div>
<div><div dir="ltr" >On Mon, Sep 3, 2018 at 4:06 PM Kenneth Waegeman <<a href="mailto:kenneth.waegeman@ugent.be" target="_blank">kenneth.waegeman@ugent.be</a>> wrote:</div>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" ><div bgcolor="#FFFFFF" text="#000000" ><p>Thank you Vasily and Simon for the clarification!</p>
<p>I was looking further into it, and I got stuck with more questions :)</p>
<p><br>- In <a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_hawc_tuning.htm" target="_blank">https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_hawc_tuning.htm</a> I read:<br> HAWC does not change the following behaviors:<br> write behavior of small files when the data is placed in the inode itself<br> write behavior of directory blocks or other metadata</p>I wondered why? Is the metadata not logged in the (same) recovery logs? (It seemed by reading <a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.ins.doc/bl1ins_logfile.htm" target="_blank">https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.ins.doc/bl1ins_logfile.htm</a> it does )<br><br><br>- Would there be a way to estimate how much of the write requests on a running cluster would benefit from enabling HAWC ?<br><br><br>Thanks again!</div>
<div bgcolor="#FFFFFF" text="#000000" ><br><br>Kenneth</div>
<div bgcolor="#FFFFFF" text="#000000" >
<div>On 31/08/18 19:49, Vasily Tarasov wrote:</div>
<blockquote type="cite" ><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt" ><div dir="ltr" >That is correct. The blocks of each recovery log are striped across the devices in the system.log pool (if it is defined). As a result, even when all clients have a local device in the system.log pool, many writes to the recovery log will go to remote devices. For a client that lacks a local device in the system.log pool, log writes will always be remote.</div>
<div dir="ltr" > </div>
<div dir="ltr" >Notice, that typically in such a setup you would enable log replication for HA. Otherwise, if a single client fails (and its recover log is lost) the whole cluster fails as there is no log to recover FS to consistent state. Therefore, at least one remote write is essential.</div>
<div dir="ltr" > </div>
<div dir="ltr" >HTH,</div>
<div dir="ltr" ><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt" ><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt" ><div dir="ltr" ><div>--</div>
<div>Vasily Tarasov,</div>
<div>Research Staff Member,</div>
<div>Storage Systems Research,</div>
<div>IBM Research - Almaden</div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote dir="ltr" style="border-left:solid #aaaaaa 2px;margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px" >----- Original message -----<br>From: Kenneth Waegeman <a href="mailto:kenneth.waegeman@ugent.be" target="_blank"><kenneth.waegeman@ugent.be></a><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br>To: gpfsug main discussion list <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><gpfsug-discuss@spectrumscale.org></a><br>Cc:<br>Subject: [gpfsug-discuss] system.log pool on client nodes for HAWC<br>Date: Tue, Aug 28, 2018 5:31 AM<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Hi all,<br><br>I was looking into HAWC , using the 'distributed fast storage in client<br>nodes' method (<br><a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_hawc_using.htm" target="_blank">https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_hawc_using.htm</a> <br>)<br><br>This is achieved by putting a local device on the clients in the<br>system.log pool. Reading another article<br>(<a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_syslogpool.htm" target="_blank">https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adv_syslogpool.htm</a> <br>) this would now be used for ALL File system recovery logs.<br><br>Does this mean that if you have a (small) subset of clients with fast<br>local devices added in the system.log pool, all other clients will use<br>these too instead of the central system pool?<br><br>Thank you!<br><br>Kenneth<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font><br> </div></blockquote>
<div dir="ltr" > </div></div>
<fieldset> </fieldset>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote></div>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></blockquote></div></blockquote>
<div dir="ltr" > </div></div><BR>