<div dir="ltr">given that i love data points, let me put some behind this thread :-)<div><br></div><div>starting in version 3.4 we enhanced the trace code of scale significant. this went on release to release all the way up to 4.2.1. since 4.2.1 we made further improvements, but much smaller changes, more optimization , e.g. reducing of trace levels verbosity, etc .</div><div>with 4.2.2 we switched from blocking traces to in memory traces as the default trace infrastructure, this infrastructure was designed to be turned on all the time with minimal impact on performance. </div><div>i just took a 4.2.3 build and did load it on one of my faster NVMe nodes and did a 100% random read test with 64 threads against very fast storage. </div><div>while this was running i started trace with mmtracectl --start ; sleep 10 ; mmtracectl --off , attached is a screenshot of this run :</div><div><br></div><div><img src="cid:15aae192e8cd0cad2201" alt="pasted1" class="" style="max-width: 100%; opacity: 1;"><br></div><div>the trace started at 14:19:15 and stopped at 14:20:00 (that includes 10 command processing, format of traces, etc) </div><div>as one can see on the chart the dip in performance is very small, measured looking at raw data its 8%. </div><div>the system runs at ~920k 4k random read iops /sec and the workload running on the node pushes the system almost to 100% cpu time even without trace running, reason i tested under this condition is because also on a HPC compute oriented workload you will run without spare cpu cycles, so the 8% is under extreme conditions. </div><div>said all this, the workload i am running is not causing a lot of metadata i/o as i am running within a small number of files, its all read, so no write, therefore this isn't representative to a lot of real world workloads, but covers one very extreme case , high iops requirements under heavy cpu contention. </div><div>therefore i repeated the test under a heavy metadata workload , SpecSFS SWBUILD. when i run the test with and without traces turned on i see a ~15% loss on max operations/sec/node. again this is another extreme case as there are only few workloads which have a metadata workload component as heavy as SWBUILD. but given that between the 2 extreme cases we are talking about 8 - 15% overhead its is very likely that all real world workloads suffer less than 15% when traces are turned on.</div><div>only a test in your environment will really show . </div><div><br></div><div>i would not recommend to try this with any version prior 4.2.1 as the impact will be significant higher than what i measured. hope this helps make a informed decision. </div><div><br></div><div>Sven<br></div><div><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Tue, Mar 7, 2017 at 9:32 PM Oesterlin, Robert <<a href="mailto:Robert.Oesterlin@nuance.com">Robert.Oesterlin@nuance.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72" class="gmail_msg">
<div class="m_566600162184951063WordSection1 gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">I’m considering enabling trace on all nodes all the time, doing something like this:<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">mmtracectl --set --trace=def --trace-recycle=global --tracedev-write-mode=overwrite --tracedev-overwrite-buffer-size=256M<br class="gmail_msg">
mmtracectl --start<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">My questions are:<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">- What is the performance penalty of leaving this on 100% of the time on a node?<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">- Does anyone have any suggestions on automation on stopping trace when a particular event occurs?
<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">- What other issues, if any?<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg">Bob Oesterlin<br class="gmail_msg">
Sr Principal Storage Engineer, Nuance<br class="gmail_msg">
<a href="tel:(507)%20269-0413" value="+15072690413" class="gmail_msg" target="_blank">507-269-0413</a><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"><span style="font-family:"Times New Roman"" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
</div>
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div></div></div>