<div dir="ltr">yes , but i would do this in stages given how large your system is. <div>pick one set of nodes (lets say) 100 out of 200 that do similar things and turn tracing on there.</div><div>this will give you data you can compare between the 2 set of nodes. </div><div>let it run for a week and if the data with your real workload doesn't show any significant degradation (which is what i expect) turn it on everywhere. </div><div>the one thing i am not 100% sure about is size of trace buffer as well as the global cut config. what this means is if you apply the settings as mentioned in this first post, if one node asserts in your cluster you will cut a trace on all nodes that will write a 256M buffer into your dump file location. </div><div>if you have a node thats in an assert loop (asserts, restarts , asserts) this can cause significant load on all nodes. therefore i would probably start without cutting a global trace and reduce the trace size to 64M. </div><div>i (and i am sure other dev folks) would be very interested in the outcome as we have this debate on a yearly basis if we shouldn't just turn tracing on by default, in the past performance was the biggest hurdle, this is solved now (my claim) . next big questions is how well does that work on larger scale systems with production workloads. as more feedback we will get in this area as better we can make informed decision how and if it could be turned on all the time and work harder on handling cases like i mentioned above to mitigate the risks .</div><div><br></div><div>sven</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Mar 8, 2017 at 2:56 PM Oesterlin, Robert <<a href="mailto:Robert.Oesterlin@nuance.com">Robert.Oesterlin@nuance.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72" class="gmail_msg">
<div class="m_282378992489647162WordSection1 gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg">As always, Sven comes in to back this up with real data :)<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg">To net this out, Sven – I should be able enable trace on my NSD servers running 4.2.2 without much impact, correct?<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p></div></div><div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72" class="gmail_msg"><div class="m_282378992489647162WordSection1 gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-family:Calibri" class="gmail_msg">Bob Oesterlin<br class="gmail_msg">
Sr Principal Storage Engineer, Nuance<br class="gmail_msg">
<br class="gmail_msg">
<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
</div></div><div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72" class="gmail_msg"><div class="m_282378992489647162WordSection1 gmail_msg"><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class="gmail_msg">
<p class="MsoNormal gmail_msg"><b class="gmail_msg"><span style="font-family:Calibri;color:black" class="gmail_msg">From: </span>
</b><span style="font-family:Calibri;color:black" class="gmail_msg"><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>> on behalf of Sven Oehme <<a href="mailto:oehmes@gmail.com" class="gmail_msg" target="_blank">oehmes@gmail.com</a>><br class="gmail_msg">
<b class="gmail_msg">Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br class="gmail_msg">
<b class="gmail_msg">Date: </b>Wednesday, March 8, 2017 at 7:37 AM</span></p></div></div></div><div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72" class="gmail_msg"><div class="m_282378992489647162WordSection1 gmail_msg"><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class="gmail_msg"><p class="MsoNormal gmail_msg"><span style="font-family:Calibri;color:black" class="gmail_msg"><br class="gmail_msg">
<b class="gmail_msg">To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br class="gmail_msg">
</span></p></div></div></div><div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72" class="gmail_msg"><div class="m_282378992489647162WordSection1 gmail_msg"><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class="gmail_msg"><p class="MsoNormal gmail_msg"><span style="font-family:Calibri;color:black" class="gmail_msg"><b class="gmail_msg">Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Potential problems - leaving trace enabled in over-write mode?<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
</div></div></div><div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72" class="gmail_msg"><div class="m_282378992489647162WordSection1 gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-family:"-webkit-standard","serif";color:black" class="gmail_msg">starting in version 3.4 we enhanced the trace code of scale significant. this went on release to release all the way up to 4.2.1. since 4.2.1 we made further improvements,
but much smaller changes, more optimization , e.g. reducing of trace levels verbosity, etc .<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-family:"-webkit-standard","serif";color:black" class="gmail_msg">with 4.2.2 we switched from blocking traces to in memory traces as the default trace infrastructure, this infrastructure was designed to be turned on all the time with minimal
impact on performance. <u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
</div>
</div></div><div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72" class="gmail_msg"><div class="m_282378992489647162WordSection1 gmail_msg"></div>
</div>
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div>