<font size=2 face="sans-serif">There is a known performance issue that
can possibly cause longer than expected network time-outs if you are running
iohist too often. So be careful it is best to collect it as a sample, instead
of all of the time. </font><br><br><br><font size=2 face="sans-serif">Scott Fadden<br>Spectrum Scale - Technical Marketing <br>Phone: (503) 880-5833 <br>sfadden@us.ibm.com<br></font><a href=http://www.ibm.com/systems/storage/spectrum/scale><font size=2 face="sans-serif">http://www.ibm.com/systems/storage/spectrum/scale</font></a><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Aaron Knister <aaron.s.knister@nasa.gov></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif"><gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">08/29/2016 11:09 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
iowait?</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>Nice! Thanks Bryan. I wonder what the implications
are of setting it to <br>something high enough that we could capture data every 10s. I figure if
<br>512 events only takes me to 1 second I would need to log in the realm of
<br>10k to capture every 10 seconds and account for spikes in I/O.<br><br>-Aaron<br><br>On 8/29/16 2:06 PM, Bryan Banister wrote:<br>> Try this:<br>><br>> mmchconfig ioHistorySize=1024 # Or however big you want!<br>><br>> Cheers,<br>> -Bryan<br>><br>> -----Original Message-----<br>> From: gpfsug-discuss-bounces@spectrumscale.org [</font></tt><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><tt><font size=2>mailto:gpfsug-discuss-bounces@spectrumscale.org</font></tt></a><tt><font size=2>]
On Behalf Of Aaron Knister<br>> Sent: Monday, August 29, 2016 1:05 PM<br>> To: gpfsug main discussion list<br>> Subject: Re: [gpfsug-discuss] iowait?<br>><br>> That's an interesting idea. I took a look at mmdig --iohist on a busy
node it doesn't seem to capture more than literally 1 second of history.<br>> Is there a better way to grab the data or have gpfs capture more of
it?<br>><br>> Just to give some more context, as part of our monthly reporting requirements
we calculate job efficiency by comparing the number of cpu cores requested
by a given job with the cpu % utilization during that job's time window.
Currently a job that's doing a sleep 9000 would show up the same as a job
blocked on I/O. Having GPFS wait time included in iowait would allow us
to easily make this distinction.<br>><br>> -Aaron<br>><br>> On 8/29/16 1:56 PM, Bryan Banister wrote:<br>>> There is the iohist data that may have what you're looking for,
-Bryan<br>>><br>>> -----Original Message-----<br>>> From: gpfsug-discuss-bounces@spectrumscale.org<br>>> [</font></tt><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><tt><font size=2>mailto:gpfsug-discuss-bounces@spectrumscale.org</font></tt></a><tt><font size=2>]
On Behalf Of Aaron<br>>> Knister<br>>> Sent: Monday, August 29, 2016 12:54 PM<br>>> To: gpfsug-discuss@spectrumscale.org<br>>> Subject: Re: [gpfsug-discuss] iowait?<br>>><br>>> Sure, we can and we do use both iostat/sar and collectl to collect
disk utilization on our nsd servers. That doesn't give us insight, though,
into any individual client node of which we've got 3500. We do log mmpmon
data from each node but that doesn't give us any insight into how much
time is being spent waiting on I/O. Having GPFS report iowait on client
nodes would give us this insight.<br>>><br>>> On 8/29/16 1:50 PM, Alex Chekholko wrote:<br>>>> Any reason you can't just use iostat or collectl or any of
a number<br>>>> of other standards tools to look at disk utilization?<br>>>><br>>>> On 08/29/2016 10:33 AM, Aaron Knister wrote:<br>>>>> Hi Everyone,<br>>>>><br>>>>> Would it be easy to have GPFS report iowait values in
linux? This<br>>>>> would be a huge help for us in determining whether a node's
low<br>>>>> utilization is due to some issue with the code running
on it or if<br>>>>> it's blocked on I/O, especially in a historical context.<br>>>>><br>>>>> I naively tried on a test system changing schedule() in<br>>>>> cxiWaitEventWait() on line ~2832 in gpl-linux/cxiSystem.c
to this:<br>>>>><br>>>>> again:<br>>>>> /* call the scheduler */<br>>>>> if ( waitFlags & INTERRUPTIBLE )<br>>>>> schedule();<br>>>>> else<br>>>>> io_schedule();<br>>>>><br>>>>> Seems to actually do what I'm after but generally bad
things happen<br>>>>> when I start pretending I'm a kernel developer.<br>>>>><br>>>>> Any thoughts? If I open an RFE would this be something
that's<br>>>>> relatively easy to implement (not asking for a commitment
*to*<br>>>>> implement it, just that I'm not asking for something seemingly<br>>>>> simple that's actually fairly hard to implement)?<br>>>>><br>>>>> -Aaron<br>>>>><br>>>><br>>><br>>> --<br>>> Aaron Knister<br>>> NASA Center for Climate Simulation (Code 606.2) Goddard Space
Flight<br>>> Center<br>>> (301) 286-2776<br>>> _______________________________________________<br>>> gpfsug-discuss mailing list<br>>> gpfsug-discuss at spectrumscale.org<br>>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>>><br>>> ________________________________<br>>><br>>> Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited,
and to please notify the sender immediately and destroy this email and
any attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to
the completeness or accuracy of this email or any attachments. This email
is for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.<br>>> _______________________________________________<br>>> gpfsug-discuss mailing list<br>>> gpfsug-discuss at spectrumscale.org<br>>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>>><br>><br>> --<br>> Aaron Knister<br>> NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight
Center<br>> (301) 286-2776<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>><br>> ________________________________<br>><br>> Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited,
and to please notify the sender immediately and destroy this email and
any attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to
the completeness or accuracy of this email or any attachments. This email
is for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>><br><br>-- <br>Aaron Knister<br>NASA Center for Climate Simulation (Code 606.2)<br>Goddard Space Flight Center<br>(301) 286-2776<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br><br></font></tt><br><br><BR>