<font size=3 face="Arial">Have you considered keeping the 1G network for
daemon traffic and moving the data traffic to another network?</font><br><br><font size=3 face="Arial">Given the description of your configuration
with only 2 manager nodes handling mmbackup and other tasks my guess is
that is where the problem lies regarding performance when mmbackup is running
with the many nodes accessing a single file. You said the fs managers
were on hardware, does that mean other nodes in this cluster are VMs of
some kind?</font><br><br><font size=3 face="Arial">You stated that your NSD servers were under
powered. Did you address that problem in any way, that is adding
memory/CPUs, or did you just move other GPFS activity off of those nodes?</font><br><br><font size=3 face="sans-serif">Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Peter Childs <p.childs@qmul.ac.uk></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">07/23/2018 07:06 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Same file opened by many nodes / processes</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>On Mon, 2018-07-23 at 22:13 +1200, José Filipe Higino
wrote:</font><br><font size=3>I think the network problems need to be cleared first.
Then I would investigate further. </font><br><br><font size=3>Buf if that is not a trivial path... </font><br><font size=3>Are you able to understand from the mmfslog what happens
when the tipping point occurs?</font><br><br><font size=3>mmfslog thats not a term I've come accross before, if
you mean /var/adm/ras/mmfs.log.latest then I'm already there is not a lot
there, In other words no expulsions or errors just a very slow filesystem,
We've not seen any significantly long waiters either (mmdiag --waiters)
so as far as I can see its just behaving like a very very busy filesystem.</font><br><br><font size=3>We've already had IBM looking at the snaps due to the
rather slow mmbackup process, all I've had back is to try increase -a ie
the number of sort threads which has speed it up to a certain extent, But
once again I think we're looking at the results of the issue not the cause.</font><br><br><br><font size=3>In my view, when troubleshooting is not easy, the usual
methods work/help to find the next step:</font><br><font size=3>- Narrow the window of troubleshooting (by discarding
"for now" events that did not happen within the same timeframe)</font><br><font size=3>- Use "as precise" as possible, timebased events
to read the reaction of the cluster (via log or others) and make
assumptions about other observed situations.</font><br><font size=3>- If possible and when the problem is happening, run some
traces, gpfs.snap and ask for support via PMR.</font><br><br><font size=3>Also,</font><br><br><font size=3>What is version of GPFS?</font><br><br><font size=3>4.2.3-8 </font><br><br><font size=3>How many quorum nodes?</font><br><br><font size=3>4 Quorum nodes with tie breaker disks, however these are
not the file system manager nodes as to fix a previous problem (with our
nsd servers not being powerful enough) our fsmanager nodes are on hardware,
We have two file system manager nodes (Which do token management, quota
management etc) they also run the mmbackup.</font><br><br><font size=3>How many filesystems?</font><br><br><font size=3>1, although we do have a second that is accessed via multi-cluster
from our older GPFS setup, (thats running 4.2.3-6 currently)</font><br><br><font size=3>Is the management network the same as the daemon network?</font><br><br><font size=3>Yes. the management network and the daemon network are
the same network. </font><br><br><font size=3>Thanks in advance</font><br><br><font size=3>Peter Childs</font><br><br><br><br><font size=3>On Mon, 23 Jul 2018 at 20:37, Peter Childs <</font><a href="mailto:p.childs@qmul.ac.uk"><font size=3 color=blue><u>p.childs@qmul.ac.uk</u></font></a><font size=3>>
wrote:</font><br><font size=3>On Mon, 2018-07-23 at 00:51 +1200, José Filipe Higino
wrote:</font><br><br><font size=3>Hi there, </font><br><br><font size=3>Have you been able to create a test case (replicate the
problem)? Can you tell us a bit more about the setup?</font><br><br><font size=3>Not really, It feels like a perfect storm, any one of
the tasks running on its own would be fine, Its the shear load, our mmpmon
data says the storage has been flat lining when it occurs.</font><br><br><font size=3>Its a reasonably standard (small) HPC cluster, with a
very mixed work load, hence while we can usually find "bad" jobs
from the point of view of io on this occasion we can see a few large array
jobs all accessing the same file, the cluster runs fine until we get to
a certain point and one more will tip the balance. We've been attempting
to limit the problem by adding limits to the number of jobs in an array
that can run at once. But that feels like fire fighting. </font><br><br><br><font size=3>Are you using GPFS API over any administrative commands?
Any problems with the network (being that Ethernet or IB)?</font><br><br><font size=3>We're not as using the GPFS API, never got it working,
which is a shame, I've never managed to figure out the setup, although
it is on my to do list.</font><br><br><font size=3>Network wise, We've just removed a great deal of noise
from arp requests by increasing the arp cache size on the nodes. Its a
mixed 1GBit/10GBit network currently, we're currently looking at removing
all the 1GBit nodes within the next few months and adding some new faster
kit. The Storage is attached at 40GBit but it does not look to want to
run much above 5Gbit I suspect due to Ethernet back off due to the mixed
speeds. </font><br><br><font size=3>While we do have some IB we don't currently run our storage
over it.</font><br><br><font size=3>Thanks in advance</font><br><br><font size=3>Peter Childs</font><br><br><br><br><br><br><font size=3>Sorry if I am un-announced here for the first time. But
I would like to help if I can.</font><br><br><font size=3>Jose Higino,</font><br><font size=3>from NIWA</font><br><font size=3>New Zealand</font><br><br><font size=3>Cheers</font><br><br><font size=3>On Sun, 22 Jul 2018 at 23:26, Peter Childs <</font><a href="mailto:p.childs@qmul.ac.uk" target="_blank"><font size=3 color=blue><u>p.childs@qmul.ac.uk</u></font></a><font size=3>>
wrote:</font><br><font size=3>Yes, we run mmbackup, using a snapshot.<br><br>The scan usally takes an hour, but for the last week has been taking many
hours (i saw it take 12 last Tuesday)<br><br>It's speeded up again now back to its normal hour, but the high io jobs
accessing the same file from many nodes also look to have come to an end
for the time being.<br><br>I was trying to figure out howto control the bad io using mmchqos, to prioritise
certain nodes over others but had not worked out if that was possible yet.<br><br>We've only previously seen this problem when we had some bad disks in our
storage, which we replaced, I've checked and I can't see that issue currently.<br><br>Thanks for the help.<br><br><br><br>Peter Childs<br>Research Storage<br>ITS Research and Teaching Support<br>Queen Mary, University of London<br><br>---- Yaron Daniel wrote ----<br></font><br><font size=2>Hi</font><font size=3><br></font><font size=2><br>Do u run mmbackup on snapshot , which is read only ?</font><font size=3><br></font><font size=1 face="Arial"><br> </font><font size=2 face="Arial"><br>Regards</font><font size=1 face="Arial"><br> </font><table width=780 style="border-collapse:collapse;"><tr height=8><td width=778 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><div align=center><hr noshade></div><br><font size=1 face="Arial"><br> </font><tr height=8><td width=778 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=1 face="Arial"> </font><tr height=8><td width=512 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=blue face="Arial"><b>Yaron
Daniel</b></font><td width=166 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial"> 94
Em Ha'Moshavot Rd</font><td width=94 rowspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><div align=right></div><tr height=8><td width=512 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=blue face="Arial"><b>Storage
Architect – IL Lab Services (Storage)</b></font><td width=166 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial"> Petach
Tiqva, 49527</font><tr height=8><td width=512 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=blue face="Arial"><b>IBM
Global Markets, Systems HW Sales</b></font><td width=166 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial"> Israel</font><tr height=8><td width=512 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=blue face="Arial"><b> </b></font><td width=166 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial"> </font><td width=94 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=1 face="Arial"> </font><tr height=8><td width=89 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">Phone:</font><td width=421 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">+972-3-916-5672</font><td width=166 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial"> </font><td width=94 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2> </font><tr height=8><td width=89 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">Fax:</font><td width=421 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">+972-3-916-5672</font><td width=166 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">
</font><td width=94 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2> </font><tr height=8><td width=89 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">Mobile:</font><td width=421 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">+972-52-8395593</font><td width=166 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">
</font><td width=94 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2> </font><tr height=8><td width=89 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">e-mail:</font><td width=421 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><a href="mailto:yard@il.ibm.com" target="_blank"><font size=2 color=blue face="Arial"><u>yard@il.ibm.com</u></font></a><td width=166 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">
</font><td width=94 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2> </font><tr height=8><td width=512 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><a href="http://www.ibm.com/il/he/" target="_blank"><font size=2 color=blue face="Arial"><u>IBM
Israel</u></font></a><td width=166 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2 color=#5f5f5f face="Arial">
</font><td width=94 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=2> </font><tr height=8><td width=778 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=1 color=#5f5f5f face="Arial"> </font><tr height=8><td width=778 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=1 color=#5f5f5f face="Arial"> </font></table><p><font size=3><br> <br><br><br></font><font size=1 color=#5f5f5f><br>From: </font><font size=1>Peter Childs <</font><a href="mailto:p.childs@qmul.ac.uk" target="_blank"><font size=1 color=blue><u>p.childs@qmul.ac.uk</u></font></a><font size=1>></font><font size=1 color=#5f5f5f><br>To: </font><font size=1>"</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><font size=1 color=blue><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=1>"
<</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><font size=1 color=blue><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=1>></font><font size=1 color=#5f5f5f><br>Date: </font><font size=1>07/10/2018 05:51 PM</font><font size=1 color=#5f5f5f><br>Subject: </font><font size=1>[gpfsug-discuss]
Same file opened by many nodes / processes</font><font size=1 color=#5f5f5f><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><font size=1 color=blue><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3><br></font><hr noshade><font size=3><br><br></font><tt><font size=2><br>We have an situation where the same file is being read by around 5000<br>"jobs" this is an array job in uge with a tc set, so the file
in<br>question is being opened by about 100 processes/jobs at the same time.<br><br>Its a ~200GB file so copying the file locally first is not an easy<br>answer, and these jobs are causing issues with mmbackup scanning the<br>file system, in that the scan is taking 3 hours instead of the normal<br>40-60 minutes.<br><br>This is read only access to the file, I don't know the specifics about<br>the job.<br><br>It looks like the metanode is moving around a fair amount (given what I<br>can see from mmfsadm saferdump file)<br><br>I'm wondering if we there is anything we can do to improve things or<br>that can be tuned within GPFS, I'm don't think we have an issue with<br>token management, but would increasing maxFileToCache on our token<br>manager node help say?<br><br>Is there anything else I should look at, to try and attempt to allow<br>GPFS to share this file better.<br><br>Thanks in advance<br><br>Peter Childs<br></font></tt><br><tt><font size=2>_______________________________________________</font></tt><br><tt><font size=2>gpfsug-discuss mailing list</font></tt><br><tt><font size=2>gpfsug-discuss at </font></tt><a href="http://spectrumscale.org" target="_blank"><tt><font size=2 color=blue><u>spectrumscale.org</u></font></tt></a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size=2 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><br><br><tt><font size=3>-- </font></tt><br><font size=3>Peter Childs</font><br><font size=3>ITS Research Storage</font><br><font size=3>Queen Mary, University of London</font><br><br><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org" target="_blank"><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><tt><font size=3>_______________________________________________</font></tt><br><tt><font size=3>gpfsug-discuss mailing list</font></tt><br><tt><font size=3>gpfsug-discuss at spectrumscale.org</font></tt><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><br><br><tt><font size=3>-- </font></tt><br><font size=3>Peter Childs</font><br><font size=3>ITS Research Storage</font><br><font size=3>Queen Mary, University of London</font><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>