<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
</head>
<body>
<div name="messageBodySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">Yes,
<div><br /></div>
<div>We have upgraded to 5.0.1-0.5, which has the patch for the issue.</div>
<div>The related IBM case number was : <span style="font-family: 'Helvetica Neue'; font-size: 12px;">TS001010674</span> </div>
</div>
<div name="messageSignatureSection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;"><br />
Regards,<br />
Lohit</div>
<div name="messageReplySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;"><br />
On Nov 2, 2018, 12:27 PM -0400, Mazurkova, Svetlana/Information Systems <sveta@cbio.mskcc.org>, wrote:<br />
<blockquote type="cite" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #1abc9c;">Hi Damir,
<div class=""><br class="" /></div>
<div class="">It was related to specific user jobs and mmap (?). We opened PMR with IBM and have patch from IBM, since than we don’t see issue.</div>
<div class=""><br class="" /></div>
<div class="">Regards,</div>
<div class=""><br class="" /></div>
<div class="">Sveta.</div>
<div class=""><br class="" />
<div>
<blockquote type="cite" class="" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #e67e22;">
<div class="">On Nov 2, 2018, at 11:55 AM, Damir Krstic <<a href="mailto:damir.krstic@gmail.com" class="">damir.krstic@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline" />
<div class="">
<div dir="ltr" class="">Hi,
<div class=""><br class="" /></div>
<div class="">Did you ever figure out the root cause of the issue? We have recently (end of the June) upgraded our storage to: gpfs.base-5.0.0-1.1.3.ppc64</div>
<div class=""><br class="" /></div>
<div class="">In the last few weeks we have seen an increasing number of ps hangs across compute and login nodes on our cluster. The filesystem version (of all filesystems on our cluster) is:</div>
<div class="">
<div class=""> -V 15.01 (4.2.0.0) File system version</div>
<div class=""><br class="" /></div>
<div class="">I am just wondering if anyone has seen this type of issue since you first reported it and if there is a known fix for it.</div>
<div class=""><br class="" /></div>
<div class="">Damir</div>
<br class="" />
<div class="gmail_quote">
<div dir="ltr" class="">On Tue, May 22, 2018 at 10:43 AM <<a href="mailto:valleru@cbio.mskcc.org" class="">valleru@cbio.mskcc.org</a>> wrote:<br class="" /></div>
<blockquote class="gmail_quote" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #3498db;">
<div class="">
<div name="messageBodySection" style="font-size:14px;font-family:-apple-system,BlinkMacSystemFont,sans-serif" class="">Hello All,
<div class=""><br class="" /></div>
<div class="">We have recently upgraded from GPFS 4.2.3.2 to GPFS 5.0.0-2 about a month ago. We have not yet converted the 4.2.2.2 filesystem version to 5. ( That is we have not run the mmchconfig release=LATEST command)</div>
<div class="">Right after the upgrade, we are seeing many “ps hangs" across the cluster. All the “ps hangs” happen when jobs run related to a Java process or many Java threads (example: GATK )</div>
<div class="">The hangs are pretty random, and have no particular pattern except that we know that it is related to just Java or some jobs reading from directories with about 600000 files.</div>
<div class=""><span style="font-family:'Helvetica Neue';font-size:12px" class=""><br class="" /></span></div>
<div class="">I have raised an IBM critical service request about a month ago related to this - PMR: 24090,L6Q,000. </div>
<div class="">However, According to the ticket - they seemed to feel that it might not be related to GPFS. </div>
<div class="">Although, we are sure that these hangs started to appear only after we upgraded GPFS to GPFS 5.0.0.2 from 4.2.3.2.</div>
<div class=""><br class="" /></div>
<div class="">One of the other reasons we are not able to prove that it is GPFS is because, we are unable to capture any logs/traces from GPFS once the hang happens.</div>
<div class="">Even GPFS trace commands hang, once “ps hangs” and thus it is getting difficult to get any dumps from GPFS.</div>
<div class=""><br class="" /></div>
<div class="">Also - According to the IBM ticket, they seemed to have a seen a “ps hang" issue and we have to run mmchconfig release=LATEST command, and that will resolve the issue.</div>
<div class="">However we are not comfortable making the permanent change to Filesystem version 5. and since we don’t see any near solution to these hangs - we are thinking of downgrading to GPFS 4.2.3.2 or the previous state that we know the cluster was stable.</div>
<div class=""><br class="" /></div>
<div class="">Can downgrading GPFS take us back to exactly the previous GPFS config state? </div>
<div class="">With respect to downgrading from 5 to 4.2.3.2 -> is it just that i reinstall all rpms to a previous version? or is there anything else that i need to make sure with respect to GPFS configuration?</div>
<div class="">Because i think that GPFS 5.0 might have updated internal default GPFS configuration parameters , and i am not sure if downgrading GPFS will change them back to what they were in GPFS 4.2.3.2</div>
<div class=""><br class="" /></div>
<div class="">Our previous state:</div>
<div class=""><br class="" /></div>
<div class="">2 Storage clusters - 4.2.3.2</div>
<div class="">1 Compute cluster - 4.2.3.2 ( remote mounts the above 2 storage clusters )</div>
<div class=""><br class="" /></div>
<div class="">Our current state:</div>
<div class=""><br class="" /></div>
<div class="">2 Storage clusters - 5.0.0.2 ( filesystem version - 4.2.2.2)<br class="" /></div>
<div class="">1 Compute cluster - 5.0.0.2</div>
<div class=""><br class="" /></div>
<div class="">Do i need to downgrade all the clusters to go to the previous state ? or is it ok if we just downgrade the compute cluster to previous version?</div>
<div class=""><br class="" /></div>
<div class="">Any advice on the best steps forward, would greatly help.</div>
<div class=""><br class="" /></div>
<div class="">Thanks,</div>
</div>
<div name="messageSignatureSection" style="font-size:14px;font-family:-apple-system,BlinkMacSystemFont,sans-serif" class=""><br class="" />
<div class="m_-1971852391905757129matchFont">Lohit</div>
</div>
</div>
_______________________________________________<br class="" />
gpfsug-discuss mailing list<br class="" />
gpfsug-discuss at <a href="http://spectrumscale.org/" rel="noreferrer" target="_blank" class="">spectrumscale.org</a><br class="" />
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="" /></blockquote>
</div>
</div>
</div>
_______________________________________________<br class="" />
gpfsug-discuss mailing list<br class="" />
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="" />
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="" /></div>
</blockquote>
</div>
<br class="" /></div>
_______________________________________________<br />
gpfsug-discuss mailing list<br />
gpfsug-discuss at spectrumscale.org<br />
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<br /></blockquote>
</div>
</body>
</html>