<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255); text-align: left;" dir="auto">
Ryan,</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255); text-align: left;" dir="auto">
<br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255); text-align: left;" dir="auto">
I appreciate your support - I finally got some on a WebEx now.</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255); text-align: left;" dir="auto">
<br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255); text-align: left;" dir="auto">
I'll share any useful information I glean from the session.</div>
<div id="ms-outlook-mobile-signature">
<div><br>
</div>
Thanks,<br>
<br>
<br>
Oluwasijibomi (Siji) Saula<br>
<br>
HPC Systems Administrator / Information Technology<br>
<br>
<br>
<br>
Research 2 Building 220B / Fargo ND 58108-6050<br>
<br>
p: 701.231.7749 / <a href="http://www.ndsu.edu" target="_blank">www.ndsu.edu</a><br>
<br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of gpfsug-discuss-request@spectrumscale.org
<gpfsug-discuss-request@spectrumscale.org><br>
<b>Sent:</b> Saturday, September 19, 2020 6:45:47 PM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> gpfsug-discuss Digest, Vol 104, Issue 19</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Send gpfsug-discuss mailing list submissions to<br>
gpfsug-discuss@spectrumscale.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
gpfsug-discuss-request@spectrumscale.org<br>
<br>
You can reach the person managing the list at<br>
gpfsug-discuss-owner@spectrumscale.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: gpfsug-discuss Digest, Vol 104, Issue 18<br>
(Saula, Oluwasijibomi)<br>
2. Re: gpfsug-discuss Digest, Vol 104, Issue 18 (Ryan Novosielski)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Sat, 19 Sep 2020 20:52:19 +0000<br>
From: "Saula, Oluwasijibomi" <oluwasijibomi.saula@ndsu.edu><br>
To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 104, Issue 18<br>
Message-ID:<br>
<DM6PR08MB5324B069B9BE106DA9EA6B2B983C0@DM6PR08MB5324.namprd08.prod.outlook.com><br>
<br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Ryan,<br>
<br>
We've been at severity 1 since about 4am with only a single response all day.<br>
<br>
Got me a bit concerned now...<br>
<br>
<br>
Thanks,<br>
<br>
<br>
Oluwasijibomi (Siji) Saula<br>
<br>
HPC Systems Administrator / Information Technology<br>
<br>
<br>
<br>
Research 2 Building 220B / Fargo ND 58108-6050<br>
<br>
p: 701.231.7749 / www.ndsu.edu<<a href="http://www.ndsu.edu/">http://www.ndsu.edu/</a>><br>
<br>
<br>
<br>
[cid:image001.gif@01D57DE0.91C300C0]<br>
<br>
<br>
<br>
________________________________<br>
From: gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of gpfsug-discuss-request@spectrumscale.org <gpfsug-discuss-request@spectrumscale.org><br>
Sent: Saturday, September 19, 2020 3:23 PM<br>
To: gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><br>
Subject: gpfsug-discuss Digest, Vol 104, Issue 18<br>
<br>
Send gpfsug-discuss mailing list submissions to<br>
gpfsug-discuss@spectrumscale.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
gpfsug-discuss-request@spectrumscale.org<br>
<br>
You can reach the person managing the list at<br>
gpfsug-discuss-owner@spectrumscale.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. CCR errors (Saula, Oluwasijibomi)<br>
2. Re: CCR errors (Ryan Novosielski)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Sat, 19 Sep 2020 20:11:31 +0000<br>
From: "Saula, Oluwasijibomi" <oluwasijibomi.saula@ndsu.edu><br>
To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br>
Subject: [gpfsug-discuss] CCR errors<br>
Message-ID:<br>
<DM6PR08MB5324EE3B4D20A1FFD6BBCF66983C0@DM6PR08MB5324.namprd08.prod.outlook.com><br>
<br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
Hello,<br>
<br>
Anyone available to assist with CCR errors:<br>
<br>
<br>
<br>
[root@nsd02 ~]# mmchnode -N nsd04-ib --quorum --manager<br>
<br>
mmchnode: Unable to obtain the GPFS configuration file lock. Retrying ...<br>
<br>
<br>
Per IBM support's direction, I already followed the Manual Repair Procedure<<a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1pdg_noccrreco_multinode.htm">https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1pdg_noccrreco_multinode.htm</a>>,
but now I'm back to square one with the same issue.<br>
<br>
Also, I have a ticket over to IBM for troubleshooting this issue during our downtime this weekend, but support's response is really slow.<br>
<br>
If possible, I'd prefer a webex session to facilitate closure, but I can send emails back and forth if necessary.<br>
<br>
<br>
Thanks,<br>
<br>
<br>
Oluwasijibomi (Siji) Saula<br>
<br>
HPC Systems Administrator / Information Technology<br>
<br>
<br>
<br>
Research 2 Building 220B / Fargo ND 58108-6050<br>
<br>
p: 701.231.7749 / www.ndsu.edu<<a href="http://www.ndsu.edu/">http://www.ndsu.edu/</a>><br>
<br>
<br>
<br>
[cid:image001.gif@01D57DE0.91C300C0]<br>
<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/f163918c/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/f163918c/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Sat, 19 Sep 2020 20:23:01 +0000<br>
From: Ryan Novosielski <novosirj@rutgers.edu><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] CCR errors<br>
Message-ID: <EBDC9F2B-9DA3-42B8-8592-2F96B04F1A76@rutgers.edu><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
I find them to be pretty fast and very experienced at severity 1. Don?t hesitate to use it (unless that?s already where you?re at).<br>
<br>
--<br>
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - novosirj@rutgers.edu<mailto:novosirj@rutgers.edu><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'<br>
<br>
On Sep 19, 2020, at 16:11, Saula, Oluwasijibomi <oluwasijibomi.saula@ndsu.edu> wrote:<br>
<br>
?<br>
Hello,<br>
<br>
Anyone available to assist with CCR errors:<br>
<br>
<br>
<br>
[root@nsd02 ~]# mmchnode -N nsd04-ib --quorum --manager<br>
<br>
mmchnode: Unable to obtain the GPFS configuration file lock. Retrying ...<br>
<br>
<br>
Per IBM support's direction, I already followed the Manual Repair Procedure<<a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1pdg_noccrreco_multinode.htm">https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1pdg_noccrreco_multinode.htm</a>>,
but now I'm back to square one with the same issue.<br>
<br>
Also, I have a ticket over to IBM for troubleshooting this issue during our downtime this weekend, but support's response is really slow.<br>
<br>
If possible, I'd prefer a webex session to facilitate closure, but I can send emails back and forth if necessary.<br>
<br>
<br>
Thanks,<br>
<br>
<br>
Oluwasijibomi (Siji) Saula<br>
<br>
HPC Systems Administrator / Information Technology<br>
<br>
<br>
<br>
Research 2 Building 220B / Fargo ND 58108-6050<br>
<br>
p: 701.231.7749 / www.ndsu.edu<<a href="http://www.ndsu.edu/">http://www.ndsu.edu/</a>><br>
<br>
<br>
<br>
[cid:image001.gif@01D57DE0.91C300C0]<br>
<br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/0b8d7715/attachment.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/0b8d7715/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 104, Issue 18<br>
***********************************************<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/0ee97239/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/0ee97239/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Sat, 19 Sep 2020 23:45:41 +0000<br>
From: Ryan Novosielski <novosirj@rutgers.edu><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 104, Issue 18<br>
Message-ID: <AE20D3E0-4824-486A-A2AD-564270BFFF07@rutgers.edu><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
I?d call in/click the lack of response thing (if I?m remembering that button right for the current system). That?s unusual. Maybe it failed to transition time zones.<br>
<br>
I?d help you, but I don?t know how to fix that one.<br>
<br>
--<br>
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - novosirj@rutgers.edu<mailto:novosirj@rutgers.edu><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'<br>
<br>
On Sep 19, 2020, at 16:52, Saula, Oluwasijibomi <oluwasijibomi.saula@ndsu.edu> wrote:<br>
<br>
?<br>
Ryan,<br>
<br>
We've been at severity 1 since about 4am with only a single response all day.<br>
<br>
Got me a bit concerned now...<br>
<br>
<br>
Thanks,<br>
<br>
<br>
Oluwasijibomi (Siji) Saula<br>
<br>
HPC Systems Administrator / Information Technology<br>
<br>
<br>
<br>
Research 2 Building 220B / Fargo ND 58108-6050<br>
<br>
p: 701.231.7749 / www.ndsu.edu<<a href="http://www.ndsu.edu/">http://www.ndsu.edu/</a>><br>
<br>
<br>
<br>
[cid:image001.gif@01D57DE0.91C300C0]<br>
<br>
<br>
<br>
________________________________<br>
From: gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of gpfsug-discuss-request@spectrumscale.org <gpfsug-discuss-request@spectrumscale.org><br>
Sent: Saturday, September 19, 2020 3:23 PM<br>
To: gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><br>
Subject: gpfsug-discuss Digest, Vol 104, Issue 18<br>
<br>
Send gpfsug-discuss mailing list submissions to<br>
gpfsug-discuss@spectrumscale.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
gpfsug-discuss-request@spectrumscale.org<br>
<br>
You can reach the person managing the list at<br>
gpfsug-discuss-owner@spectrumscale.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. CCR errors (Saula, Oluwasijibomi)<br>
2. Re: CCR errors (Ryan Novosielski)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Sat, 19 Sep 2020 20:11:31 +0000<br>
From: "Saula, Oluwasijibomi" <oluwasijibomi.saula@ndsu.edu><br>
To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br>
Subject: [gpfsug-discuss] CCR errors<br>
Message-ID:<br>
<DM6PR08MB5324EE3B4D20A1FFD6BBCF66983C0@DM6PR08MB5324.namprd08.prod.outlook.com><br>
<br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
Hello,<br>
<br>
Anyone available to assist with CCR errors:<br>
<br>
<br>
<br>
[root@nsd02 ~]# mmchnode -N nsd04-ib --quorum --manager<br>
<br>
mmchnode: Unable to obtain the GPFS configuration file lock. Retrying ...<br>
<br>
<br>
Per IBM support's direction, I already followed the Manual Repair Procedure<<a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1pdg_noccrreco_multinode.htm">https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1pdg_noccrreco_multinode.htm</a>>,
but now I'm back to square one with the same issue.<br>
<br>
Also, I have a ticket over to IBM for troubleshooting this issue during our downtime this weekend, but support's response is really slow.<br>
<br>
If possible, I'd prefer a webex session to facilitate closure, but I can send emails back and forth if necessary.<br>
<br>
<br>
Thanks,<br>
<br>
<br>
Oluwasijibomi (Siji) Saula<br>
<br>
HPC Systems Administrator / Information Technology<br>
<br>
<br>
<br>
Research 2 Building 220B / Fargo ND 58108-6050<br>
<br>
p: 701.231.7749 / www.ndsu.edu<<a href="http://www.ndsu.edu/">http://www.ndsu.edu/</a>><br>
<br>
<br>
<br>
[cid:image001.gif@01D57DE0.91C300C0]<br>
<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/f163918c/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/f163918c/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Sat, 19 Sep 2020 20:23:01 +0000<br>
From: Ryan Novosielski <novosirj@rutgers.edu><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] CCR errors<br>
Message-ID: <EBDC9F2B-9DA3-42B8-8592-2F96B04F1A76@rutgers.edu><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
I find them to be pretty fast and very experienced at severity 1. Don?t hesitate to use it (unless that?s already where you?re at).<br>
<br>
--<br>
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - novosirj@rutgers.edu<mailto:novosirj@rutgers.edu><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'<br>
<br>
On Sep 19, 2020, at 16:11, Saula, Oluwasijibomi <oluwasijibomi.saula@ndsu.edu> wrote:<br>
<br>
?<br>
Hello,<br>
<br>
Anyone available to assist with CCR errors:<br>
<br>
<br>
<br>
[root@nsd02 ~]# mmchnode -N nsd04-ib --quorum --manager<br>
<br>
mmchnode: Unable to obtain the GPFS configuration file lock. Retrying ...<br>
<br>
<br>
Per IBM support's direction, I already followed the Manual Repair Procedure<<a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1pdg_noccrreco_multinode.htm">https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1pdg_noccrreco_multinode.htm</a>>,
but now I'm back to square one with the same issue.<br>
<br>
Also, I have a ticket over to IBM for troubleshooting this issue during our downtime this weekend, but support's response is really slow.<br>
<br>
If possible, I'd prefer a webex session to facilitate closure, but I can send emails back and forth if necessary.<br>
<br>
<br>
Thanks,<br>
<br>
<br>
Oluwasijibomi (Siji) Saula<br>
<br>
HPC Systems Administrator / Information Technology<br>
<br>
<br>
<br>
Research 2 Building 220B / Fargo ND 58108-6050<br>
<br>
p: 701.231.7749 / www.ndsu.edu<<a href="http://www.ndsu.edu/">http://www.ndsu.edu/</a>><br>
<br>
<br>
<br>
[cid:image001.gif@01D57DE0.91C300C0]<br>
<br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/0b8d7715/attachment.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/0b8d7715/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 104, Issue 18<br>
***********************************************<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/4f220e5f/attachment.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200919/4f220e5f/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 104, Issue 19<br>
***********************************************<br>
</div>
</span></font></div>
</body>
</html>