<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >add this line to the ticket/record information, when opening a service ticket ..</div>
<div dir="ltr" > </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" ><div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >fput failed: Version mismatch on conditional put (err 805)</font></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Ursprüngliche Nachricht -----<br>Von: "Luis Bolinches" <luis.bolinches@fi.ibm.com><br>Gesendet von: gpfsug-discuss-bounces@spectrumscale.org<br>An: gpfsug-discuss@spectrumscale.org<br>CC: gpfsug-discuss@spectrumscale.org<br>Betreff: [EXTERNAL] Re: [gpfsug-discuss] gpfsgui in a core dump/restart loop<br>Datum: Di, 30. Nov 2021 14:30<br>
<div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:12pt" ><div dir="ltr" >Hi</div>
<div dir="ltr" > </div>
<div dir="ltr" >Not really a solution ...</div>
<div dir="ltr" > </div>
<div dir="ltr" >first disable the systemd service</div>
<div dir="ltr" > </div>
<div dir="ltr" >systemd disable gpfsgui</div>
<div dir="ltr" > </div>
<div dir="ltr" >So at least does not go on this loop</div>
<div dir="ltr" > </div>
<div dir="ltr" >This can be indicative of few issues going on. 2 or more nodes trying to modify the same file; removed nodes that were perfmon; "too many" collectors on certain conditions; ... and probably many other.</div>
<div dir="ltr" > </div>
<div dir="ltr" >I strongly suggest you get the last round of generated dump data and open a case to IBM (assuming this is IBM, whoever else the vendor is if not). Maybe a snap with it to speed up things so there is a clear picture of the cluster and CCR nodes and collectors.</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" ><span style="font-size:10pt;" ><font face="Arial" >--<br>Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations</font> / <font face="Arial" >Salutacions</font><br><font face="Arial" >Luis Bolinches</font></span></div>
<div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" ><div><span style="font-size:10pt;" ><font face="Arial" >IBM Spectrum Scale development</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Mobile Phone: +358503112585</font></span></div>
<div> </div>
<div><span style="font-size:10pt;" ><a href="https://www.youracclaim.com/user/luis-bolinches" rel="noopener" target="_blank"><b><font face="Arial" >https://www.youracclaim.com/user/luis-bolinches</font></b></a></span></div>
<div> </div>
<div><div><span style="font-size:10pt;" ><font face="Arial" >Ab IBM Finland Oy</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Laajalahdentie 23</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >00330 Helsinki</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Uusimaa - Finland</font></span></div><br><span style="font-size:10pt;" ><b><font face="Arial" color="#7C7C5F" >"If you always give you will always have" -- Anonymous</font></b></span></div></div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div> </div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Losen, Stephen C (scl)" <scl@virginia.edu><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: "gpfsug main discussion list" <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] gpfsgui in a core dump/restart loop<br>Date: Tue, Nov 30, 2021 14:48<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Hi folks,<br>Our gpfsgui service keeps crashing and restarting. About every three minutes we get files like these in /var/crash/scalemgmt<br><br>-rw------- 1 scalemgmt scalemgmt 1067843584 Nov 30 06:54 core.20211130.065414.59174.0001.dmp<br>-rw-r--r-- 1 scalemgmt scalemgmt 2636747 Nov 30 06:54 javacore.20211130.065414.59174.0002.txt<br>-rw-r--r-- 1 scalemgmt scalemgmt 1903304 Nov 30 06:54 Snap.20211130.065414.59174.0003.trc<br>-rw-r--r-- 1 scalemgmt scalemgmt 202 Nov 30 06:54 jitdump.20211130.065414.59174.0004.dmp<br><br>The core.*.dmp files are cores from the java command.<br><br>And the below errors keep repeating in /var/adm/ras/mmsysmonitor.log.<br><br>Any suggestions? Thanks for any help.<br><br><br>2021-11-30_07:25:09.944-0500: [W] ET_gui Event=gui_down identifier= arg0=started arg1=stopped<br>2021-11-30_07:25:09.961-0500: [I] ET_gui state_change for service: gui to FAILED at 2021.11.30 07.25.09.961572<br>2021-11-30_07:25:09.963-0500: [I] ClientThread-4 received command: 'thresholds refresh collectors 4021694'<br>2021-11-30_07:25:09.964-0500: [I] ClientThread-4 reload collectors <br>2021-11-30_07:25:09.964-0500: [I] ClientThread-4 read_collectors <br>2021-11-30_07:25:10.059-0500: [W] ClientThread-4 QueryHandler: query response has no data results <br>2021-11-30_07:25:10.059-0500: [W] ClientThread-4 QueryProcessor::execute: Error sending query in execute, quitting<br>2021-11-30_07:25:10.060-0500: [W] ClientThread-4 QueryHandler: query response has no data results <br>2021-11-30_07:25:10.060-0500: [W] ClientThread-4 QueryProcessor::execute: Error sending query in execute, quitting<br>2021-11-30_07:25:10.061-0500: [I] ClientThread-4 _activate_rules_scheduler completed <br>2021-11-30_07:25:10.147-0500: [I] ET_gui Event=component_state_change identifier= arg0=GUI arg1=FAILED<br>2021-11-30_07:25:10.148-0500: [I] ET_gui StateChange: change_to=FAILED nodestate=DEGRADED CESState=UNKNOWN<br>2021-11-30_07:25:10.148-0500: [I] ET_gui Service gui state changed. isInRunningState=True, wasInRunningState=True. New state=4<br>2021-11-30_07:25:10.148-0500: [I] ET_gui Monitor: LocalState:FAILED Events:607 Entities:0 RT: 0.83<br>2021-11-30_07:25:11.975-0500: [W] ET_perfmon got rc (153) while executing ['/usr/lpp/mmfs/bin/mmccr', 'fput', 'collectors', '/var/mmfs/tmp/tmpq4ac8o', '-c 4021693']<br>2021-11-30_07:25:11.975-0500: [E] ET_perfmon fput failed: Version mismatch on conditional put (err 805)<br> - CCRProxy._run_ccr_command:256<br>2021-09-29_20:03:53.322-0500: [I] MainThread --------------------------------- <br>2021-11-30_07:25:04.553-0500: [D] ET_perfmon File collectors has no newer version than 4021693 - CCRProxy.get<a href="File:119" target="_blank">File:119</a><br>2021-11-30_07:25:11.975-0500: [W] ET_perfmon Conditional put for file collectors with version 4021693 failed<br>2021-11-30_07:25:11.975-0500: [W] ET_perfmon New version received, start new collectors update cycle<br>2021-11-30_07:25:11.976-0500: [I] ET_perfmon read_collectors <br>2021-11-30_07:25:12.077-0500: [I] ET_perfmon write_collectors <br>2021-11-30_07:25:13.333-0500: [I] ClientThread-20 received command: 'thresholds refresh collectors 4021695'<br>2021-11-30_07:25:13.334-0500: [I] ClientThread-20 reload collectors <br>2021-11-30_07:25:13.335-0500: [I] ClientThread-20 read_collectors <br>2021-11-30_07:25:13.453-0500: [W] ClientThread-20 QueryHandler: query response has no data results <br>2021-11-30_07:25:13.454-0500: [W] ClientThread-20 QueryProcessor::execute: Error sending query in execute, quitting<br>2021-11-30_07:25:13.463-0500: [W] ClientThread-20 QueryHandler: query response has no data results <br>2021-11-30_07:25:13.463-0500: [W] ClientThread-20 QueryProcessor::execute: Error sending query in execute, quitting<br>2021-11-30_07:25:13.464-0500: [I] ClientThread-20 _activate_rules_scheduler completed <br>2021-11-30_07:25:15.528-0500: [W] ET_perfmon got rc (153) while executing ['/usr/lpp/mmfs/bin/mmccr', 'fput', 'collectors', '/var/mmfs/tmp/tmpKTN69I', '-c 4021694']<br>2021-11-30_07:25:15.528-0500: [E] ET_perfmon fput failed: Version mismatch on conditional put (err 805)<br> - CCRProxy._run_ccr_command:256<br>2021-09-29_20:03:53.322-0500: [I] MainThread --------------------------------- <br>2021-11-30_07:25:12.076-0500: [D] ET_perfmon File collectors has no newer version than 4021694 - CCRProxy.get<a href="File:119" target="_blank">File:119</a><br>2021-11-30_07:25:15.529-0500: [W] ET_perfmon Conditional put for file collectors with version 4021694 failed<br>2021-11-30_07:25:15.529-0500: [W] ET_perfmon New version received, start new collectors update cycle<br>2021-11-30_07:25:15.529-0500: [I] ET_perfmon read_collectors <br>2021-11-30_07:25:15.626-0500: [I] ET_perfmon write_collectors <br>2021-11-30_07:25:16.594-0500: [I] ClientThread-3 received command: 'thresholds refresh collectors 4021696'<br>2021-11-30_07:25:16.595-0500: [I] ClientThread-3 reload collectors <br>2021-11-30_07:25:16.595-0500: [I] ClientThread-3 read_collectors <br>2021-11-30_07:25:19.780-0500: [W] ET_perfmon got rc (153) while executing ['/usr/lpp/mmfs/bin/mmccr', 'fput', 'collectors', '/var/mmfs/tmp/tmp3joeUB', '-c 4021695']<br>2021-11-30_07:25:19.780-0500: [E] ET_perfmon fput failed: Version mismatch on conditional put (err 805)<br> - CCRProxy._run_ccr_command:256<br>2021-09-29_20:03:53.322-0500: [I] MainThread --------------------------------- <br>2021-11-30_07:25:15.625-0500: [D] ET_perfmon File collectors has no newer version than 4021695 - CCRProxy.get<a href="File:119" target="_blank">File:119</a><br>2021-11-30_07:25:16.781-0500: [D] ClientThread-3 File zmrules.json has no newer version than 1 - CCRProxy.get<a href="File:119" target="_blank">File:119</a><br>2021-11-30_07:25:19.780-0500: [W] ET_perfmon Conditional put for file collectors with version 4021695 failed<br>2021-11-30_07:25:19.781-0500: [W] ET_perfmon New version received, start new collectors update cycle<br>2021-11-30_07:25:19.781-0500: [I] ET_perfmon read_collectors <br>2021-11-30_07:25:19.881-0500: [I] ET_perfmon write_collectors <br>2021-11-30_07:25:21.238-0500: [I] ClientThread-7 received command: 'thresholds refresh collectors 4021697'<br>2021-11-30_07:25:21.239-0500: [I] ClientThread-7 reload collectors <br>2021-11-30_07:25:21.239-0500: [I] ClientThread-7 read_collectors <br>2021-11-30_07:25:21.324-0500: [W] NMES monitor event arrived while still busy for perfmon<br>2021-11-30_07:25:21.481-0500: [I] ET_threshold Event=thresh_monitor_del_active identifier=active_thresh_monitor arg0=active_thresh_monitor<br>2021-11-30_07:25:21.482-0500: [I] ET_threshold Monitor: LocalState:HEALTHY Events:1 Entities:1 RT: 0.16<br>2021-11-30_07:25:24.211-0500: [W] ET_perfmon got rc (153) while executing ['/usr/lpp/mmfs/bin/mmccr', 'fput', 'collectors', '/var/mmfs/tmp/tmp8HAusb', '-c 4021696']<br>2021-11-30_07:25:24.211-0500: [E] ET_perfmon fput failed: Version mismatch on conditional put (err 805)<br> - CCRProxy._run_ccr_command:256<br>2021-09-29_20:03:53.322-0500: [I] MainThread --------------------------------- <br>2021-11-30_07:25:19.881-0500: [D] ET_perfmon File collectors has no newer version than 4021696 - CCRProxy.get<a href="File:119" target="_blank">File:119</a><br>2021-11-30_07:25:21.411-0500: [D] ClientThread-7 File zmrules.json has no newer version than 1 - CCRProxy.get<a href="File:119" target="_blank">File:119</a><br>2021-11-30_07:25:24.211-0500: [W] ET_perfmon Conditional put for file collectors with version 4021696 failed<br>2021-11-30_07:25:24.212-0500: [W] ET_perfmon New version received, start new collectors update cycle<br>2021-11-30_07:25:24.212-0500: [I] ET_perfmon read_collectors <br>2021-11-30_07:25:24.314-0500: [I] ET_perfmon write_collectors <br>2021-11-30_07:25:24.543-0500: [I] ET_gui ServiceMonitor => out=Type=notify<br><br>And then gpfsgui apparently crashes and systemd automatically restarts it.<br><br><br>Steve Losen<br>Research Computing<br>University of Virginia<br>scl@virginia.edu 434-924-0640<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div><br><br><br>Ellei edellä ole toisin mainittu: / Unless stated otherwise above:<br><br>Oy IBM Finland Ab<br><br>PL 265, 00101 Helsinki, Finland<br><br>Business ID, Y-tunnus: 0195876-3<br><br>Registered in Finland<br><br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div><BR>
<BR>