<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>
<blockquote type="cite">
<div>On Sep 13, 2024, at 08:15, Dietrich, Stefan <stefan.dietrich@desy.de> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div>Hello Peter,<br>
<br>
<blockquote type="cite">Since we upgraded to 5.1.9-5 we're getting random nodes moaning about lost<br>
connections when ever a another machine is rebooted or stops working. This is<br>
great, however there does not seam to be any great way to acknowledge the<br>
alerts, or close the connections gracefully if the machine is turned off rather<br>
than actually failing.<br>
</blockquote>
<br>
it's possible to resolve event in mmhealth:<br>
<br>
# mmhealth event resolve<br>
Missing arguments.<br>
Usage:<br>
mmhealth event resolve {EventName} [Identifier]<br>
<br>
-> `mmhealth event resolve cluster_connections_down AFFECTED_IP` should do the trick.<br>
<br>
In our clusters, a regular reboot doesn't seem to trigger this event. All our nodes are running Scale >= 5.2.0<br>
</div>
</div>
</blockquote>
</div>
<br>
<div>Our clusters (5.1.9-3 on the client side, and either 5.1.5-1 or 5.1.9-2 on the storage side) also show downed connections, but I wish this were somehow tunable. A single downed client that’s not even part of the same cluster is not a reason to alert us
on our storage cluster. We monitor MMHEALTH via Nagios, and so we’re occasionally getting messages about a single client.</div>
<div><br>
</div>
<div>
<div>
<div dir="auto" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
--<br>
#BlackLivesMatter</div>
<div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State<span class="Apple-tab-span" style="white-space: pre;"> </span> | Ryan Novosielski - novosirj@rutgers.edu<br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ<span class="Apple-tab-span" style="white-space: pre;"> </span> | Office of Advanced Research Computing - MSB A555B, Newark<br>
`'</div>
</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite"></blockquote>
</div>
</div>
</body>
</html>