<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt">Another data point for Keith/Kristy,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">I’ve been using Zimon for about 18 months now, and I’ll have to admit it’s been less than robust for long-term data. The biggest issue I’ve run into is the stability of the collector process. I have it crash
on a fairly regular basis, most due to memory usage. This results in data loss You can configure it in a highly-available mode that should mitigate this to some degree. However, I don’t think IBM has published any details on how reliable the data collection
process is.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Bob Oesterlin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Sr Principal Storage Engineer, Nuance<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Kristy Kallback-Rose <kkr@lbl.gov><br>
<b>Reply-To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Date: </b>Sunday, September 24, 2017 at 2:29 PM<br>
<b>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Experience with zimon database stability, and best practices for backup?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hi Keith, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> We have barely begun with Zimon and have not (knock, knock) run up against any loss or corruption issues with Zimon. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> However, getting data out of Zimon for various reasons is something I have been thinking about. I'm interested partly because of the granularity that is lost over time like with any round robin style data collection scheme. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So I guess one question is whether you have considered pulling the data out to another database, looked at the SS GUI which uses a postgres db (iirc, about to take off on a flight and can't check), or looked at the Grafana bridge which
would get data into OpenTsdb format, again iirc. Anyway, just some things for consideration and a request to share back whatever you find out if it's off list.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks, getting stink eye to go to airplane mode.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">More later.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Cheers<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Kristy<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Sep 24, 2017 11:05 AM, "Keith Ball" <<a href="mailto:bipcuds@gmail.com">bipcuds@gmail.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello All,<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">In a recent Spectrum Scale performance study, we used zimon/mmperfmon to gather metrics. During a period of 2 months, we ended up losing data twice from the zimon database; once after the virtual disk serving
both the OS files and zimon collector and DB storage was resized, and a second time after an unknown event (the loss was discovered when plotting in Grafana only went back to a certain data and time; likewise, mmperfmon query output only went back to the same
time).<o:p></o:p></p>
</div>
<p class="MsoNormal">Details:<o:p></o:p></p>
</div>
<p class="MsoNormal">- Spectrum Scale 4.2.1.1 (on NSD servers); 4.2.1.2 on the zimon collector node and other clients<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- Data retention in the "raw" stratum was set to 2 months; the "domains" settings were as follows (note that we did not hit the ceiling of 60GB (1GB/file * 60 files):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">domains = {<br>
# this is the raw domain<br>
aggregation = 0 # aggregation factor for the raw domain is always 0.<br>
ram = "12g" # amount of RAM to be used<br>
duration = "2m" # amount of time that data with the highest precision is kept.<br>
filesize = "1g" # maximum file size<br>
files = 60 # number of files.<br>
},<br>
{<br>
# this is the first aggregation domain that aggregates to 10 seconds<br>
aggregation = 10<br>
ram = "800m" # amount of RAM to be used<br>
duration = "6m" # keep aggregates for 1 week.<br>
filesize = "1g" # maximum file size<br>
files = 10 # number of files.<br>
},<br>
{<br>
# this is the second aggregation domain that aggregates to 30*10 seconds == 5 minutes<br>
aggregation = 30<br>
ram = "800m" # amount of RAM to be used<br>
duration = "1y" # keep averages for 2 months.<br>
filesize = "1g" # maximum file size<br>
files = 5 # number of files.<br>
},<br>
{<br>
# this is the third aggregation domain that aggregates to 24*30*10 seconds == 2 hours<br>
aggregation = 24<br>
ram = "800m" # amount of RAM to be used<br>
duration = "2y" #<br>
filesize = "1g" # maximum file size<br>
files = 5 # number of files.<br>
}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Questions:<o:p></o:p></p>
</div>
<p class="MsoNormal">1.) Has anyone had similar issues with losing data from zimon?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">2.) Are there known circumstances where data could be lost, e.g. changing the aggregation domain definitions, or even simply restarting the zimon collector?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">3.) Does anyone have any "best practices" for backing up the zimon database? We were taking weekly "snapshots" by shutting down the collector, and making a tarball copy of the /opt/ibm/zimon directory (but the database corruption/data loss
still crept through for various reasons).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">In terms of debugging, we do not have Scale or zimon logs going back to the suspected dates of data loss; we do have a gpfs.snap from about a month after the last data loss - would it have any useful clues? Opening a PMR could be tricky,
as it was the customer who has the support entitlement, and the environment (specifically the old cluster definitino and the zimon collector VM) was torn down.<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Many Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Keith<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#888888">-- <br>
Keith D. Ball, PhD<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888">RedLine Performance Solutions, LLC<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">web: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.redlineperf.com_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Qda4XyOAjxfIGGSuRrYemKl8f0MXB4mp6nhdbmkjh20&s=dUvbBoiPFANvyGsOER5MAnt9-mwK69adFuLFatx2Rmw&e=" target="_blank">http://www.redlineperf.com/</a><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:#888888">email: <a href="mailto:aqualkenbush@redlineperf.com" target="_blank">
kball@redlineperf.com</a><o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="color:#888888">cell: <a href="tel:%28540%29%20557-7851" target="_blank">
540-557-7851</a></span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Qda4XyOAjxfIGGSuRrYemKl8f0MXB4mp6nhdbmkjh20&s=d6CkXN5mbyGvJQOduzX-LhJMANQgfvAV-nw_6ZgG-D4&e=" target="_blank">
spectrumscale.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=Qda4XyOAjxfIGGSuRrYemKl8f0MXB4mp6nhdbmkjh20&s=LkO3HEtokkzigjYqB4dIOUWLPhtikMbwcsXEakFp8DU&e=" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>