<div dir="ltr">Hi Michael, Norbert, <div><br></div><div> Thanks for your replies, we did do all the setup as Michael described, and stop and restart services more than once ;-). I believe the issue is resolved with the PTF. I am still checking, but it seems to be working with symmetric peering between those two nodes. I will test further and expand to other nodes and make sure it continue to work. I will report back if I run into any other issues.</div><div><br></div><div>Cheers,</div><div>Kristy</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 25, 2017 at 6:46 AM, Michael L Taylor <span dir="ltr"><<a href="mailto:taylorm@us.ibm.com" target="_blank">taylorm@us.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><p><font size="2">Hi Kristy,</font><br><font size="2">At first glance your config looks ok. Here are a few things to check.</font><br><br><font size="2">Is 4.2.3 the first time you have installed and configured performance monitoring? Or have you configured it at some version < 4.2.3 and then upgraded to 4.2.3?</font><br><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adv_guienableperfmon.htm" target="_blank"></a><br><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adv_guienableperfmon.htm" target="_blank"><font size="2">Did you restart pmcollector after changing the configuration?</font></a><br><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adv_guienableperfmon.htm" target="_blank"><font size="2">https://www.ibm.com/support/<wbr>knowledgecenter/STXKQY_4.2.3/<wbr>com.ibm.spectrum.scale.v4r23.<wbr>doc/bl1adv_guienableperfmon.<wbr>htm</font></a><br><font size="2">"Configure peer configuration for the collectors. The collector configuration is stored in the /opt/IBM/zimon/ZIMonCollector.<wbr>cfg file. This file defines collector peer configuration and the aggregation rules. If you are using only a single collector, you can skip this step. Restart the pmcollector service after making changes to the configuration file. The GUI must have access to all data from each GUI node. "</font><br><br><font size="2">Firewall ports are open for performance monitoring and MGMT GUI?</font><br><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adv_firewallforgui.htm?cp=STXKQY" target="_blank"><font size="2">https://www.ibm.com/support/<wbr>knowledgecenter/STXKQY_4.2.3/<wbr>com.ibm.spectrum.scale.v4r23.<wbr>doc/bl1adv_firewallforgui.htm?<wbr>cp=STXKQY</font></a><br><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adv_firewallforPMT.htm" target="_blank"><font size="2">https://www.ibm.com/support/<wbr>knowledgecenter/STXKQY_4.2.3/<wbr>com.ibm.spectrum.scale.v4r23.<wbr>doc/bl1adv_firewallforPMT.htm</font></a><br><br><font size="2">Did you setup the collectors with :</font><br><font size="2">prompt# mmperfmon config generate --collectors <a href="http://collector1.domain.com" target="_blank">collector1.domain.com</a>,<a href="http://collector2.domain.com" target="_blank">collecto<wbr>r2.domain.com</a>,… </font><br><br><font size="2">Once the configuration file has been stored within IBM Spectrum Scale, it can be activated as follows.</font><br><font size="2">prompt# mmchnode --perfmon –N nodeclass1,nodeclass2,… <br></font><br><font size="2">Perhaps once you make sure the federated mode is set between hostA and hostB as you like then 'systemctl restart pmcollector' and then 'systemctl restart gpfsgui' on both nodes?<br><br><br></font><br><img width="16" height="16" border="0" alt="Inactive hide details for gpfsug-discuss-request---05/24/2017 12:58:21 PM---Send gpfsug-discuss mailing list submissions to gp"><font size="2" color="#424282">gpfsug-discuss-request---05/<wbr>24/2017 12:58:21 PM---Send gpfsug-discuss mailing list submissions to <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a></font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2"><a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank">gpfsug-discuss-request@<wbr>spectrumscale.org</a></font><br><font size="2" color="#5F5F5F">To: </font><font size="2"><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">05/24/2017 12:58 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">gpfsug-discuss Digest, Vol 64, Issue 61</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a></font><br></p><hr width="100%" size="2" align="left" noshade style="color:#8091a5"><br><br><br><tt><font size="2">Send gpfsug-discuss mailing list submissions to<br> <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a><br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> </font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br>or, via email, send a message with subject or body 'help' to<br> <a href="mailto:gpfsug-discuss-request@spectrumscale.org" target="_blank">gpfsug-discuss-request@<wbr>spectrumscale.org</a><br><br>You can reach the person managing the list at<br> <a href="mailto:gpfsug-discuss-owner@spectrumscale.org" target="_blank">gpfsug-discuss-owner@<wbr>spectrumscale.org</a><br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of gpfsug-discuss digest..."<br><br><br>Today's Topics:<br><br> 1. SS Metrics (Zimon) and SS GUI, Federation not working<br> (Kristy Kallback-Rose)<br><br><br>------------------------------<wbr>------------------------------<wbr>----------<br><br>Message: 1<br>Date: Wed, 24 May 2017 12:57:49 -0700<br>From: Kristy Kallback-Rose <<a href="mailto:kkr@lbl.gov" target="_blank">kkr@lbl.gov</a>><br>To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a><br>Subject: [gpfsug-discuss] SS Metrics (Zimon) and SS GUI, Federation<br> not working<br>Message-ID:<br> <<a href="mailto:CAA9oNus2BRyJcQEHXa7j1Vmz_Z6swTwRDatMw93P0_sD8X76vg@mail.gmail.com" target="_blank">CAA9oNus2BRyJcQEHXa7j1Vmz_<wbr>Z6swTwRDatMw93P0_sD8X76vg@<wbr>mail.gmail.com</a>><br>Content-Type: text/plain; charset="utf-8"<div><div class="h5"><br><br>Hello,<br><br> We have been experimenting with Zimon and the SS GUI on our dev cluster<br>under 4.2.3. Things work well with one collector, but I'm running into<br>issues when trying to use symmetric collector peers, i.e. federation.<br><br> hostA and hostB are setup as both collectors and sensors with each a<br>collector peer for the other. When this is done I can use mmperfmon to<br>query hostA from hostA or hostB and vice versa. However, with this<br>federation setup, the GUI fails to show data. The GUI is running on hostB.<br>>From the collector candidate pool, hostA has been selected (automatically,<br>not manually) as can be seen in the sensor configuration file. The GUI is<br>unable to load data (just shows "Loading" on the graph), *unless* I change<br>the setting of the ZIMonAddress variable in<br>/usr/lpp/mmfs/gui/conf/<wbr>gpfsgui.properties<br>from localhost to hostA explicitly, it does not work if I change it to<br>hostB explicitly. The GUI also works fine if I remove the peer entries<br>altogether and just have one collector.<br><br> I thought that federation meant that no matter which collector was<br>queried the data would be returned. This appears to work for mmperfmon, but<br>not the GUI. Can anyone advise? I also don't like the idea of having a pool<br>of collector candidates and hard-coding one into the GUI configuration. I<br>am including some output below to show the configs and query results.<br><br>Thanks,<br><br>Kristy<br><br><br> The peers are added into the ZIMonCollector.cfg using the default port<br>9085:<br><br> peers = {<br><br> host = "hostA"<br><br> port = "9085"<br><br> },<br><br> {<br><br> host = "hostB"<br><br> port = "9085"<br><br> }<br><br><br>And the nodes are added as collector candidates, on hostA and hostB you<br>see, looking at the config file directly, in /opt/IBM/zimon/ZIMonSensors.<br>cfg:<br><br></div></div>colCandidates = "<a href="http://hostA.nersc.gov" target="_blank">hostA.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hosta.nersc.gov/" target="_blank">http://hosta.nersc.gov/</a></font></tt><tt><font size="2">>", "<br><a href="http://hostB.nersc.gov" target="_blank">hostB.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hostb.nersc.gov/" target="_blank">http://hostb.nersc.gov/</a></font></tt><tt><font size="2">>"<span class=""><br><br>colRedundancy = 1<br><br>collectors = {<br><br></span>host = "<a href="http://hostA.nersc.gov" target="_blank">hostA.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hosta.nersc.gov/" target="_blank">http://hosta.nersc.gov/</a></font></tt><tt><font size="2">>"<span class=""><br><br>port = "4739"<br><br>}<br><br><br>Showing the config with mmperfmon config show:<br><br></span>colCandidates = "<a href="http://hostA.nersc.gov" target="_blank">hostA.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hosta.nersc.gov/" target="_blank">http://hosta.nersc.gov/</a></font></tt><tt><font size="2">>", "<br><a href="http://hostB.nersc.gov" target="_blank">hostB.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hostb.nersc.gov/" target="_blank">http://hostb.nersc.gov/</a></font></tt><tt><font size="2">>"<span class=""><br><br>colRedundancy = 1<br><br>collectors = {<br><br>host = ""<br><br><br>Using mmperfmon I can query either host.<br><br><br>[root@hostA ~]# mmperfmon query cpu -N hostB<br><br><br>Legend:<br><br></span> 1: <a href="http://hostB.nersc.gov" target="_blank">hostB.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hostb.nersc.gov/" target="_blank">http://hostb.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_system<br><br> 2: <a href="http://hostB.nersc.gov" target="_blank">hostB.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hostb.nersc.gov/" target="_blank">http://hostb.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_user<br><br> 3: <a href="http://hostB.nersc.gov" target="_blank">hostB.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hostb.nersc.gov/" target="_blank">http://hostb.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_contexts<span class=""><br><br><br><br>Row Timestamp cpu_system cpu_user cpu_contexts<br><br> 1 2017-05-23-17:03:54 0.54 3.67 4961<br><br> 2 2017-05-23-17:03:55 0.63 3.55 6199<br><br> 3 2017-05-23-17:03:56 1.59 3.76 7914<br><br> 4 2017-05-23-17:03:57 1.38 5.34 5393<br><br> 5 2017-05-23-17:03:58 0.54 2.21 2435<br><br> 6 2017-05-23-17:03:59 0.13 0.29 2519<br><br> 7 2017-05-23-17:04:00 0.13 0.25 2197<br><br> 8 2017-05-23-17:04:01 0.13 0.29 2473<br><br> 9 2017-05-23-17:04:02 0.08 0.21 2336<br><br> 10 2017-05-23-17:04:03 0.13 0.21 2312<br><br><br>[root@ hostB ~]# mmperfmon query cpu -N hostB<br><br><br>Legend:<br><br></span> 1: <a href="http://hostB.nersc.gov" target="_blank">hostB.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hostb.nersc.gov/" target="_blank">http://hostb.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_system<br><br> 2: <a href="http://hostB.nersc.gov" target="_blank">hostB.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hostb.nersc.gov/" target="_blank">http://hostb.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_user<br><br> 3: <a href="http://hostB.nersc.gov" target="_blank">hostB.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hostb.nersc.gov/" target="_blank">http://hostb.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_contexts<span class=""><br><br><br><br>Row Timestamp cpu_system cpu_user cpu_contexts<br><br> 1 2017-05-23-17:04:07 0.13 0.21 2010<br><br> 2 2017-05-23-17:04:08 0.04 0.21 2571<br><br> 3 2017-05-23-17:04:09 0.08 0.25 2766<br><br> 4 2017-05-23-17:04:10 0.13 0.29 3147<br><br> 5 2017-05-23-17:04:11 0.83 0.83 2596<br><br> 6 2017-05-23-17:04:12 0.33 0.54 2530<br><br> 7 2017-05-23-17:04:13 0.08 0.33 2428<br><br> 8 2017-05-23-17:04:14 0.13 0.25 2326<br><br> 9 2017-05-23-17:04:15 0.13 0.29 4190<br><br> 10 2017-05-23-17:04:16 0.58 1.92 5882<br><br><br>[root@ hostB ~]# mmperfmon query cpu -N hostA<br><br><br>Legend:<br><br></span> 1: <a href="http://hostA.nersc.gov" target="_blank">hostA.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hosta.nersc.gov/" target="_blank">http://hosta.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_system<br><br> 2: <a href="http://hostA.nersc.gov" target="_blank">hostA.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hosta.nersc.gov/" target="_blank">http://hosta.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_user<br><br> 3: <a href="http://hostA.nersc.gov" target="_blank">hostA.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hosta.nersc.gov/" target="_blank">http://hosta.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_contexts<span class=""><br><br><br><br>Row Timestamp cpu_system cpu_user cpu_contexts<br><br> 1 2017-05-23-17:05:45 0.33 0.46 7460<br><br> 2 2017-05-23-17:05:46 0.33 0.42 8993<br><br> 3 2017-05-23-17:05:47 0.42 0.54 8709<br><br> 4 2017-05-23-17:05:48 0.38 0.5 5923<br><br> 5 2017-05-23-17:05:49 0.54 1.46 7381<br><br> 6 2017-05-23-17:05:50 0.58 3.51 10381<br><br> 7 2017-05-23-17:05:51 1.05 1.13 10995<br><br> 8 2017-05-23-17:05:52 0.88 0.92 10855<br><br> 9 2017-05-23-17:05:53 0.5 0.63 10958<br><br> 10 2017-05-23-17:05:54 0.5 0.59 10285<br><br><br>[root@ hostA ~]# mmperfmon query cpu -N hostA<br><br><br>Legend:<br><br></span> 1: <a href="http://hostA.nersc.gov" target="_blank">hostA.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hosta.nersc.gov/" target="_blank">http://hosta.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_system<br><br> 2: <a href="http://hostA.nersc.gov" target="_blank">hostA.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hosta.nersc.gov/" target="_blank">http://hosta.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_user<br><br> 3: <a href="http://hostA.nersc.gov" target="_blank">hostA.nersc.gov</a> <</font></tt><tt><font size="2"><a href="http://hosta.nersc.gov/" target="_blank">http://hosta.nersc.gov/</a></font></tt><tt><font size="2">>|CPU|<wbr>cpu_contexts<span class=""><br><br><br><br>Row Timestamp cpu_system cpu_user cpu_contexts<br><br> 1 2017-05-23-17:05:50 0.58 3.51 10381<br><br> 2 2017-05-23-17:05:51 1.05 1.13 10995<br><br> 3 2017-05-23-17:05:52 0.88 0.92 10855<br><br> 4 2017-05-23-17:05:53 0.5 0.63 10958<br><br> 5 2017-05-23-17:05:54 0.5 0.59 10285<br><br> 6 2017-05-23-17:05:55 0.46 0.63 11621<br><br> 7 2017-05-23-17:05:56 0.84 0.92 11477<br><br> 8 2017-05-23-17:05:57 1.47 1.88 11084<br><br> 9 2017-05-23-17:05:58 0.46 1.76 9125<br><br> 10 2017-05-23-17:05:59 0.42 0.63 11745<br></span>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><tt><font size="2"><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20170524/e64509b9/attachment.html" target="_blank">http://gpfsug.org/pipermail/<wbr>gpfsug-discuss/attachments/<wbr>20170524/e64509b9/attachment.<wbr>html</a></font></tt><tt><font size="2">><br><br>------------------------------<br><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br><br><br>End of gpfsug-discuss Digest, Vol 64, Issue 61<br>******************************<wbr>****************<br><br></font></tt><br><br>
<p></p></div>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>