[gpfsug-discuss] gui_refresh_task_failed for HW_INVENTORY with two active GUI nodes

Thu Jan 30 15:31:32 GMT 2020

Hello,

The GUI tasks which are not daily tasks will start periodically at a random
time.

The exception are daily tasks which are defined at fixed start times.
It seems this is the issue you are experiencing, as the HW_INVENTORY task
only runs once a day adn starts at identical times on both GUI nodes.
Tweaking the cache database  is unfortunately not a workaround as the hard
coded and fixed starting times will be reset for every GUI restart.

I have created a task to address this issue in a future release.
We could for example add a random delay to the daily tasks, or a fixed
delay based on the number of GUI nodes that are active.

Mit freundlichen Grüßen / Kind regards

Dr. Markus Rohwedder

Spectrum Scale GUI Development

 Phone:  +49 162 4159920       IBM Deutschland Research &                          
                              Development                                          

 E-Mail: rohwedder at de.ibm.com  Am Weiher 24                                        

                               65451 Kelsterbach                                   

                               Germany                                             

From:	"Billich  Heinrich Rainer (ID SD)"
            <heinrich.billich at id.ethz.ch>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:	29.01.2020 14:41
Subject:	[EXTERNAL] [gpfsug-discuss] gui_refresh_task_failed for
            HW_INVENTORY with two	active GUI nodes
Sent by:	gpfsug-discuss-bounces at spectrumscale.org

Hello,

Can I change the times at which the GUI runs HW_INVENTORY and related
tasks?

we frequently get  messages like

   gui_refresh_task_failed     GUI           WARNING     12 hours ago
The following GUI refresh task(s) failed: HW_INVENTORY

The tasks fail due to timeouts. Running the task manually most times
succeeds. We do run two gui nodes per cluster and I noted that both servers
seem run the HW_INVENTORY at the exact same time which may lead to locking
or congestion issues, actually the logs show messages like

EFSSA0194I Waiting for concurrent operation to complete.

The gui calls ‘rinv’ on the xCat servers. Rinv for a single   little-endian
server takes a long time – about 2-3 minutes , while it finishes in  about
15s for big-endian server.

Hence the long runtime of rinv on little-endian systems may be an issue,
too

We run 5.0.4-1 efix9 on the gui and ESS  5.3.4.1 on the GNR systems
(5.0.3.2 efix4). We run a mix of ppc64 and ppc64le systems, which a
separate xCat/ems server for each type. The GUI nodes are ppc64le.

We did see this issue with several gpfs version on the gui and with at
least two ESS/xCat versions.

Just to be sure I did purge the Posgresql tables.

I did try

/usr/lpp/mmfs/gui/cli/lstasklog HW_INVENTORY
/usr/lpp/mmfs/gui/cli/runtask HW_INVENTORY –debug

And also tried to read the logs in /var/log/cnlog/mgtsrv/ - but they are
difficult.

Thank you,

Heiner

--
=======================
Heinrich Billich
ETH Zürich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=3j7GTkRFLANP-V9nMPiOuUX-2D3ybbNTEc64kU-OQAM&s=sR1v63lEVWuEZTBgspG3imB0MN_-7ggA6zrmyvqfCzE&e=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200130/871b3083/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200130/871b3083/attachment-0006.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 14272346.gif
Type: image/gif
Size: 4659 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200130/871b3083/attachment-0007.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200130/871b3083/attachment-0008.gif>