[gpfsug-discuss] mmvdisk version/communication issues?
Luke Sudbery
l.r.sudbery at bham.ac.uk
Thu Apr 6 13:12:48 BST 2023
Just to close the loop on this, if anyone is having similar problems and wants to reference the cases:
IBM created efix 37 for 5.1.5.1 for this as g515p1e37.x86_64-linux-gnr.tar.gz under TS012531429.
Lenovo have a copy of this and have provided update instructions for the DSSG under case # 1001583845 (basically fail the recovery groups over and install the updated RPMs).
And this all appears to be working well for us now, so I can proceed with updating the other building blocks. A new version is due which will probably come out while I’m in the middle of doing that of course…
Many thanks,
Luke
--
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road
Please note I don’t work on Monday.
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Luke Sudbery
Sent: 21 March 2023 13:47
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] mmvdisk version/communication issues?
Yes, that’s exactly what happened to me. If Lenovo had tested an upgrade they would have seen it too.
We have 3 DSSG, but they are all this cluster, and we don’t run GNR outside this cluster.
The 5.1.5.1 nodes can talk to each other OK. And to the downlevel nodes. But the downlevel nodes can’t talk to the newer nodes.
At present it doesn’t seem to present any major problem – but we haven’t had to move RGs around or do anything like that yet.
And we have a new DSSG to add to this cluster soon, and then some pool migrations. This needs to sorted before we can start that in confidence.
Many thanks,
Luke
--
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road
Please note I don’t work on Monday.
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Ryan Novosielski
Sent: 21 March 2023 13:33
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] mmvdisk version/communication issues?
CAUTION: This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.
You know, I ran into this during the upgrade, didn’t have an explanation for it, and decided that I would see what happened when it was finished (problem was gone). I was going from DSS-G 3.2a to DSS-G 4.3a. It was at the stage where I had one machine upgraded and one machine not upgraded, which makes sense. I believe I then tried it from the upgraded node, it worked fine, and I figured “well, I’m blowing that one away shortly anyway.”
Would one expect to only see this while nodes in the same cluster are not running the same version? I guess what I’m asking is would one expect to have that kind of traffic coming from any place else except the same cluster?
Sent from my iPhone
On Mar 21, 2023, at 08:38, Luke Sudbery <L.R.Sudbery at bham.ac.uk<mailto:L.R.Sudbery at bham.ac.uk>> wrote:
Thank you, I’ll pass that onto Lenovo. I think I’ve also identified it as this in the fixies in 5.1.6.1 too:
https://www.ibm.com/support/pages/apar/IJ44607
IJ44607: SPECTRUM SCALE V5.1.3 AND GREATER FAILS TO INTEROPERATE WITH SPECTRUMSCALE VERSIONS PRIOR TO 5.1.3.
Problem summary
1. GNR RPCs fail when received by a GPFS daemon 5.1.3 or later
2. from a GPFS daemon older than version 5.1.3.
3. Kernel assert going off: privVfsP != NULL
Symptom:
1. Hang in the command
As such, AFAICT it will affect all Lenovo customers going from DSSG 3.x (GPFS 5.1.1.0) to 4.x (GPFS 5.1.5.1 efix20) and I’m a bit annoyed that didn’t pick it up before release.
Many thanks,
Luke
--
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road
Please note I don’t work on Monday.
From: Gang Qiu <gangqiu at cn.ibm.com<mailto:gangqiu at cn.ibm.com>>
Sent: 21 March 2023 12:31
To: Luke Sudbery (Advanced Research Computing) <l.r.sudbery at bham.ac.uk<mailto:l.r.sudbery at bham.ac.uk>>; gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: 答复: mmvdisk version/communication issues?
CAUTION: This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.
Luke,
1. The case number is TS011014198 (Internal defect number is 1163600)
5.1.3.1 efix47 includes this fix. (I didn't find the efix for 5.1.4 or 5.1.5)
2. The impact is that any GNR-related command from 5.1.2 to 5.1.5 will hang.
Regards,
Gang Qiu
====================================================
Gang Qiu(邱钢)
Spectrum Scale Development - ECE
IBM China Systems Lab
Mobile: +86-18612867902
====================================================
发件人: Luke Sudbery <l.r.sudbery at bham.ac.uk<mailto:l.r.sudbery at bham.ac.uk>>
日期: 星期二, 2023年3月21日 17:31
收件人: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>, Gang Qiu <gangqiu at cn.ibm.com<mailto:gangqiu at cn.ibm.com>>
主题: [EXTERNAL] RE: mmvdisk version/communication issues?
Thank you, that’s far more useful than Lenovo support have been so far! Unfortunately Lenovo only support particular versions of scale on their DSSG and they are our only source of the required GNR packages. So a couple more questions if you
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Thank you, that’s far more useful than Lenovo support have been so far!
Unfortunately Lenovo only support particular versions of scale on their DSSG and they are our only source of the required GNR packages.
So a couple more questions if you don’t mind!
1. Can you provide a link the APAR or similar I can share with Lenovo?
2. Do you know of any workaround or other impact of this issue? (I think I’ve seen mmheath show false errors because of it). This may help us to decide whether to just press ahead upgrading to 5.1.5 everywhere if the issue is not present 5.1.5 -> 5.1.5.
Many thanks,
Luke
--
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road
Please note I don’t work on Monday.
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Gang Qiu
Sent: 21 March 2023 03:49
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: [gpfsug-discuss] 答复: mmvdisk version/communication issues?
CAUTION: This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.
It is not mmvdisk issue. Mmvdisk hangs because ‘tslsrecgroup’ command hangs which executed by mmvdisk.
I recreated it on my environment (coexist 5.1.3 or 5.1.4, 5.1.5 with 5.1.2 or previous version)
After further investigation, it should be a known issue which is fixed in 5.1.6 PTF1.
To resolve this issue, we need to upgrade the version to 5.1.6.1 at least.
Regards,
Gang Qiu
====================================================
Gang Qiu(邱钢)
Spectrum Scale Development - ECE
IBM China Systems Lab
Mobile: +86-18612867902
====================================================
发件人: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> 代表 Luke Sudbery <l.r.sudbery at bham.ac.uk<mailto:l.r.sudbery at bham.ac.uk>>
日期: 星期六, 2023年3月18日 00:06
收件人: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
主题: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk version/communication issues?
On further investigation the command does eventually complete, after 11 minutes rather than a couple of seconds. [root@ rds-pg-dssg01 ~]# time mmvdisk pdisk list --rg rds_er_dssg02 --not-ok mmvdisk: All pdisks of recovery group 'rds_er_dssg02'
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
On further investigation the command does eventually complete, after 11 minutes rather than a couple of seconds.
[root at rds-pg-dssg01 ~]# time mmvdisk pdisk list --rg rds_er_dssg02 --not-ok
mmvdisk: All pdisks of recovery group 'rds_er_dssg02' are ok.
real 11m14.106s
user 0m1.430s
sys 0m0.555s
[root at rds-pg-dssg01 ~]#
Looking at the process tree, the bits that hang are:
[root at rds-pg-dssg01 ~]# time tslsrecgroup rds_er_dssg02 -Y --v2 --failure-domain
Failed to connect to file system daemon: Connection timed out
real 5m30.181s
user 0m0.001s
sys 0m0.003s
and then
[root at rds-pg-dssg01 ~]# time tslspdisk --recovery-group rds_er_dssg02 --notOK
Failed to connect to file system daemon: Connection timed out
real 5m30.247s
user 0m0.003s
sys 0m0.002s
[root at rds-pg-dssg01 ~]#
Which adds up to the 11 minutes.... then it does something else and just works. Or maybe it doesn't work and just wouldn't report any failed disks is there were any….
While hanging, the ts commands appear to be LISTENing, not attempting to make connections:
[root at rds-pg-dssg01 ~]# pidof tslspdisk
2156809
[root at rds-pg-dssg01 ~]# netstat -apt | grep 2156809
tcp 0 0 0.0.0.0:60000 0.0.0.0:* LISTEN 2156809/tslspdisk
[root at rds-pg-dssg01 ~]#
Port 60000 is the lowest of our tscCmdPortRange.
Don’t know if that helps anyone….
Cheers,
Luke
--
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road
Please note I don’t work on Monday.
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Luke Sudbery
Sent: 17 March 2023 15:11
To: gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] mmvdisk version/communication issues?
Hello,
We 3 Lenovo DSSG “Building Blocks” as they call them – 2x GNR server pairs.
We’ve just upgraded the 1st of them from 3.2a (GPFS 5.1.1.0) to 4.3a (5.1.5.1 efix 20).
Now the older systems can’t communicated with the newer in certain circumstances, specifically querying recovery groups hosted on other servers.
It works old->old, new->old and new->new but not old->new.
If fairly sure it is not a TCP comms problem. I can ssh between the node as root and as the GPFS sudoUser. Port 1191 and the tscCmdPortRange are open and accessible in both direction between the nodes. There are connections present between the nodes in netstat and in mmfsd.latest.log. No pending message (to that node) in mmdiag --network.
In these examples rds-er-dssg01/2 are upgraded, rds-pg-dssg01/2 are downlevel:
[root at rds-er-dssg01 ~]# mmvdisk pdisk list --rg rds_er_dssg02 --not-ok # New to new
mmvdisk: All pdisks of recovery group 'rds_er_dssg02' are ok.
[root at rds-er-dssg01 ~]# mmvdisk pdisk list --rg rds_pg_dssg02 --not-ok # New to old
mmvdisk: All pdisks of recovery group 'rds_pg_dssg02' are ok.
[root at rds-er-dssg01 ~]#
[root at rds-pg-dssg01 ~]# mmvdisk pdisk list --rg rds_pg_dssg02 --not-ok # Old to old
mmvdisk: All pdisks of recovery group 'rds_pg_dssg02' are ok.
[root at rds-pg-dssg01 ~]# mmvdisk pdisk list --rg rds_er_dssg02 --not-ok # Old to new [HANGS]
^Cmmvdisk: Command failed. Examine previous error messages to determine cause.
[root at rds-pg-dssg01 ~]#
Has anyone come across this? mmvdisk should work across slightly different versions of 5.1, right? No recovery group, cluster or filesystem versions have been changed yet.
We will also log a ticket with snaps and more info but wondered if anyone had seen this.
And while this particular command is not a major issue, we don’t know what else it may affect, before we proceed with the reset of the cluster.
Many thanks,
Luke
--
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road
Please note I don’t work on Monday.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230406/0e57497c/attachment-0001.htm>
More information about the gpfsug-discuss
mailing list