[gpfsug-discuss] Fw: Flash (Alert) IBM Spectrum Scale V4.2.1/4.2.2 parallel log recovery function may result in undetected data corruption

Bryan Banister bbanister at jumptrading.com
Fri Feb 24 16:21:07 GMT 2017


Here is the latest I got from IBM:

The fix only needs to be installed on the file system manager nodes.

About how to know if your cluster is affected already, you can check if there was any MMFS_FSSTRUCT error in the system logs. If you encounter any lookup failure, funny ls cmd outputs. Or if any cmd would give some replica mismatch error or warning. If you encountered the following kind of Assertion failure you hit the bug.

Thu Jul 21 03:26:32.373 2016: [X] *** Assert exp(prevIndEntryP->nextP->dataBlockNum > dataBlockNum) in line 4552 of file /project/sprelbmd/build/rbmd1629a/src/avs/fs/mmfs/ts/log/repUpdate.C
Thu Jul 21 03:26:32.374 2016: [E] *** Traceback:
Thu Jul 21 03:26:32.375 2016: [E] 2:0x7FE6E141AB36 logAssertFailed + 0x2D6 at Logger.C:546
Thu Jul 21 03:26:32.376 2016: [E] 3:0x7FE6E13FCD25 InodeRecoveryList::addInodeAndIndBlock(long long, unsigned int, RepDiskAddr const&, InodeRecoveryList::FlagsToSet, long long, RepDiskAddr const&) + 0x355 at repUpdate.C:4552
Thu Jul 21 03:26:32.377 2016: [E] 4:0x7FE6E1066879 RecoverDirEntry(StripeGroup*, LogRecovery*, LogFile*, LogRecordType, long long, int, unsigned int*, char*, int*, RepDiskAddr) + 0x1089 at direct.C:2312
Thu Jul 21 03:26:32.378 2016: [E] 5:0x7FE6E13F8741 LogRecovery::recoverOneObject(long long) + 0x1E1 at recoverlog.C:362
Thu Jul 21 03:26:32.379 2016: [E] 6:0x7FE6E0F29B25 MultiThreadWork::doNextStep() + 0xC5 at workthread.C:533
Thu Jul 21 03:26:32.380 2016: [E] 7:0x7FE6E0F29FBB MultiThreadWork::helperThreadBody(void*) + 0xCB at workthread.C:455
Thu Jul 21 03:26:32.381 2016: [E] 8:0x7FE6E0F5FB26 Thread::callBody(Thread*) + 0x46 at thread.C:393
Thu Jul 21 03:26:32.382 2016: [E] 9:0x7FE6E0F4DD12 Thread::callBodyWrapper(Thread*) + 0xA2 at mastdep.C:1077
Thu Jul 21 03:26:32.383 2016: [E] 10:0x7FE6E0667851 start_thread + 0xD1 at mastdep.C:1077
Thu Jul 21 03:26:32.384 2016: [E] 11:0x7FE6DF7BE90D clone + 0x6D at mastdep.C:1077

Hope that helps,
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Fosburgh,Jonathan
Sent: Friday, February 24, 2017 9:30 AM
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Fw: Flash (Alert) IBM Spectrum Scale V4.2.1/4.2.2 parallel log recovery function may result in undetected data corruption

FWIW, my contact said to do everything, even client only clusters.

--
Jonathan Fosburgh
Principal Application Systems Analyst
Storage Team
IT Operations
jfosburg at mdanderson.org<mailto:jfosburg at mdanderson.org>
(713) 745-9346

-----Original Message-----

Date: Fri, 24 Feb 2017 15:25:14 +0000
Subject: Re: [gpfsug-discuss] Fw: Flash (Alert) IBM Spectrum Scale V4.2.1/4.2.2 parallel log recovery function may result in undetected data corruption
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug%20main%20discussion%20list%20%3cgpfsug-discuss at spectrumscale.org%3e>>
Reply-to: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
From: Bryan Banister <bbanister at jumptrading.com<mailto:Bryan%20Banister%20%3cbbanister at jumptrading.com%3e>>
I just got word that you only need to update the active file system manager node… I’ll let you know if I hear differently,
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sanchez, Paul
Sent: Friday, February 24, 2017 9:16 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] Fw: Flash (Alert) IBM Spectrum Scale V4.2.1/4.2.2 parallel log recovery function may result in undetected data corruption

Can anyone from IBM confirm whether this only affects manager nodes or if parallel log recovery is expected to happen on any other nodes?

Thx
Paul

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Bryan Banister
Sent: Friday, February 24, 2017 9:08 AM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Fw: Flash (Alert) IBM Spectrum Scale V4.2.1/4.2.2 parallel log recovery function may result in undetected data corruption

Has anyone been hit by this data corruption issue and if so how did you determine the file system had corruption?

Thanks!
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Oesterlin, Robert
Sent: Thursday, February 23, 2017 9:46 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: [gpfsug-discuss] Fw: Flash (Alert) IBM Spectrum Scale V4.2.1/4.2.2 parallel log recovery function may result in undetected data corruption

For those not subscribed, see below.

Bob Oesterlin
Sr Principal Storage Engineer, Nuance


From: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Reply-To: "dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>" <dW-notify at us.ibm.com<mailto:dW-notify at us.ibm.com>>
Date: Thursday, February 23, 2017 at 9:42 AM
Subject: [EXTERNAL] [Forums] 'gpfs at us.ibm.com' replied to the 'IBM Spectrum Scale V4.2.2 announcements' topic thread in the 'General Parallel File System - Announce (GPFS - Announce)' forum.

[cid:image001.png at 01D28E87.B52EFB90] gpfs at us.ibm.com<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ibm.com_developerworks_community_profiles_html_profileView.do-3Fuserid-3D060000T9GF&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=VnicQ7hM5kAbUhEZZxbyL6g9pAlwAYXG0f12gJWrEew&s=TdzECmVhE3t4jdrNWogfBMSo8vgP_met3YTFRRdJARc&e=> replied to the IBM Spectrum Scale V4.2.2 announcements<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_topic-3Fid-3Da1939921-2D633b-2D4a45-2D8f0f-2D8f9181fc2bc5&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=VnicQ7hM5kAbUhEZZxbyL6g9pAlwAYXG0f12gJWrEew&s=DPv9xlNyJnRXkqQPzn7f9lieOuJi5GHz3HRuN7MhFXA&e=> topic thread in the General Parallel File System - Announce (GPFS - Announce)<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ibm.com_developerworks_community_forums_html_forum-3Fid-3D11111111-2D0000-2D0000-2D0000-2D000000001606&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=VnicQ7hM5kAbUhEZZxbyL6g9pAlwAYXG0f12gJWrEew&s=NYTL-LzWTlb2LrZOkD1DwKonf9YHX5ujTpJYQmfQYiE&e=> forum.
Flash (Alert) IBM Spectrum Scale V4.2.1/4.2.2 parallel log recovery function may result in undetected data corruption Abstract

IBM has identified a problem with the IBM Spectrum Scale parallel log recovery function in V4.2.1/V4.2.2, which may result in undetected data corruption during the course of a file system recovery.



See the complete Flash at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009965<https://urldefense.proofpoint.com/v2/url?u=http-3A__www-2D01.ibm.com_support_docview.wss-3Fuid-3Dssg1S1009965&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=VnicQ7hM5kAbUhEZZxbyL6g9pAlwAYXG0f12gJWrEew&s=GZ88LDgDtQk90LnHRUEAuf08BL83oSKb0NFcUH-j5mU&e=>



________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.

________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.

_______________________________________________

gpfsug-discuss mailing list

gpfsug-discuss at spectrumscale.org

http://gpfsug.org/mailman/listinfo/gpfsug-discuss
The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems.

________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170224/4a608ebc/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 289 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170224/4a608ebc/attachment-0002.png>


More information about the gpfsug-discuss mailing list