From Philipp.Rehs at uni-duesseldorf.de  Sat May  1 13:12:05 2021
From: Philipp.Rehs at uni-duesseldorf.de (Philipp Rehs)
Date: Sat, 1 May 2021 14:12:05 +0200
Subject: [gpfsug-discuss] reinstalled gpfs gui but no admin user created
Message-ID: <7c1e1de0-3a98-e5d5-f424-cb2c71b23a77@uni-duesseldorf.de>

Hello,

today our gpfs gui crashed because of a broken hard drive and i needed
to reinstall the node. It is a legacy system still at 4.2.3.24

After the installation the gui is stuck at "Loading" and in the logs i
see the following errors:

Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]


For me it looks like it is related to this thread but it has no solution.

http://www.spectrumscale.org/pipermail/gpfsug-discuss/2017-December/004319.html

I already tried to install older versions of the gui but it did not help.

Is it possible to remove all old configurations even from ccr?

Kind regards
 Philipp

-- 
---------------------------

Zentrum f?r Informations- und Medientechnologie
Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern

Heinrich-Heine-Universit?t D?sseldorf
Universit?tsstr. 1
Raum 25.41.00.51
40225 D?sseldorf / Germany
Tel: +49-211-81-15557


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5523 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210501/18bb7e32/attachment.bin>

From scale at us.ibm.com  Mon May  3 12:04:57 2021
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Mon, 3 May 2021 16:34:57 +0530
Subject: [gpfsug-discuss] reinstalled gpfs gui but no admin user created
In-Reply-To: <7c1e1de0-3a98-e5d5-f424-cb2c71b23a77@uni-duesseldorf.de>
References: <7c1e1de0-3a98-e5d5-f424-cb2c71b23a77@uni-duesseldorf.de>
Message-ID: <OFC801667D.40F5A0FE-ON852586CA.003CE36A-652586CA.003CE108@notes.na.collabserv.com>


Hi Ratan,

Can you please look into this GUI issue.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------

If you feel that your question can benefit other users of  Spectrum Scale
(GPFS), then please post it to the public IBM developerWroks Forum at
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479.


If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact
1-800-237-5511 in the United States or your local IBM Service Center in
other countries.

The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.



From:	Philipp Rehs <Philipp.Rehs at uni-duesseldorf.de>
To:	gpfsug-discuss at spectrumscale.org
Date:	01-05-2021 05.50 PM
Subject:	[EXTERNAL] [gpfsug-discuss] reinstalled gpfs gui but no admin
            user created
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hello,

today our gpfs gui crashed because of a broken hard drive and i needed
to reinstall the node. It is a legacy system still at 4.2.3.24

After the installation the gui is stuck at "Loading" and in the logs i
see the following errors:

Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings
(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin
(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet
(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service
(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter
(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter
(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings
(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin
(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet
(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service
(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter
(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter
(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]


For me it looks like it is related to this thread but it has no solution.

http://www.spectrumscale.org/pipermail/gpfsug-discuss/2017-December/004319.html


I already tried to install older versions of the gui but it did not help.

Is it possible to remove all old configurations even from ccr?

Kind regards
 Philipp

--
---------------------------

Zentrum f?r Informations- und Medientechnologie
Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern

Heinrich-Heine-Universit?t D?sseldorf
Universit?tsstr. 1
Raum 25.41.00.51
40225 D?sseldorf / Germany
Tel: +49-211-81-15557


[attachment "smime.p7s" deleted by Huzefa H Pancha/India/IBM]
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210503/66e5866c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210503/66e5866c/attachment.gif>

From committee at io500.org  Wed May  5 17:38:45 2021
From: committee at io500.org (IO500 Committee)
Date: Wed, 05 May 2021 10:38:45 -0600
Subject: [gpfsug-discuss] Call For Submissions IO500 ISC21 List
Message-ID: <561bba3f84c7ef935c1dd67004bec46a@io500.org>

https://io500.org/cfs
Stabilization Period: 05 - 14 May 2021 AoE
Submission Deadline: 11 June 2021 AoE

The IO500 is now accepting and encouraging submissions for the upcoming 
8th IO500 list. Once again, we are also accepting submissions to the 10 
Node Challenge to encourage the submission of small scale results. The 
new ranked lists will be announced via live-stream at a virtual session. 
We hope to see many new results.

What's New
Starting with ISC'21, the IO500 now follows a two-staged approach. 
First, there will be a two-week stabilization period during which we 
encourage the community to verify that the benchmark runs properly. 
During this period the benchmark will be updated based upon feedback 
from the community. The final benchmark will then be released on Monday, 
May 1st. We expect that runs compliant with the rules made during the 
stabilization period are valid as the final submission unless a 
significant defect is found.
We are now creating a more detailed schema to describe the hardware and 
software of the system under test and provide the first set of tools to 
ease capturing of this information for inclusion with the submission. 
Further details will be released on the submission page.

Background
The benchmark suite is designed to be easy to run and the community has 
multiple active support channels to help with any questions. Please note 
that submissions of all sizes are welcome; the site has customizable 
sorting, so it is possible to submit on a small system and still get a 
very good per-client score, for example. Additionally, the list is about 
much more than just the raw rank; all submissions help the community by 
collecting and publishing a wider corpus of data. More details below.

Following the success of the Top500 in collecting and analyzing 
historical trends in supercomputer technology and evolution, the IO500 
was created in 2017, published its first list at SC17, and has grown 
exponentially since then. The need for such an initiative has long been 
known within High-Performance Computing; however, defining appropriate 
benchmarks had long been challenging. Despite this challenge, the 
community, after long and spirited discussion, finally reached consensus 
on a suite of benchmarks and a metric for resolving the scores into a 
single ranking.

The multi-fold goals of the benchmark suite are as follows:

Maximizing simplicity in running the benchmark suite
Encouraging optimization and documentation of tuning parameters for 
performance
Allowing submitters to highlight their "hero run" performance numbers
Forcing submitters to simultaneously report performance for challenging 
IO patterns.

Specifically, the benchmark suite includes a hero-run of both IOR and 
mdtest configured however possible to maximize performance and establish 
an upper-bound for performance. It also includes an IOR and mdtest run 
with highly prescribed parameters in an attempt to determine a 
lower-bound. Finally, it includes a namespace search as this has been 
determined to be a highly sought-after feature in HPC storage systems 
that has historically not been well-measured. Submitters are encouraged 
to share their tuning insights for publication.

The goals of the community are also multi-fold:

Gather historical data for the sake of analysis and to aid predictions 
of storage futures
Collect tuning information to share valuable performance optimizations 
across the community
Encourage vendors and designers to optimize for workloads beyond "hero 
runs"
Establish bounded expectations for users, procurers, and administrators

10 Node I/O Challenge
The 10 Node Challenge is conducted using the regular IO500 benchmark, 
however, with the rule that exactly 10 client nodes must be used to run 
the benchmark. You may use any shared storage with, e.g., any number of 
servers. When submitting for the IO500 list, you can opt-in for 
"Participate in the 10 compute node challenge only", then we will not 
include the results into the ranked list. Other 10-node node submissions 
will be included in the full list and in the ranked list. We will 
announce the result in a separate derived list and in the full list but 
not on the ranked IO500 list at io500.org.

Birds-of-a-Feather
Once again, we encourage you to submit to join our community, and to 
attend our virtual BoF "The IO500 and the Virtual Institute of I/O" at 
ISC 2021, (time to be announced), where we will announce the new IO500 
and 10 node challenge lists. The current list includes results from 
BeeGFS, CephFS, DAOS, DataWarp, GekkoFS, GFarm, IME, Lustre, MadFS, 
Qumulo, Spectrum Scale, Vast, WekaIO, and YRCloudFile. We hope that the 
upcoming list grows even more.


-- 
The IO500 Committee


From heinrich.billich at id.ethz.ch  Mon May 10 13:16:57 2021
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Mon, 10 May 2021 12:16:57 +0000
Subject: [gpfsug-discuss] migrate to new metadata GNR systems while
 maintaining a fallback possibilty
Message-ID: <62CAB006-A4B1-4E19-9ADD-A3DA5E9C3BC2@id.ethz.ch>

Hello,

 

I need to update/replace our metadata disks but I want to keep old and new in parallel for a while before I remove the old storage: as active/active pair with double copies. This will allow an immediate fall-back if we ever need.

 

Maybe you want to comment on this procedure ? I probably found it some time ago on this mailing list, sorry if I don?t remember the author.

 

Some concerns I have

I?m a bit worried what happens when we remove the vdisks with the second copy in the last step ? will be get millions of error messages? ?Is the sequence of commands right? 

I need to change the failure groups of some vdisks while in use ? I wonder if this poses some risk? As I understand this will change the order of block allocation among the nsds (not the allocation map I guess)

This will double metadata write-io, the systems should be able to handle it. 

We will get? better metadata read-io during the transition than what we?ll finally get.

 

== Start ? ?old? ESS/GNR only

 

 -m 1 (single metadata copy), 

?-M 2

 -K whenpossible

 Metadata vdisks in failure groups 1 and 2

 

== Preparation

 

Use? mmvdisk filesystems? and move all metadata vdisk-sets to a single failuregroup (this has to be done in operation, under load)

Set ?-m 2, while we still have one failure group only. -K whenpossible will keep us running

Add the new vdisk-set with a second failure group on ?new? ESS/GNR systems

Now all new inodes have two copies, one on old and one on new

 

== Create copies on ?old? and ?new?

 

 mmrestripefs -R with ?qos maintenance and -N helper-nodes to minimize the load. This may create some locks on the filesystem/cluster and interfere with backup and snapshots??

Maybe better: use a policy ?replicate? rule to replicate all files, I can run this in small chunks and run mmrestripefs just once to crosscheck.

 

== Observe for some days, handle remaining filesystems in the same way

 

== Finally

 

Suspend ?old? disks

Run mmrestripefs -m

Remove ?old? vdisk sets with mmvidisk ? this will run another mmdeldisk

Change to -m 1

Run fix replication setting mmrestripefs -R (if needed?)

 

Thank you for reading and for any comments or suggestions.

 

Heiner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210510/bf8bb6f9/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5254 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210510/bf8bb6f9/attachment.bin>

From S.J.Thompson at bham.ac.uk  Mon May 10 18:08:24 2021
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Mon, 10 May 2021 17:08:24 +0000
Subject: [gpfsug-discuss] UK UG survey
Message-ID: <A2712CC4-3877-41E3-9B9C-159D10776C74@bham.ac.uk>

I?m currently trying to gather some feedback on running an in-person SSUG event in the UK later this year ? (subject to rules etc etc). If you would normally attend the event in the UK (or you would be very likely to), I?d appreciate 3 mins of time to fill out a quick survey to gather opinion ?

https://www.surveymonkey.co.uk/r/TVWVWGC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210510/e8dabada/attachment.htm>

From mutantllama at gmail.com  Wed May 12 04:40:50 2021
From: mutantllama at gmail.com (Carl)
Date: Wed, 12 May 2021 13:40:50 +1000
Subject: [gpfsug-discuss] Experience running CrowdStrike with Spectrum Scale
Message-ID: <CAGedhdf0bi0np1z-bpMgEzvumuK3j2ZbOgFV0wQfh3842Mb06w@mail.gmail.com>

Hi folks,

We are currently looking at deploying CrowdStrike on our login nodes.
Has anyone had experience with running CrowdStrike with Spectrum
Scale? Did you have any issues?

Cheers,

Carl.


From TROPPENS at de.ibm.com  Fri May 14 12:52:25 2021
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Fri, 14 May 2021 11:52:25 +0000
Subject: [gpfsug-discuss] Charts of German User Meeting are now available
Message-ID: <OF8E05C51D.D4F014E9-ON002586D5.00412744-002586D5.004139A2@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210514/b26ee9ec/attachment.htm>

From chair at spectrumscale.org  Sun May 16 19:14:05 2021
From: chair at spectrumscale.org (Simon Thompson (Spectrum Scale User Group Chair))
Date: Sun, 16 May 2021 19:14:05 +0100
Subject: [gpfsug-discuss] SSUG::Digital: What is new in Spectrum Scale 5.1.1?
Message-ID: <>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210516/d3275165/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: meeting.ics
Type: text/calendar
Size: 2418 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210516/d3275165/attachment.ics>

From Kritika.Gulati at ibm.com  Tue May 18 15:29:45 2021
From: Kritika.Gulati at ibm.com (Kritika Gulati)
Date: Tue, 18 May 2021 14:29:45 +0000
Subject: [gpfsug-discuss] reinstalled gpfs gui but no admin user created
Message-ID: <OF8971FFD2.1980C2A9-ON002586D9.004EED6E-002586D9.004FA113@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210518/8fc48e6a/attachment.htm>

From davebond7787 at gmail.com  Thu May 20 13:56:51 2021
From: davebond7787 at gmail.com (Dave Bond)
Date: Thu, 20 May 2021 13:56:51 +0100
Subject: [gpfsug-discuss] FAL audit events not always triggered
In-Reply-To: <CAHuryA0f86GG33D9_0+YVtSnMVfBFoPcaQq6z4BjD9xB8M0T6A@mail.gmail.com>
References: <CAHuryA0f86GG33D9_0+YVtSnMVfBFoPcaQq6z4BjD9xB8M0T6A@mail.gmail.com>
Message-ID: <CAHuryA2ymNHGkLJE=20UMcpnaOq6XOHgjLnM9hEU0ZPN_r_BRw@mail.gmail.com>

Hello

I am currently testing out the FAL and watch folders auditing mechanism in
5.1.0.  I have noticed that audit events such as file access or creation
are not always recorded in the FAL log on the filesystem.  It would seem
necessary to disable and re enable FAL auditing for the same file access to
be recorded.  I have only triggered this condition twice so far, but it has
set in my mind a few questions.

1) Have others seen this?  If so what is the trigger, as I do not yet have
a reproducer.
2) Is it intended for the audit mechanism to be a soft audit?  i.e the
audit is best effort, and non blocking for file access.

This is using a single NSD and a single remote cluster doing the file
writing.  FAL is running on the NSD node.

Regards

Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/c4895935/attachment.htm>

From davebond7787 at gmail.com  Thu May 20 13:58:01 2021
From: davebond7787 at gmail.com (Dave Bond)
Date: Thu, 20 May 2021 13:58:01 +0100
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <CAHuryA11v_7AWSssSoKzD8PLb8gyNeGsoZe39hZPCN=TDcJ8Pw@mail.gmail.com>
References: <CAHuryA11v_7AWSssSoKzD8PLb8gyNeGsoZe39hZPCN=TDcJ8Pw@mail.gmail.com>
Message-ID: <CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>

Hello

As part of a project I am doing I am looking if there are any de
duplication options for GPFS?  I see there is no native de dupe for the
filesystem. The scenario would be user A creates a file or folder and user
B takes a copy within the same filesystem, though separate independent
filesets.  The intention would be to store 1 copy.    So I was wondering
....

1) Is this is planned to be implemented into GPFS in the future?
2) Is anyone is using any other solutions that have good GPFS integration?

Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/71a20baa/attachment.htm>

From abeattie at au1.ibm.com  Thu May 20 14:27:57 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Thu, 20 May 2021 13:27:57 +0000
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>
Message-ID: <OF162CFDAD.3050490E-ON002586DB.0049F835-1621517277192@notes.na.collabserv.com>

Dave,

Spectrum Scale does not support de-duplication, it does support compression. 
You can however use block storage that supports over subscription / thin provisioning / deduplication for data only NSD?s, we do not recommend them for metadata.

In your scenario is user B planning on making changes to the data which is why you need a copy? 

I know of customers that do this regularly with block storage such as the IBM Flashsystem product family In conjunction with IBM Spectrum Copy Data Management.  But I don?t believe CDM supports file based storage.

Regards, 

Andrew Beattie
Technical Sales Specialist
Storage for Data and AI
IBM Australia and New Zealand
P. +61 421 337 927
E. Abeattie at au1.ibm.com

> On 20 May 2021, at 22:58, Dave Bond <davebond7787 at gmail.com> wrote:
> 
> ?
> This Message Is From an External Sender
> This message came from outside your organization.
> 
> 
> Hello
> 
> As part of a project I am doing I am looking if there are any de duplication options for GPFS?  I see there is no native de dupe for the filesystem. The scenario would be user A creates a file or folder and user B takes a copy within the same filesystem, though separate independent filesets.  The intention would be to store 1 copy.    So I was wondering ....
> 
> 1) Is this is planned to be implemented into GPFS in the future?
> 2) Is anyone is using any other solutions that have good GPFS integration?
> 
> Dave

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/670d8c55/attachment.htm>

From Luis.I.Teran at ibm.com  Thu May 20 23:26:36 2021
From: Luis.I.Teran at ibm.com (Luis I Teran)
Date: Thu, 20 May 2021 22:26:36 +0000
Subject: [gpfsug-discuss] FAL audit events not always triggered
Message-ID: <OF7630AC17.4C49DFA1-ON002586DB.00792014-002586DB.007B494D@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/3336ec92/attachment.htm>

From ulmer at ulmer.org  Fri May 21 00:42:30 2021
From: ulmer at ulmer.org (Stephen Ulmer)
Date: Thu, 20 May 2021 19:42:30 -0400
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <OF162CFDAD.3050490E-ON002586DB.0049F835-1621517277192@notes.na.collabserv.com>
References: <OF162CFDAD.3050490E-ON002586DB.0049F835-1621517277192@notes.na.collabserv.com>
Message-ID: <16E2BBAA-3DC2-41B7-83C2-F875CDA3D57E@ulmer.org>

Do file clones meet the workflow requirement? That is, can you control from whence the second (and further) copies are made?

 -- 
Stephen


> On May 20, 2021, at 9:01 AM, Andrew Beattie <abeattie at au1.ibm.com> wrote:
> 
> ?Dave,
> 
> Spectrum Scale does not support de-duplication, it does support compression. 
> You can however use block storage that supports over subscription / thin provisioning / deduplication for data only NSD?s, we do not recommend them for metadata.
> 
> In your scenario is user B planning on making changes to the data which is why you need a copy? 
> 
> I know of customers that do this regularly with block storage such as the IBM Flashsystem product family In conjunction with IBM Spectrum Copy Data Management.  But I don?t believe CDM supports file based storage.
> 
> Regards, 
> 
> Andrew Beattie
> Technical Sales Specialist
> Storage for Data and AI
> IBM Australia and New Zealand
> P. +61 421 337 927
> E. Abeattie at au1.ibm.com
> 
>>> On 20 May 2021, at 22:58, Dave Bond <davebond7787 at gmail.com> wrote:
>>> 
>> ?
>> 
>> 
>> Hello
>> 
>> As part of a project I am doing I am looking if there are any de duplication options for GPFS?  I see there is no native de dupe for the filesystem. The scenario would be user A creates a file or folder and user B takes a copy within the same filesystem, though separate independent filesets.  The intention would be to store 1 copy.    So I was wondering ....
>> 
>> 1) Is this is planned to be implemented into GPFS in the future?
>> 2) Is anyone is using any other solutions that have good GPFS integration?
>> 
>> Dave
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/ce7a47ee/attachment.htm>

From david.reynolds at tes-es.com  Fri May 21 09:42:11 2021
From: david.reynolds at tes-es.com (David Reynolds)
Date: Fri, 21 May 2021 08:42:11 +0000
Subject: [gpfsug-discuss] Spectrum Scale & S3
Message-ID: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>

When we talk about supported protocols on Scale, S3 is listed but its normally prefixed with Object. Does this mean that the S3 connectivity is used purely to connect and migrate data in object stores to parallel systems (and vice versa) or can you connect to applications/workflows using S3?

David Reynolds
TES Enterprise Solutions
david.reynolds at tes-es.com<mailto:david.reynolds at tes-es.com>
07594 524986

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210521/4d7c6b04/attachment.htm>

From jonathan.buzzard at strath.ac.uk  Fri May 21 09:57:23 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Fri, 21 May 2021 09:57:23 +0100
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>
References: <CAHuryA11v_7AWSssSoKzD8PLb8gyNeGsoZe39hZPCN=TDcJ8Pw@mail.gmail.com>
	<CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>
Message-ID: <616926ad-e16b-2e7d-d45d-da3a803ae185@strath.ac.uk>

On 20/05/2021 13:58, Dave Bond wrote:

> As part of a project I am doing I am looking if there are any de 
> duplication options for GPFS?? I see there is no native de dupe for the 
> filesystem. The scenario would be user A creates a file or folder and 
> user B takes a copy within the same filesystem, though separate 
> independent filesets.? The intention would be to store 1 copy.? ? So I 
> was wondering ....
> 
> 1) Is this is planned to be implemented into GPFS in the future?
> 2) Is anyone is using any other solutions that have good GPFS integration?
> 

Disk space in 2021 is insanely cheap. With an ESS/DSS you can get many 
PB in a single rack. The complexity that dedup introduces is simple not 
worth it IMHO.

Or put another way there is better things the developers at IBM can be 
working on than dedup code.

Historically if you crunched the numbers the licensing for dedup on 
NetApp was similar to just buying more disk unless you where storing 
hundreds of copies of the same data. About the only use case scenario 
would be storing lots of virtual machines. However I refer you back to 
my original point :-)


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From abeattie at au1.ibm.com  Fri May 21 10:56:38 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Fri, 21 May 2021 09:56:38 +0000
Subject: [gpfsug-discuss] Spectrum Scale & S3
In-Reply-To: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>
Message-ID: <OF964F942C.E8EC157A-ON002586DC.00369F4C-1621590998632@notes.na.collabserv.com>

David,

You can use the CES protocol nodes to present a swift Object interface or S3 Object interface for applications to access data from the filesystem / write data into the filesystem over the object protocol. 

Check the openstack Swift -S3 compatabioity matrix to make sure that the functionality from swift aligns with your App requirements 

Sent from my iPhone

> On 21 May 2021, at 18:42, David Reynolds <david.reynolds at tes-es.com> wrote:
> 
> ?
> This Message Is From an External Sender
> This message came from outside your organization.
> When we talk about supported protocols on Scale, S3 is listed but its normally prefixed with Object. Does this mean that the S3 connectivity is used purely to connect and migrate data in object stores to parallel systems (and vice versa) or can you connect to applications/workflows using S3?
>  
> David Reynolds
> TES Enterprise Solutions
> david.reynolds at tes-es.com
> 07594 524986
>  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210521/173855a9/attachment.htm>

From janfrode at tanso.net  Fri May 21 15:16:17 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 21 May 2021 16:16:17 +0200
Subject: [gpfsug-discuss] Spectrum Scale & S3
In-Reply-To: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>
References: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>
Message-ID: <CAHwPath6YvRcJhijqHoedsXXqgx6GTBvOW82rDvseM18fDU9uA@mail.gmail.com>

It has features for both being an Object store for other applications
(running openstack swift/S3), and for migrating/tiering filesystem data to
an object store like Amazon S3, IBM COS, etc...


  -jf

fre. 21. mai 2021 kl. 10:42 skrev David Reynolds <david.reynolds at tes-es.com
>:

> When we talk about supported protocols on Scale, S3 is listed but its
> normally prefixed with Object. Does this mean that the S3 connectivity is
> used purely to connect and migrate data in object stores to parallel
> systems (and vice versa) or can you connect to applications/workflows using
> S3?
>
>
>
> David Reynolds
>
> TES Enterprise Solutions
>
> david.reynolds at tes-es.com
>
> 07594 524986
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210521/a8eb7280/attachment.htm>

From tomasz.rachobinski at gmail.com  Mon May 24 14:06:01 2021
From: tomasz.rachobinski at gmail.com (=?UTF-8?Q?Tomasz_Rachobi=C5=84ski?=)
Date: Mon, 24 May 2021 15:06:01 +0200
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0
Message-ID: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>

Hello everyone,
we are trying to implement a mixed linux/windows environment and we have
one thing at the top - is there any global method to avoid asynchronous I/O
and write everything in synchronous mode?
Another thing is - if there is no global sync setting, how to enforce sync
i/o from linux/windows client?

Greetings
Tom Rachobinski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210524/be2a202b/attachment.htm>

From Alec.Effrat at wellsfargo.com  Mon May 24 18:23:38 2021
From: Alec.Effrat at wellsfargo.com (Alec.Effrat at wellsfargo.com)
Date: Mon, 24 May 2021 17:23:38 +0000
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0
In-Reply-To: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
References: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
Message-ID: <22a8b9b6fa474a62927ed5e215a548c6@wellsfargo.com>

In the past we?d used EMC RecoverPoint for a Spectrum Scale file system.  It isn?t officially supported, and it doesn?t provide for an online replica, but it can keep the I/O synchronous and DVR style file system playback and we had written our own integrations for it via it?s SSH interface.  It is a very competent solution and does pair well with GPFS Spectrum Scale.

Our Spectrum Scale interface basically had to do the following:

1)      Ship updates to the mmdrfs file to the target cluster.

2)      In control failover we would tag the image once GPFS was stopped

3)      Come up on the tagged image on BCP

4)      Had an awk script that would strip out all the host details from the mmdrfs file and update it to 1 host in the BCP cluster.

5)      Import the GPFS file system using the modified mmdrfs file.

Just one way to skin the cat for consideration.  Can provide more actuals on the replication scripts if interested.

Alec Effrat
SAS Lead, AVP
Business Intelligence Competency Center
SAS Administration
Cell 949-246-7713
alec.effrat at wellsfargo.com<mailto:alec.effrat at wellsfargo.com>




From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Tomasz Rachobinski
Sent: Monday, May 24, 2021 6:06 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0

Hello everyone,
we are trying to implement a mixed linux/windows environment and we have one thing at the top - is there any global method to avoid asynchronous I/O and write everything in synchronous mode?
Another thing is - if there is no global sync setting, how to enforce sync i/o from linux/windows client?

Greetings
Tom Rachobinski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210524/75bb9936/attachment.htm>

From krajaram at geocomputing.net  Mon May 24 22:39:59 2021
From: krajaram at geocomputing.net (Kumaran Rajaram)
Date: Mon, 24 May 2021 21:39:59 +0000
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0
In-Reply-To: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
References: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
Message-ID: <SJ0PR18MB4011C053C80A70BEA944CACCBB269@SJ0PR18MB4011.namprd18.prod.outlook.com>

Hi Tom,

>>we are trying to implement a mixed linux/windows environment and we have one thing at the top - is there any global method to avoid asynchronous I/O and write everything in >>synchronous mode?

If the local and remote sites have good inter-site network bandwidth and low-latency, then you may consider using GPFS synchronous replication at the file-system level (-m 2 -r 2).  The Spectrum Scale documentation (link below) has further details.

https://www.ibm.com/docs/en/spectrum-scale/5.1.0?topic=data-synchronous-mirroring-gpfs-replication

Regards,
-Kums

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Tomasz Rachobinski
Sent: Monday, May 24, 2021 9:06 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0

Hello everyone,
we are trying to implement a mixed linux/windows environment and we have one thing at the top - is there any global method to avoid asynchronous I/O and write everything in synchronous mode?
Another thing is - if there is no global sync setting, how to enforce sync i/o from linux/windows client?

Greetings
Tom Rachobinski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210524/bf198bcd/attachment.htm>

From coetzee.ray at gmail.com  Wed May 26 18:33:20 2021
From: coetzee.ray at gmail.com (Ray Coetzee)
Date: Wed, 26 May 2021 18:33:20 +0100
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
Message-ID: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>

Hello all

I'd be interested to know if anyone else has experienced a problem with Kernel
> 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).

We noticed that python shut.copy() is failing against a GPFS mount with:

BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
-> 'test2.file'

To reproduce the error:

```
[user at login01]$ module load python-3.8.9-gcc-9.3.0-soqwnzh

[ user at login01]$ truncate --size 640MB test.file
[ user at login01]$ python3 -c "import shutil; shutil.copy('test.file',
'test2.file')"
Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 418, in copy
 copyfile(src, dst, follow_symlinks=follow_symlinks)
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 275, in copyfile
 _fastcopy_sendfile(fsrc, fdst)
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 172, in _fastcopy_sendfile
 raise err
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 152, in _fastcopy_sendfile
 sent = os.sendfile(outfd, infd, offset, blocksize)
BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
-> 'test2.file'



 Investigating into why this is happening revealed that:


1. It is failing for python3.8 and above.
2. It is happening only a GPFS mount
3. It is happening with files whose size is multiple of 4KB (OS Page size)

Relevant links:
https://bugs.python.org/issue43743
https://www.ibm.com/support/pages/apar/IJ28891


Doing an strace revealed that at the lower level, it seems to be related to
the Linux Syscall of ?sendfile?, which seems to fail in these cases on GPFS.


Strace for a 4096 B file:

```
sendfile(4, 3, [0] => [4096], 8388608) = 4096
sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily
unavailable)

```

The same file on other disk:
```
sendfile(4, 3, [0] => [4096], 8388608) = 4096
sendfile(4, 3, [4096], 8388608) = 0


IBM's "fix" for the problem of "Do not use a file size which that is a
multiple of the page size."  sounds really blas?.


```


Kind regards

Ray Coetzee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/cf991430/attachment.htm>

From knop at us.ibm.com  Wed May 26 18:45:38 2021
From: knop at us.ibm.com (Felipe Knop)
Date: Wed, 26 May 2021 17:45:38 +0000
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
References: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
Message-ID: <OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/ff9d3d0c/attachment.htm>

From coetzee.ray at gmail.com  Wed May 26 19:27:10 2021
From: coetzee.ray at gmail.com (Ray Coetzee)
Date: Wed, 26 May 2021 19:27:10 +0100
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
References: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
	<OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
Message-ID: <CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>

Hi Felipe

We looked at APAR  IJ28891 & IJ29942, as both look identical but are closed
as a "program error" with a workaround of "Do not use a file size which
that is a multiple of the page size."

Kind regards

Ray Coetzee


On Wed, May 26, 2021 at 6:45 PM Felipe Knop <knop at us.ibm.com> wrote:

> Ray,
>
> I wonder if you are hitting the problem which was fixed on the following
> APAR:
>
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
>   Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
>
>
> ----- Original message -----
> From: Ray Coetzee <coetzee.ray at gmail.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
> Date: Wed, May 26, 2021 1:38 PM
>
> Hello all
>
> I'd be interested to know if anyone else has experienced a problem with Kernel
> > 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).
>
>
> We noticed that python shut.copy() is failing against a GPFS mount with:
>
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
> To reproduce the error:
>
> ```
> [user at login01]$ module load python-3.8.9-gcc-9.3.0-soqwnzh
>
> [ user at login01]$ truncate --size 640MB test.file
> [ user at login01]$ python3 -c "import shutil; shutil.copy('test.file',
> 'test2.file')"
> Traceback (most recent call last):
>  File "<string>", line 1, in <module>
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 418, in copy
>  copyfile(src, dst, follow_symlinks=follow_symlinks)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 275, in copyfile
>  _fastcopy_sendfile(fsrc, fdst)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 172, in _fastcopy_sendfile
>  raise err
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 152, in _fastcopy_sendfile
>  sent = os.sendfile(outfd, infd, offset, blocksize)
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
>
>
>  Investigating into why this is happening revealed that:
>
>
> 1. It is failing for python3.8 and above.
> 2. It is happening only a GPFS mount
> 3. It is happening with files whose size is multiple of 4KB (OS Page size)
>
> Relevant links:
> https://bugs.python.org/issue43743
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
> Doing an strace revealed that at the lower level, it seems to be related
> to the Linux Syscall of ?sendfile?, which seems to fail in these cases on
> GPFS.
>
>
> Strace for a 4096 B file:
>
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily
> unavailable)
>
> ```
>
> The same file on other disk:
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = 0
>
>
>
> IBM's "fix" for the problem of "Do not use a file size which that is a
> multiple of the page size."  sounds really blas?.
>
>
> ```
>
>
> Kind regards
>
> Ray Coetzee
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/450af1a7/attachment.htm>

From knop at us.ibm.com  Wed May 26 19:36:40 2021
From: knop at us.ibm.com (Felipe Knop)
Date: Wed, 26 May 2021 18:36:40 +0000
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>
References: <CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>,
	<CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
	<OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
Message-ID: <OFC1A21264.1ACD353E-ON002586E1.0065CC6B-002586E1.00663C06@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/29fc799d/attachment.htm>

From coetzee.ray at gmail.com  Wed May 26 21:21:32 2021
From: coetzee.ray at gmail.com (Ray Coetzee)
Date: Wed, 26 May 2021 21:21:32 +0100
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <OFC1A21264.1ACD353E-ON002586E1.0065CC6B-002586E1.00663C06@notes.na.collabserv.com>
References: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
	<OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
	<CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>
	<OFC1A21264.1ACD353E-ON002586E1.0065CC6B-002586E1.00663C06@notes.na.collabserv.com>
Message-ID: <CABQRq0EAO5aXKFR+UY28mjUrphOTMrH-orC2meLxRr1ReoZ5vg@mail.gmail.com>

Thank you for the info Felipe.

I'll test with 5.0.5-7 in the morning.

Kind regards

Ray Coetzee




On Wed, May 26, 2021 at 7:36 PM Felipe Knop <knop at us.ibm.com> wrote:

> Ray,
>
> Apologies; I should have added more details. My records show that there is
> a fix for IJ28891 and IJ29942, which was delivered in
>
> 5.1.0.2
> 5.0.5.5
>
> "Program error" I believe is a code that indicates "needs to be fixed in
> our product".
>
> I agree that the workaround mentioned is not "actionable" . The APAR page
> should have been clear that there is a fix available.
>
> Regards,
>
>   Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
>
>
> ----- Original message -----
> From: Ray Coetzee <coetzee.ray at gmail.com>
> To: Felipe Knop <knop at us.ibm.com>
> Cc: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
> Date: Wed, May 26, 2021 2:27 PM
>
> Hi Felipe We looked at APAR IJ28891 & IJ29942, as both look identical but
> are closed as a "program error" with a workaround of "Do not use a file
> size which that is a multiple of the page size." Kind regards ? ? ? ? ? ? ?
> ? ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
> ZjQcmQRYFpfptBannerEnd
> Hi Felipe
>
> We looked at APAR  IJ28891 & IJ29942, as both look identical but are
> closed as a "program error" with a workaround of "Do not use a file size
> which that is a multiple of the page size."
>
> Kind regards
>
> Ray Coetzee
>
>
> On Wed, May 26, 2021 at 6:45 PM Felipe Knop <knop at us.ibm.com> wrote:
>
> Ray,
>
> I wonder if you are hitting the problem which was fixed on the following
> APAR:
>
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
>   Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
>
>
> ----- Original message -----
> From: Ray Coetzee <coetzee.ray at gmail.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
> Date: Wed, May 26, 2021 1:38 PM
>
> Hello all
>
> I'd be interested to know if anyone else has experienced a problem with Kernel
> > 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).
>
>
> We noticed that python shut.copy() is failing against a GPFS mount with:
>
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
> To reproduce the error:
>
> ```
> [user at login01]$ module load python-3.8.9-gcc-9.3.0-soqwnzh
>
> [ user at login01]$ truncate --size 640MB test.file
> [ user at login01]$ python3 -c "import shutil; shutil.copy('test.file',
> 'test2.file')"
> Traceback (most recent call last):
>  File "<string>", line 1, in <module>
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 418, in copy
>  copyfile(src, dst, follow_symlinks=follow_symlinks)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 275, in copyfile
>  _fastcopy_sendfile(fsrc, fdst)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 172, in _fastcopy_sendfile
>  raise err
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 152, in _fastcopy_sendfile
>  sent = os.sendfile(outfd, infd, offset, blocksize)
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
>
>
>  Investigating into why this is happening revealed that:
>
>
> 1. It is failing for python3.8 and above.
> 2. It is happening only a GPFS mount
> 3. It is happening with files whose size is multiple of 4KB (OS Page size)
>
> Relevant links:
> https://bugs.python.org/issue43743
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
> Doing an strace revealed that at the lower level, it seems to be related
> to the Linux Syscall of ?sendfile?, which seems to fail in these cases on
> GPFS.
>
>
> Strace for a 4096 B file:
>
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily
> unavailable)
>
> ```
>
> The same file on other disk:
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = 0
>
>
>
> IBM's "fix" for the problem of "Do not use a file size which that is a
> multiple of the page size."  sounds really blas?.
>
>
> ```
>
>
> Kind regards
>
> Ray Coetzee
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/9096ca4a/attachment.htm>

From pbasmaji at us.ibm.com  Thu May 27 00:59:23 2021
From: pbasmaji at us.ibm.com (Peter Basmajian)
Date: Wed, 26 May 2021 23:59:23 +0000
Subject: [gpfsug-discuss] Be a Spectrum Scale *private* reference,
	get a Spectrum Scale t-shirt!
Message-ID: <OF0163861D.E8273E15-ON002586E1.007605D4-002586E1.0083C7E7@ibm.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/06eba1f6/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Image.1622066097985.png
Type: image/png
Size: 155627 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/06eba1f6/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Image.1622066115686.png
Type: image/png
Size: 133174 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/06eba1f6/attachment-0001.png>

From henrik at morsing.cc  Thu May 27 16:10:39 2021
From: henrik at morsing.cc (Henrik Morsing)
Date: Thu, 27 May 2021 16:10:39 +0100
Subject: [gpfsug-discuss] Ransom attacks
Message-ID: <20210527151039.GC8399@morsing.cc>


Hi,

It struck me that switching a Spectrum Protect solution from tapes to a GPFS filesystem offers much less protection against ransom encryption should the SP server be compromised. Same goes really for compromising an ESS node itself, it is an awful lot of data that can be encrypted very quickly.

Is there anything that can protect the GPFS filesystem against this kind of attack?

Regards,
Henrik


From anobre at br.ibm.com  Thu May 27 16:20:08 2021
From: anobre at br.ibm.com (Anderson Ferreira Nobre)
Date: Thu, 27 May 2021 15:20:08 +0000
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <OF65FE85E0.F99DD819-ON002586E2.005411E1-002586E2.00543DD2@ibm.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210527/c5e51dc4/attachment.htm>

From skylar2 at uw.edu  Thu May 27 16:23:48 2021
From: skylar2 at uw.edu (Skylar Thompson)
Date: Thu, 27 May 2021 08:23:48 -0700
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <20210527152348.3h6qmitg2cyhaqus@thargelion>

You can get clever/complicated (the interpretation could go either way)
with ACLs and SELinux but, at the end of the day, nothing beats the air-gap
of tape backups, IMHO. You might consider a belt&suspenders approach that
includes all of the above plus other controls (2FA, network security,
etc.), and in my experience combining multiple solutions gives flexibility
in that it can be easier to avoid the higher-cost aspects of one solution
taken to an extreme by having one layer mitigate the shortcomings of
another layer.

On Thu, May 27, 2021 at 04:10:39PM +0100, Henrik Morsing wrote:
> 
> Hi,
> 
> It struck me that switching a Spectrum Protect solution from tapes to a GPFS filesystem offers much less protection against ransom encryption should the SP server be compromised. Same goes really for compromising an ESS node itself, it is an awful lot of data that can be encrypted very quickly.
> 
> Is there anything that can protect the GPFS filesystem against this kind of attack?

-- 
-- Skylar Thompson (skylar2 at u.washington.edu)
-- Genome Sciences Department (UW Medicine), System Administrator
-- Foege Building S046, (206)-685-7354
-- Pronouns: He/Him/His


From jonathan.buzzard at strath.ac.uk  Thu May 27 16:49:02 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Thu, 27 May 2021 16:49:02 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527152348.3h6qmitg2cyhaqus@thargelion>
References: <20210527151039.GC8399@morsing.cc>
	<20210527152348.3h6qmitg2cyhaqus@thargelion>
Message-ID: <64fa4f59-a73a-19d4-a789-d63c9955cf64@strath.ac.uk>

On 27/05/2021 16:23, Skylar Thompson wrote:

[SNIP]

> at the end of the day, nothing beats the air-gap of tape backups, IMHO. 

Changing/deleting lots of data on tape takes time. So tape is a really 
good starting point even if you never take the tapes out the library 
except to dispose of them. Your backup is your get out of jail card. 
Protect it like it's Fort Knox.

A bit of security through obscurity by using Power and AIX will not go 
amiss. Even if it only buys you a couple of hours that can be enough to 
save the backup from harm.

Passwords on the Spectrum Protect server should be good *never* be used 
anywhere else, and only a handful of trusted people should have access 
to them.

Make sure you have a reuse delay on those tapes so even if the bastards 
do a del filespace (if they even know how to use TSM) you can roll back 
the database.

I also have the notion that you should be able to cut the power to the 
Spectrum Protect server and tape libraries such that it requires an on 
site visit to manually power them backup by flicking a nice big molly 
switch. I have a notion in my mind of tripping a residual-current 
device/ground fault circuit interrupter by using a relay to create a 
neutral earth fault. First sign of trouble disconnect the backup system :-)


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From rltodd.ml1 at gmail.com  Thu May 27 19:17:37 2021
From: rltodd.ml1 at gmail.com (Lindsay Todd)
Date: Thu, 27 May 2021 14:17:37 -0400
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>

Henrik,

Generally you need to begin with a good backup or replica, as well as
suitable air-gaps to isolate contamination.  You also need to be able to
quickly detect unusual activity - an SIEM tool like QRadar might help.
Assume that a cyber-incident will happen and plan accordingly.  Use
in-depth security.  But you are right - you lose one of the advantages of
tape - you can make duplicate copies, maybe even a WORM copy, and store it
offsite.

You might at very least want to take snapshots of the storage being used by
Spectrum Protect, and have separate administrators for the ESS and SP
server (to reduce inside risk).  If it was actually GPFS being backed up to
SP, you could have a second GPFS file system that is a point-in-time
synchronized copy of the original GPFS file system - with its own
snapshots.  It could have yet another sysadmin, and you could isolate the
second copy from the network when not actively synchronizing. See
https://www.redbooks.ibm.com/abstracts/redp5559.html?Open

That might not make sense if GPFS is holding the SP backup data, but SP can
do its own replication too - and could replicate using storage from a
second GPFS file system off-site.  Take snapshots of this second storage,
as well as SP database, and again manage with a second sysadmin team.


*Lindsay Todd, PhD*
*Spectrum Scale (GPFS) Solution Architect*
*IBM Advanced Technology Group ? Storage*
*Mobile:** 1-518-369-6108*
*E-mail:* *lindsay at us.ibm.com* <lindsay at us.ibm.com>


On Thu, May 27, 2021 at 11:10 AM Henrik Morsing <henrik at morsing.cc> wrote:

>
> Hi,
>
> It struck me that switching a Spectrum Protect solution from tapes to a
> GPFS filesystem offers much less protection against ransom encryption
> should the SP server be compromised. Same goes really for compromising an
> ESS node itself, it is an awful lot of data that can be encrypted very
> quickly.
>
> Is there anything that can protect the GPFS filesystem against this kind
> of attack?
>
> Regards,
> Henrik
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210527/966d8a32/attachment.htm>

From henrik at morsing.cc  Fri May 28 07:46:35 2021
From: henrik at morsing.cc (Henrik Morsing)
Date: Fri, 28 May 2021 07:46:35 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>
References: <20210527151039.GC8399@morsing.cc>
	<CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>
Message-ID: <20210528064635.GD8399@morsing.cc>

On Thu, May 27, 2021 at 02:17:37PM -0400, Lindsay Todd wrote:
>
>That might not make sense if GPFS is holding the SP backup data, but SP can
>do its own replication too - and could replicate using storage from a
>second GPFS file system off-site.  Take snapshots of this second storage,
>as well as SP database, and again manage with a second sysadmin team.
>

Thanks all for some useful replies, something to take forward.

In this case, SP is using GPFS for storing backup data, this solution was meant to replace the tape libraries completely.

We protect the storage pools cross-site, but our solutions are identical, so if you hacked one, you have hacked both.

Regards,
Henrik


From henrik at morsing.cc  Fri May 28 08:15:37 2021
From: henrik at morsing.cc (Henrik Morsing)
Date: Fri, 28 May 2021 08:15:37 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <OF65FE85E0.F99DD819-ON002586E2.005411E1-002586E2.00543DD2@ibm.com>
References: <20210527151039.GC8399@morsing.cc>
	<OF65FE85E0.F99DD819-ON002586E2.005411E1-002586E2.00543DD2@ibm.com>
Message-ID: <20210528071537.GE8399@morsing.cc>

On Thu, May 27, 2021 at 03:20:08PM +0000, Anderson Ferreira Nobre wrote:
>   Henrik,
>   ?
>   One way would integrate Scale with QRadar. If I'm not wrong, you can
>   configure QRadar to take a snapshot when it detects there's an attack
>   happening. The details you can take from here:
>   [1]https://www.redbooks.ibm.com/redpapers/pdfs/redp5560.pdf
>   [2]https://www.youtube.com/watch?v=Zyw84dvoFR8
>   ?

Hi,

Looking at the video (not read the document yet) I'm not sure QRadar is advanced enough to detect someone encrypting a storage pool from the SP server. It's a single file pretty much access 24x7, but I will look into it further, thanks.

Regards,
Henrik


From jonathan.buzzard at strath.ac.uk  Fri May 28 09:07:12 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Fri, 28 May 2021 09:07:12 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210528064635.GD8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
	<CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>
	<20210528064635.GD8399@morsing.cc>
Message-ID: <546408b3-0617-f293-8d15-c36f776f5bd8@strath.ac.uk>

On 28/05/2021 07:46, Henrik Morsing wrote:
>>
>> That might not make sense if GPFS is holding the SP backup data, but 
>> SP can do its own replication too - and could replicate using storage from a
>> second GPFS file system off-site.? Take snapshots of this second storage,
>> as well as SP database, and again manage with a second sysadmin team.
>>
> 
> Thanks all for some useful replies, something to take forward.
> 
> In this case, SP is using GPFS for storing backup data, this solution 
> was meant to replace the tape libraries completely.
> 

If your backup is for disaster recovery that's fine. If you expand your 
disaster to include ransom attacks then disk based backups are IMHO 
inadequate simply because they can be gone forever in the blink of an eye.

> We protect the storage pools cross-site, but our solutions are 
> identical, so if you hacked one, you have hacked both.
> 

Currently we use a home grown disk based system for the backup (home 
grown because it's cheap) however we are looking to augment it with tape 
because tape is firstly ransom attack resistant, second tape is "green" 
with a very low carbon footprint.

 From a TSM perspective backup goes to the disk run as a bunch of 
sequential access files like "tapes", and the copy pool will exists on 
tape. We get the benefit of having the backup on disk aka the short 
access times to files, with the protection offered by tape should we get 
hit by a ransom attack.

JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From macthev at gmail.com  Fri May 28 09:23:08 2021
From: macthev at gmail.com (macthev at gmail.com)
Date: Fri, 28 May 2021 18:23:08 +1000
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <326E4E15-014F-4A28-8EBB-1295CF899859@gmail.com>

Take a look at IAM nodes.

Sent from my iPhone

> On 28 May 2021, at 01:10, Henrik Morsing <henrik at morsing.cc> wrote:
> 
> ?
> Hi,
> 
> It struck me that switching a Spectrum Protect solution from tapes to a GPFS filesystem offers much less protection against ransom encryption should the SP server be compromised. Same goes really for compromising an ESS node itself, it is an awful lot of data that can be encrypted very quickly.
> 
> Is there anything that can protect the GPFS filesystem against this kind of attack?
> 
> Regards,
> Henrik
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From heinrich.billich at id.ethz.ch  Fri May 28 09:25:20 2021
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Fri, 28 May 2021 08:25:20 +0000
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to estimate
 remaining execution time
Message-ID: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>

 Hello,

 

I want to estimate how much longer a running 

 

mmrestripefs -R ?metadata-only ?qos maintenance

 

job will take to finish. We did switch from ?-m 1? to ?-m 2 ? and now run mmrestripefs to get the second copy of all metadata. I know that the % values in the output are useless, but now it looks like the number of inodes processed is of no use, too? The filesystem has about 1e9 allocated inodes, but the mmrestripefs reports up to? 7e9 inodes processed? The number of inodes reported increases heavily since it passed the number of allocated inodes???

 

?? 4.41 % complete on Fri May 28 09:22:05 2021? (7219160122 inodes with total??? 8985067 MB data processed)

 

 

Should I consider the ?MB data processed? instead ? will the job finish when all used metadata disk space is processed? 

 

I see little iops on the metadata disks and wonder if something is stuck, I?m not sure if it makes sense to wait any longer. But without an estimate on how much longer the job will take I hesitate to restart. A restart will need to scan all metadata again ? Metadata is on SSD or NVMe, disks iops never were limiting. I turned qos off on the filesystem, previously I restricted the iops. But I don?t see any increase in iops.

 

Thank you. I know this is an old issue, but still I would strongly welcome any advice or comment.

 

Cheers,

 

Heiner

 

I?m aware of RFE ID:150409 mmrestripefs % complete metric is near useless. I?m not sure if it covers the metadata-only case, too.

 

Inode Information

-----------------

Total number of used inodes in all Inode spaces:????????? 735005692

Total number of free inodes in all Inode spaces:????????? 335462660

Total number of allocated inodes in all Inode spaces:??? 1070468352

Total of Maximum number of inodes in all Inode spaces:?? 1223782912

 

Mmrestripefs ?output:

 

START Mon May 24 20:27:25 CEST 2021

Scanning file system metadata, phase 1 ...

 100 % complete on Mon May 24 21:49:27 2021

Scan completed successfully.

Scanning file system metadata, phase 2 ...

 100 % complete on Mon May 24 21:49:28 2021

Scanning file system metadata for data storage pool

? 90 % complete on Mon May 24 21:49:34 2021

 100 % complete on Mon May 24 21:49:35 2021

Scanning file system metadata for Capacity storage pool

? 49 % complete on Mon May 24 21:49:41 2021

 100 % complete on Mon May 24 21:49:46 2021

Scan completed successfully.

Scanning file system metadata, phase 3 ...

Scan completed successfully.

Scanning file system metadata, phase 4 ...

 100 % complete on Mon May 24 21:49:48 2021

Scan completed successfully.

Scanning file system metadata, phase 5 ...

 100 % complete on Mon May 24 21:49:52 2021

Scan completed successfully.

Scanning user file metadata ...

?? 0.01 % complete on Mon May 24 21:50:13 2021? (??? 613108 inodes with total????? 76693 MB data processed)

?? 0.01 % complete on Mon May 24 21:50:33 2021? (?? 1090048 inodes with total????? 80495 MB data processed)

?? 0.01 % complete on Mon May 24 21:50:53 2021? (?? 1591808 inodes with total????? 84576 MB data processed)

?

? 3.97 % complete on Thu May 27 22:01:02 2021? (1048352497 inodes with total??? 8480467 MB data processed)

?? 3.99 % complete on Thu May 27 22:30:18 2021? (1050254855 inodes with total??? 8495969 MB data processed)

?? 4.01 % complete on Thu May 27 22:59:39 2021? (1052304294 inodes with total??? 8512683 MB data processed)

?? 4.03 % complete on Thu May 27 23:29:11 2021? (1055390220 inodes with total??? 8537615 MB data processed)

?? 4.05 % complete on Thu May 27 23:58:58 2021? (1059333989 inodes with total??? 8568871 MB data processed)

?? 4.07 % complete on Fri May 28 00:28:48 2021? (1064728403 inodes with total??? 8611605 MB data processed)

?? 4.09 % complete on Fri May 28 00:58:50 2021? (1067749260 inodes with total??? 8636120 MB data processed). <<< approximately number of allocated inodes

 

?? 4.11 % complete on Fri May 28 01:29:00 2021? (1488665433 inodes with total??? 8661588 MB data processed) <<< whats going??

?? 4.13 % complete on Fri May 28 01:59:23 2021? (1851124480 inodes with total??? 8682324 MB data processed)

?? 4.15 % complete on Fri May 28 02:29:55 2021? (1885948840 inodes with total??? 8700082 MB data processed)

?? 4.17 % complete on Fri May 28 03:00:38 2021? (2604503808 inodes with total?? ?8724069 MB data processed)

?? 4.19 % complete on Fri May 28 03:31:38 2021? (2877196260 inodes with total??? 8746504 MB data processed)

?? 4.21 % complete on Fri May 28 04:02:38 2021? (2933166080 inodes with total??? 8762555 MB data processed)

?? 4.23 % complete on Fri May 28 04:33:48 2021? (2956295936 inodes with total??? 8782298 MB data processed)

?? 4.25 % complete on Fri May 28 05:05:09 2021? (3628799151 inodes with total??? 8802452 MB data processed)

?? 4.27 % complete on Fri May 28 05:36:40 2021? (3970093965 inodes with total??? 8823885 MB data processed)

?? 4.29 % complete on Fri May 28 06:08:20 2021? (4012553472 inodes with total??? 8841407 MB data processed)

?? 4.31 % complete on Fri May 28 06:40:11 2021? (4029545087 inodes with total??? 8858676 MB data processed)

?? 4.33 % complete on Fri May 28 07:12:11 2021? (6080613874 inodes with total??? 8889395 MB data processed)

?? 4.35 % complete on Fri May 28 07:44:21 2021? (6146937531 inodes with total??? 8907253 MB data processed)

?? 4.37 % complete on Fri May 28 08:16:45 2021? (6167408718 inodes with total??? 8925236 MB data processed)

?? 4.39 % complete on Fri May 28 08:49:18 2021? (7151592448 inodes with total??? 8968126 MB data processed)

?? 4.41 % complete on Fri May 28 09:22:05 2021? (7219160122 inodes with total??? 8985067 MB data processed)

 

 

# mmdf fsxxxx -m --block-size M

disk??????????????? disk size? failure holds??? holds?????????? free in MB????????? free in MB

name??????????????????? in MB??? group metadata data??????? in full blocks??????? in fragments

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 32.20 TB)

fsxxxx_rg07b_1??????? 2861295? ??????3 yes????? no????????? 1684452 ( 59%)???????? 35886 ( 1%). << ?1?140?957MB used ? old nsd

fsxxxx_rg07a_1??????? 2861295??????? 3 yes????? no????????? 1685002 ( 59%)???????? 35883 ( 1%

fsxxxx_rg03b_1??????? 2861295??????? 3 yes????? no????????? 1681515 ( 59%)???????? 35627 ( 1%)

fsxxxx_rg03a_1??????? 2861295??????? 3 yes????? no????????? 1680859 ( 59%)???????? 35651 ( 1%)

 

RG003LG004VS015?????? 2046239?????? 12 yes????? no?????????? 904369 ( 44%)?????????? 114 ( 0%) << 1?141?756MB used ? newly added nsd

RG003LG003VS015?????? 2046239?????? 12 yes????? no?????????? 904291 ( 44%)?????????? 118 ( 0%)

RG003LG002VS015?????? 2046239?????? 12 yes????? no?????????? 903632 ( 44%)?????????? 114 ( 0%)

RG003LG001VS015?????? 2046239?????? 12 yes????? no?????????? 903247 ( 44%)?????????? 115 ( 0%)

??????????????? -------------??????? ?????????????????-------------------- -------------------

(pool total)???????? 19630136????????????????????????????? 10347367 ( 53%)??????? 143503 ( 1%)

 

We run spectrum scale 5.0.5.4/5.0.5.5 on Power. 

 

Filesystem:

 

-f???????????????? 32768??????????????????? Minimum fragment (subblock) size in bytes

 -i???????????????? 4096???????????????????? Inode size in bytes

 -I???????????????? 32768??????????????????? Indirect block size in bytes

 -m???????????????? 2??????????????????????? Default number of metadata replicas

 -M???????????????? 2??????????????????????? Maximum number of metadata replicas

 -r???????????????? 1??????????????????????? Default number of data replicas

 -R???????????????? 2??????????????????????? Maximum number of data replicas

 -j???????????????? scatter????????????????? Block allocation type

 

-V???????????????? 23.00 (5.0.5.0)????????? Current file system version

???????????????? ???15.01 (4.2.0.0)????????? Original file system version

 

-L???????????????? 33554432???????????????? Logfile size

 

--subblocks-per-full-block 32?????????????? Number of subblocks per full block

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/0f901305/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5254 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/0f901305/attachment.bin>

From heinrich.billich at id.ethz.ch  Fri May 28 09:55:53 2021
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Fri, 28 May 2021 08:55:53 +0000
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to
 estimate remaining execution time
In-Reply-To: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>
References: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>
Message-ID: <A54D4506-01BF-4E0D-8AA6-194BA3CA89B6@id.ethz.ch>

Hello,

 

I just noticed:

 

Maybe mmrestripefs does some extra processing on snapshots? The output looks much more as expected on filesystems with no snapshots present, both number of inodes and MB data processed allow to estimate the remaining runtime. ?Unfortunately all our large production filesystems use snapshots ?

 

Cheers,

 

Heiner

 

 

?? 1.23 % complete on Tue May 18 21:07:10 2021? (? 37981765 inodes with total???? 328617 MB data processed)

?? 1.25 % complete on Tue May 18 21:17:44 2021? (? 39439584 inodes with total???? 341032 MB data processed)

 100.00 % complete on Tue May 18 21:22:44 2021? (? 41312000 inodes with total???? 356088 MB data processed).??? <<<< # of inodes matches # of allocated inodes, MB processed matches used metadata disk space

 

Scan completed successfully.

# mmdf fsxxx -F

Inode Information

-----------------

Total number of used inodes in all Inode spaces:?????????? 29007227

Total number of free inodes in all Inode spaces:?????????? 12304773

Total number of allocated inodes in all Inode spaces:????? 41312000????? <<<<<<<<<< allocated inodes

Total of Maximum number of inodes in all Inode spaces:???? 87323648

 

 

]# mmdf fsxxx -m --block-size M

disk??????????????? disk size? failure holds??? holds?????????? free in MB????????? free in MB

name??????????????????? in MB??? group metadata data??????? in full blocks??????? in fragments

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 4.13 TB)

??????????????? -------------???????????????????????? -------------------- -------------------

(pool total)????????? 2425152?????????????????????????????? 2067720 ( 85%)????????? 1342 ( 0%).? <<<<<< 356190MB used

 

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Billich Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
Reply to: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Friday, 28 May 2021 at 10:25
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to estimate remaining execution time

 

Hello,

 

I want to estimate how much longer a running 

 

mmrestripefs -R ?metadata-only ?qos maintenance

 

job will take to finish. We did switch from ?-m 1? to ?-m 2 ? and now run mmrestripefs to get the second copy of all metadata. I know that the % values in the output are useless, but now it looks like the number of inodes processed is of no use, too? The filesystem has about 1e9 allocated inodes, but the mmrestripefs reports up to  7e9 inodes processed? The number of inodes reported increases heavily since it passed the number of allocated inodes???

 

   4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with total    8985067 MB data processed)

 

 

Should I consider the ?MB data processed? instead ? will the job finish when all used metadata disk space is processed? 

 

I see little iops on the metadata disks and wonder if something is stuck, I?m not sure if it makes sense to wait any longer. But without an estimate on how much longer the job will take I hesitate to restart. A restart will need to scan all metadata again ? Metadata is on SSD or NVMe, disks iops never were limiting. I turned qos off on the filesystem, previously I restricted the iops. But I don?t see any increase in iops.

 

Thank you. I know this is an old issue, but still I would strongly welcome any advice or comment.

 

Cheers,

 

Heiner

 

I?m aware of RFE ID:150409 mmrestripefs % complete metric is near useless. I?m not sure if it covers the metadata-only case, too.

 

Inode Information

-----------------

Total number of used inodes in all Inode spaces:          735005692

Total number of free inodes in all Inode spaces:          335462660

Total number of allocated inodes in all Inode spaces:    1070468352

Total of Maximum number of inodes in all Inode spaces:   1223782912

 

Mmrestripefs  output:

 

START Mon May 24 20:27:25 CEST 2021

Scanning file system metadata, phase 1 ...

100 % complete on Mon May 24 21:49:27 2021

Scan completed successfully.

Scanning file system metadata, phase 2 ...

100 % complete on Mon May 24 21:49:28 2021

Scanning file system metadata for data storage pool

  90 % complete on Mon May 24 21:49:34 2021

100 % complete on Mon May 24 21:49:35 2021

Scanning file system metadata for Capacity storage pool

  49 % complete on Mon May 24 21:49:41 2021

100 % complete on Mon May 24 21:49:46 2021

Scan completed successfully.

Scanning file system metadata, phase 3 ...

Scan completed successfully.

Scanning file system metadata, phase 4 ...

100 % complete on Mon May 24 21:49:48 2021

Scan completed successfully.

Scanning file system metadata, phase 5 ...

100 % complete on Mon May 24 21:49:52 2021

Scan completed successfully.

Scanning user file metadata ...

   0.01 % complete on Mon May 24 21:50:13 2021  (    613108 inodes with total      76693 MB data processed)

   0.01 % complete on Mon May 24 21:50:33 2021  (   1090048 inodes with total      80495 MB data processed)

   0.01 % complete on Mon May 24 21:50:53 2021  (   1591808 inodes with total      84576 MB data processed)

?

  3.97 % complete on Thu May 27 22:01:02 2021  (1048352497 inodes with total    8480467 MB data processed)

   3.99 % complete on Thu May 27 22:30:18 2021  (1050254855 inodes with total    8495969 MB data processed)

   4.01 % complete on Thu May 27 22:59:39 2021  (1052304294 inodes with total    8512683 MB data processed)

   4.03 % complete on Thu May 27 23:29:11 2021  (1055390220 inodes with total    8537615 MB data processed)

   4.05 % complete on Thu May 27 23:58:58 2021  (1059333989 inodes with total    8568871 MB data processed)

   4.07 % complete on Fri May 28 00:28:48 2021  (1064728403 inodes with total    8611605 MB data processed)

   4.09 % complete on Fri May 28 00:58:50 2021  (1067749260 inodes with total    8636120 MB data processed). <<< approximately number of allocated inodes

 

   4.11 % complete on Fri May 28 01:29:00 2021  (1488665433 inodes with total    8661588 MB data processed) <<< whats going??

   4.13 % complete on Fri May 28 01:59:23 2021  (1851124480 inodes with total    8682324 MB data processed)

   4.15 % complete on Fri May 28 02:29:55 2021  (1885948840 inodes with total    8700082 MB data processed)

   4.17 % complete on Fri May 28 03:00:38 2021  (2604503808 inodes with total    8724069 MB data processed)

   4.19 % complete on Fri May 28 03:31:38 2021  (2877196260 inodes with total    8746504 MB data processed)

   4.21 % complete on Fri May 28 04:02:38 2021  (2933166080 inodes with total    8762555 MB data processed)

   4.23 % complete on Fri May 28 04:33:48 2021  (2956295936 inodes with total    8782298 MB data processed)

   4.25 % complete on Fri May 28 05:05:09 2021  (3628799151 inodes with total    8802452 MB data processed)

   4.27 % complete on Fri May 28 05:36:40 2021  (3970093965 inodes with total    8823885 MB data processed)

   4.29 % complete on Fri May 28 06:08:20 2021  (4012553472 inodes with total    8841407 MB data processed)

   4.31 % complete on Fri May 28 06:40:11 2021  (4029545087 inodes with total    8858676 MB data processed)

   4.33 % complete on Fri May 28 07:12:11 2021  (6080613874 inodes with total    8889395 MB data processed)

   4.35 % complete on Fri May 28 07:44:21 2021  (6146937531 inodes with total    8907253 MB data processed)

   4.37 % complete on Fri May 28 08:16:45 2021  (6167408718 inodes with total    8925236 MB data processed)

   4.39 % complete on Fri May 28 08:49:18 2021  (7151592448 inodes with total    8968126 MB data processed)

   4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with total    8985067 MB data processed)

 

 

# mmdf fsxxxx -m --block-size M

disk                disk size  failure holds    holds           free in MB          free in MB

name                    in MB    group metadata data        in full blocks        in fragments

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 32.20 TB)

fsxxxx_rg07b_1        2861295        3 yes      no          1684452 ( 59%)         35886 ( 1%). <<  1?140?957MB used ? old nsd

fsxxxx_rg07a_1        2861295        3 yes      no          1685002 ( 59%)         35883 ( 1%

fsxxxx_rg03b_1        2861295        3 yes      no          1681515 ( 59%)         35627 ( 1%)

fsxxxx_rg03a_1        2861295        3 yes      no          1680859 ( 59%)         35651 ( 1%)

 

RG003LG004VS015       2046239       12 yes      no           904369 ( 44%)           114 ( 0%) << 1?141?756MB used ? newly added nsd

RG003LG003VS015       2046239       12 yes      no           904291 ( 44%)           118 ( 0%)

RG003LG002VS015       2046239       12 yes      no           903632 ( 44%)           114 ( 0%)

RG003LG001VS015       2046239       12 yes      no           903247 ( 44%)           115 ( 0%)

                -------------                         -------------------- -------------------

(pool total)         19630136                              10347367 ( 53%)        143503 ( 1%)

 

We run spectrum scale 5.0.5.4/5.0.5.5 on Power. 

 

Filesystem:

 

-f                 32768                    Minimum fragment (subblock) size in bytes

-i                 4096                     Inode size in bytes

-I                 32768                    Indirect block size in bytes

-m                 2                        Default number of metadata replicas

-M                 2                        Maximum number of metadata replicas

-r                 1                        Default number of data replicas

-R                 2                        Maximum number of data replicas

-j                 scatter                  Block allocation type

 

-V                 23.00 (5.0.5.0)          Current file system version

                    15.01 (4.2.0.0)          Original file system version

 

-L                 33554432                 Logfile size

 

--subblocks-per-full-block 32               Number of subblocks per full block

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/ec0c5063/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5254 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/ec0c5063/attachment.bin>

From oluwasijibomi.saula at ndsu.edu  Fri May 28 16:57:52 2021
From: oluwasijibomi.saula at ndsu.edu (Saula, Oluwasijibomi)
Date: Fri, 28 May 2021 15:57:52 +0000
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
Message-ID: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>

Hi Folks,

So, we are experiencing some very long IO waiters in our GPFS cluster:


#  mmdiag --waiters


=== mmdiag: waiters ===

Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for I/O completion

Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for I/O completion

Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for I/O completion

Waiting 15.2013 sec since 10:41:03, monitored, thread 22175

However, GPFS support is pointing to our IBM Storwize V5030 disk system as the source of latency. Unfortunately, we don't have paid support for the system so we are polling for anyone who might be able to assist.

Does anyone by chance have any experience with IBM Storwize V5030 or possess a problem determination guide for the V5030?

We've briefly reviewed the V5030 management portal, but we still haven't identified a cause for the increased latencies (i.e. read ~129ms, write ~198ms).

Granted, we have some heavy client workloads, yet we seem to experience this drastic drop in performance every couple of months, probably exacerbated by heavy IO demands.

Any assistance would be much appreciated.



Thanks,


Oluwasijibomi (Siji) Saula

HPC Systems Administrator  /  Information Technology



Research 2 Building 220B / Fargo ND 58108-6050

p: 701.231.7749 / www.ndsu.edu<http://www.ndsu.edu/>



[cid:image001.gif at 01D57DE0.91C300C0]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/f778d057/attachment.htm>

From erich at uw.edu  Fri May 28 17:25:49 2021
From: erich at uw.edu (Eric Horst)
Date: Fri, 28 May 2021 09:25:49 -0700
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to
 estimate remaining execution time
In-Reply-To: <A54D4506-01BF-4E0D-8AA6-194BA3CA89B6@id.ethz.ch>
References: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>
	<A54D4506-01BF-4E0D-8AA6-194BA3CA89B6@id.ethz.ch>
Message-ID: <CAF-To-2FFJHvt7OzSz5gfMur9bKtpG7ixLJtgT6bvLToRRO+TQ@mail.gmail.com>

Yes Heiner, my experience is that the inode count in those operations is:
inodes * snapshots = total. I observed that as it starts processing the
snapshot inodes it moves faster as inode usage in snapshots is more sparse.

-Eric

On Fri, May 28, 2021 at 1:56 AM Billich Heinrich Rainer (ID SD) <
heinrich.billich at id.ethz.ch> wrote:

> Hello,
>
>
>
> I just noticed:
>
>
>
> Maybe mmrestripefs does some extra processing on snapshots? The output
> looks much more as expected on filesystems with no snapshots present, both
> number of inodes and MB data processed allow to estimate the remaining
> runtime.  Unfortunately all our large production filesystems use snapshots ?
>
>
>
> Cheers,
>
>
>
> Heiner
>
>
>
>
>
>    1.23 % complete on Tue May 18 21:07:10 2021  (  37981765 inodes with
> total     328617 MB data processed)
>
>    1.25 % complete on Tue May 18 21:17:44 2021  (  39439584 inodes with
> total     341032 MB data processed)
>
> 100.00 % complete on Tue May 18 21:22:44 2021  (  41312000 inodes with
> total     356088 MB data processed).    <<<< # of inodes matches # of
> allocated inodes, MB processed matches used metadata disk space
>
>
>
> Scan completed successfully.
>
> # mmdf fsxxx -F
>
> Inode Information
>
> -----------------
>
> Total number of used inodes in all Inode spaces:           29007227
>
> Total number of free inodes in all Inode spaces:           12304773
>
> Total number of allocated inodes in all Inode spaces:      41312000
> <<<<<<<<<< allocated inodes
>
> Total of Maximum number of inodes in all Inode spaces:     87323648
>
>
>
>
>
> ]# mmdf fsxxx -m --block-size M
>
> disk                disk size  failure holds    holds           free in
> MB          free in MB
>
> name                    in MB    group metadata data        in full
> blocks        in fragments
>
> --------------- ------------- -------- -------- ----- --------------------
> -------------------
>
> Disks in storage pool: system (Maximum disk size allowed is 4.13 TB)
>
>                 -------------                         --------------------
> -------------------
>
> (pool total)          2425152                               2067720 (
> 85%)          1342 ( 0%).  <<<<<< 356190MB used
>
>
>
> *From: *<gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Billich
> Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
> *Reply to: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Date: *Friday, 28 May 2021 at 10:25
> *To: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Subject: *[gpfsug-discuss] Mmrestripefs -R --metadat-only - how to
> estimate remaining execution time
>
>
>
> Hello,
>
>
>
> I want to estimate how much longer a running
>
>
>
> mmrestripefs -R ?metadata-only ?qos maintenance
>
>
>
> job will take to finish. We did switch from ?-m 1? to ?-m 2 ? and now run
> mmrestripefs to get the second copy of all metadata. I know that the %
> values in the output are useless, but now it looks like the number of
> inodes processed is of no use, too? The filesystem has about 1e9 allocated
> inodes, but the mmrestripefs reports up to  7e9 inodes processed? The
> number of inodes reported increases heavily since it passed the number of
> allocated inodes???
>
>
>
>    4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with
> total    8985067 MB data processed)
>
>
>
>
>
> Should I consider the ?MB data processed? instead ? will the job finish
> when all used metadata disk space is processed?
>
>
>
> I see little iops on the metadata disks and wonder if something is stuck,
> I?m not sure if it makes sense to wait any longer. But without an estimate
> on how much longer the job will take I hesitate to restart. A restart will
> need to scan all metadata again ? Metadata is on SSD or NVMe, disks iops
> never were limiting. I turned qos off on the filesystem, previously I
> restricted the iops. But I don?t see any increase in iops.
>
>
>
> Thank you. I know this is an old issue, but still I would strongly welcome
> any advice or comment.
>
>
>
> Cheers,
>
>
>
> Heiner
>
>
>
> I?m aware of RFE ID:150409 mmrestripefs % complete metric is near useless.
> I?m not sure if it covers the metadata-only case, too.
>
>
>
> Inode Information
>
> -----------------
>
> Total number of used inodes in all Inode spaces:          735005692
>
> Total number of free inodes in all Inode spaces:          335462660
>
> Total number of allocated inodes in all Inode spaces:    1070468352
>
> Total of Maximum number of inodes in all Inode spaces:   1223782912
>
>
>
> Mmrestripefs  output:
>
>
>
> START Mon May 24 20:27:25 CEST 2021
>
> Scanning file system metadata, phase 1 ...
>
> 100 % complete on Mon May 24 21:49:27 2021
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 2 ...
>
> 100 % complete on Mon May 24 21:49:28 2021
>
> Scanning file system metadata for data storage pool
>
>   90 % complete on Mon May 24 21:49:34 2021
>
> 100 % complete on Mon May 24 21:49:35 2021
>
> Scanning file system metadata for Capacity storage pool
>
>   49 % complete on Mon May 24 21:49:41 2021
>
> 100 % complete on Mon May 24 21:49:46 2021
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 3 ...
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 4 ...
>
> 100 % complete on Mon May 24 21:49:48 2021
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 5 ...
>
> 100 % complete on Mon May 24 21:49:52 2021
>
> Scan completed successfully.
>
> Scanning user file metadata ...
>
>    0.01 % complete on Mon May 24 21:50:13 2021  (    613108 inodes with
> total      76693 MB data processed)
>
>    0.01 % complete on Mon May 24 21:50:33 2021  (   1090048 inodes with
> total      80495 MB data processed)
>
>    0.01 % complete on Mon May 24 21:50:53 2021  (   1591808 inodes with
> total      84576 MB data processed)
>
> ?
>
>   3.97 % complete on Thu May 27 22:01:02 2021  (1048352497 inodes with
> total    8480467 MB data processed)
>
>    3.99 % complete on Thu May 27 22:30:18 2021  (1050254855 inodes with
> total    8495969 MB data processed)
>
>    4.01 % complete on Thu May 27 22:59:39 2021  (1052304294 inodes with
> total    8512683 MB data processed)
>
>    4.03 % complete on Thu May 27 23:29:11 2021  (1055390220 inodes with
> total    8537615 MB data processed)
>
>    4.05 % complete on Thu May 27 23:58:58 2021  (1059333989 inodes with
> total    8568871 MB data processed)
>
>    4.07 % complete on Fri May 28 00:28:48 2021  (1064728403 inodes with
> total    8611605 MB data processed)
>
>    4.09 % complete on Fri May 28 00:58:50 2021  (1067749260 inodes with
> total    8636120 MB data processed). <<< approximately number of allocated
> inodes
>
>
>
>    4.11 % complete on Fri May 28 01:29:00 2021  (1488665433 inodes with
> total    8661588 MB data processed) <<< whats going??
>
>    4.13 % complete on Fri May 28 01:59:23 2021  (1851124480 inodes with
> total    8682324 MB data processed)
>
>    4.15 % complete on Fri May 28 02:29:55 2021  (1885948840 inodes with
> total    8700082 MB data processed)
>
>    4.17 % complete on Fri May 28 03:00:38 2021  (2604503808 inodes with
> total    8724069 MB data processed)
>
>    4.19 % complete on Fri May 28 03:31:38 2021  (2877196260 inodes with
> total    8746504 MB data processed)
>
>    4.21 % complete on Fri May 28 04:02:38 2021  (2933166080 inodes with
> total    8762555 MB data processed)
>
>    4.23 % complete on Fri May 28 04:33:48 2021  (2956295936 inodes with
> total    8782298 MB data processed)
>
>    4.25 % complete on Fri May 28 05:05:09 2021  (3628799151 inodes with
> total    8802452 MB data processed)
>
>    4.27 % complete on Fri May 28 05:36:40 2021  (3970093965 inodes with
> total    8823885 MB data processed)
>
>    4.29 % complete on Fri May 28 06:08:20 2021  (4012553472 inodes with
> total    8841407 MB data processed)
>
>    4.31 % complete on Fri May 28 06:40:11 2021  (4029545087 inodes with
> total    8858676 MB data processed)
>
>    4.33 % complete on Fri May 28 07:12:11 2021  (6080613874 inodes with
> total    8889395 MB data processed)
>
>    4.35 % complete on Fri May 28 07:44:21 2021  (6146937531 inodes with
> total    8907253 MB data processed)
>
>    4.37 % complete on Fri May 28 08:16:45 2021  (6167408718 inodes with
> total    8925236 MB data processed)
>
>    4.39 % complete on Fri May 28 08:49:18 2021  (7151592448 inodes with
> total    8968126 MB data processed)
>
>    4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with
> total    8985067 MB data processed)
>
>
>
>
>
> # mmdf fsxxxx -m --block-size M
>
> disk                disk size  failure holds    holds           free in
> MB          free in MB
>
> name                    in MB    group metadata data        in full
> blocks        in fragments
>
> --------------- ------------- -------- -------- ----- --------------------
> -------------------
>
> Disks in storage pool: system (Maximum disk size allowed is 32.20 TB)
>
> fsxxxx_rg07b_1        2861295        3 yes      no          1684452 (
> 59%)         35886 ( 1%). <<  1?140?957MB used ? old nsd
>
> fsxxxx_rg07a_1        2861295        3 yes      no          1685002 (
> 59%)         35883 ( 1%
>
> fsxxxx_rg03b_1        2861295        3 yes      no          1681515 (
> 59%)         35627 ( 1%)
>
> fsxxxx_rg03a_1        2861295        3 yes      no          1680859 (
> 59%)         35651 ( 1%)
>
>
>
> RG003LG004VS015       2046239       12 yes      no           904369 (
> 44%)           114 ( 0%) << 1?141?756MB used ? newly added nsd
>
> RG003LG003VS015       2046239       12 yes      no           904291 (
> 44%)           118 ( 0%)
>
> RG003LG002VS015       2046239       12 yes      no           903632 (
> 44%)           114 ( 0%)
>
> RG003LG001VS015       2046239       12 yes      no           903247 (
> 44%)           115 ( 0%)
>
>                 -------------                         --------------------
> -------------------
>
> (pool total)         19630136                              10347367 (
> 53%)        143503 ( 1%)
>
>
>
> We run spectrum scale 5.0.5.4/5.0.5.5 on Power.
>
>
>
> Filesystem:
>
>
>
> -f                 32768                    Minimum fragment (subblock)
> size in bytes
>
> -i                 4096                     Inode size in bytes
>
> -I                 32768                    Indirect block size in bytes
>
> -m                 2                        Default number of metadata
> replicas
>
> -M                 2                        Maximum number of metadata
> replicas
>
> -r                 1                        Default number of data replicas
>
> -R                 2                        Maximum number of data replicas
>
> -j                 scatter                  Block allocation type
>
>
>
> -V                 23.00 (5.0.5.0)          Current file system version
>
>                     15.01 (4.2.0.0)          Original file system version
>
>
>
> -L                 33554432                 Logfile size
>
>
>
> --subblocks-per-full-block 32               Number of subblocks per full
> block
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/f46d7a39/attachment.htm>

From janfrode at tanso.net  Fri May 28 18:50:21 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 28 May 2021 19:50:21 +0200
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>
References: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>
Message-ID: <CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>

One thing to check: Storwize/SVC code will *always* guess wrong on
prefetching for GPFS. You can see this with having a lot higher read data
throughput on mdisk vs. on on vdisks in the webui. To fix it, disable
cache_prefetch with "chsystem -cache_prefetch off".

This being a global setting, you probably only should set it if the system
is only used for GPFS.


   -jf

On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
oluwasijibomi.saula at ndsu.edu> wrote:

> Hi Folks,
>
> So, we are experiencing some very long IO waiters in our GPFS cluster:
>
> #  mmdiag --waiters
>
>
> === mmdiag: waiters ===
>
> Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for
> I/O completion
>
> Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for
> I/O completion
>
> Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for
> I/O completion
>
> Waiting 15.2013 sec since 10:41:03, monitored, thread 22175
>
> However, GPFS support is pointing to our IBM Storwize V5030 disk system
> as the source of latency. Unfortunately, we don't have paid support for the
> system so we are polling for anyone who might be able to assist.
>
> Does anyone by chance have any experience with IBM Storwize V5030 or
> possess a problem determination guide for the V5030?
>
> We've briefly reviewed the V5030 management portal, but we still haven't
> identified a cause for the increased latencies (i.e. read ~129ms, write
> ~198ms).
>
> Granted, we have some heavy client workloads, yet we seem to experience
> this drastic drop in performance every couple of months, probably
> exacerbated by heavy IO demands.
>
> Any assistance would be much appreciated.
>
>
> Thanks,
>
> *Oluwasijibomi (Siji) Saula*
>
> HPC Systems Administrator  /  Information Technology
>
>
>
> Research 2 Building 220B / Fargo ND 58108-6050
>
> p: 701.231.7749 / www.ndsu.edu
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/832c7e75/attachment.htm>

From UWEFALKE at de.ibm.com  Fri May 28 21:04:19 2021
From: UWEFALKE at de.ibm.com (Uwe Falke)
Date: Fri, 28 May 2021 22:04:19 +0200
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>
References: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>
	<CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>
Message-ID: <OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>

Hi, odd prefetch strategy would affect read performance, but write latency 
is claimed to be even worse ...
Have you simply checked what the actual IO performance of the v5k box 
under that load is and how it compares to its nominal performance and that 
of its disks?
how is the storage organised? how many LUNs/NSDs, what RAID code (V5k 
cannot do declustered RAID, can it?), any thin provisioning or other 
gimmicks in the game?
what IO sizes ?
tons of things to look at. 

Mit freundlichen Gr??en / Kind regards

Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation 
Services
+49 175 575 2877 Mobile
Rochlitzer Str. 19, 09111 Chemnitz, Germany
uwefalke at de.ibm.com

IBM Services

IBM Data Privacy Statement

IBM Deutschland Business & Technology Services GmbH
Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122



From:   Jan-Frode Myklebust <janfrode at tanso.net>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   28/05/2021 19:50
Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM 
Storwize V5030
Sent by:        gpfsug-discuss-bounces at spectrumscale.org




One thing to check: Storwize/SVC code will *always* guess wrong on 
prefetching for GPFS. You can see this with having a lot higher read data 
throughput on mdisk vs. on on vdisks in the webui. To fix it, disable 
cache_prefetch with "chsystem -cache_prefetch off".

This being a global setting, you probably only should set it if the system 
is only used for GPFS. 


   -jf

On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
oluwasijibomi.saula at ndsu.edu> wrote:
Hi Folks,

So, we are experiencing some very long IO waiters in our GPFS cluster:

#  mmdiag --waiters 

=== mmdiag: waiters ===
Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for 
I/O completion
Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for 
I/O completion
Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for 
I/O completion
Waiting 15.2013 sec since 10:41:03, monitored, thread 22175 

However, GPFS support is pointing to our IBM Storwize V5030 disk system as 
the source of latency. Unfortunately, we don't have paid support for the 
system so we are polling for anyone who might be able to assist.

Does anyone by chance have any experience with IBM Storwize V5030 or 
possess a problem determination guide for the V5030?

We've briefly reviewed the V5030 management portal, but we still haven't 
identified a cause for the increased latencies (i.e. read ~129ms, write 
~198ms). 

Granted, we have some heavy client workloads, yet we seem to experience 
this drastic drop in performance every couple of months, probably 
exacerbated by heavy IO demands.

Any assistance would be much appreciated.


Thanks,

Oluwasijibomi (Siji) Saula
HPC Systems Administrator  /  Information Technology
 
Research 2 Building 220B / Fargo ND 58108-6050
p: 701.231.7749 / www.ndsu.edu
 



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 







From UWEFALKE at de.ibm.com  Fri May 28 21:16:12 2021
From: UWEFALKE at de.ibm.com (Uwe Falke)
Date: Fri, 28 May 2021 22:16:12 +0200
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>
References: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com><CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>
	<OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>
Message-ID: <OF7233E8BC.78902F9C-ONC12586E3.006ED69A-C12586E3.006F5924@notes.na.collabserv.com>

You say you see that every few months: does it mean with about the same 
load sometimes the system knocks out and sometimes it behaves ok?
Have you checked the v5k event log, if there is anything going on (write 
performance may suffer if write cache is off which might happen if the 
buffer batteries are low on charge - but again that does not directly 
explain the high read latency). 
Are the latencies you gave derived from GPFS, from the OS, or from the 
V5k?

Mit freundlichen Gr??en / Kind regards

Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation 
Services
+49 175 575 2877 Mobile
Rochlitzer Str. 19, 09111 Chemnitz, Germany
uwefalke at de.ibm.com

IBM Services

IBM Data Privacy Statement

IBM Deutschland Business & Technology Services GmbH
Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122



From:   "Uwe Falke" <UWEFALKE at de.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   28/05/2021 22:04
Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM 
Storwize V5030
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi, odd prefetch strategy would affect read performance, but write latency 

is claimed to be even worse ...
Have you simply checked what the actual IO performance of the v5k box 
under that load is and how it compares to its nominal performance and that 

of its disks?
how is the storage organised? how many LUNs/NSDs, what RAID code (V5k 
cannot do declustered RAID, can it?), any thin provisioning or other 
gimmicks in the game?
what IO sizes ?
tons of things to look at. 

Mit freundlichen Gr??en / Kind regards

Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation 
Services
+49 175 575 2877 Mobile
Rochlitzer Str. 19, 09111 Chemnitz, Germany
uwefalke at de.ibm.com

IBM Services

IBM Data Privacy Statement

IBM Deutschland Business & Technology Services GmbH
Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122



From:   Jan-Frode Myklebust <janfrode at tanso.net>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   28/05/2021 19:50
Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM 
Storwize V5030
Sent by:        gpfsug-discuss-bounces at spectrumscale.org




One thing to check: Storwize/SVC code will *always* guess wrong on 
prefetching for GPFS. You can see this with having a lot higher read data 
throughput on mdisk vs. on on vdisks in the webui. To fix it, disable 
cache_prefetch with "chsystem -cache_prefetch off".

This being a global setting, you probably only should set it if the system 

is only used for GPFS. 


   -jf

On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
oluwasijibomi.saula at ndsu.edu> wrote:
Hi Folks,

So, we are experiencing some very long IO waiters in our GPFS cluster:

#  mmdiag --waiters 

=== mmdiag: waiters ===
Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for 

I/O completion
Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for 

I/O completion
Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for 

I/O completion
Waiting 15.2013 sec since 10:41:03, monitored, thread 22175 

However, GPFS support is pointing to our IBM Storwize V5030 disk system as 

the source of latency. Unfortunately, we don't have paid support for the 
system so we are polling for anyone who might be able to assist.

Does anyone by chance have any experience with IBM Storwize V5030 or 
possess a problem determination guide for the V5030?

We've briefly reviewed the V5030 management portal, but we still haven't 
identified a cause for the increased latencies (i.e. read ~129ms, write 
~198ms). 

Granted, we have some heavy client workloads, yet we seem to experience 
this drastic drop in performance every couple of months, probably 
exacerbated by heavy IO demands.

Any assistance would be much appreciated.


Thanks,

Oluwasijibomi (Siji) Saula
HPC Systems Administrator  /  Information Technology
 
Research 2 Building 220B / Fargo ND 58108-6050
p: 701.231.7749 / www.ndsu.edu
 



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 





_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 








From abeattie at au1.ibm.com  Fri May 28 22:27:12 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Fri, 28 May 2021 21:27:12 +0000
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>
Message-ID: <OF09635240.267CCCEC-ON002586E3.0075D8B6-1622237232907@notes.na.collabserv.com>



Hi Oluwasijibomi,

If you set up a Storage Insights Standard account
You can monitor the performance of your 5030, and pull the performance
metrics of the block storage array when you see poor performance in your
scale cluster.
This will give you some idea as to what is happening,
but The 5030 is designed to be a backup / low IOPS Storage controller, the
processing power and memory in the controllers is very limited.

If you have significant workload happening on your file system in terms of
user access (reads / writes) I am not at all surprised your seeing
performance bottleneck from the 5030.

You could ask you local IBM Presales team to perform a StorM disk model of
the expected performance using your current configuration to show you what
you performance should look like.

Regards

Andrew Beattie
Technical Sales - Storage for Data and AI
IBM Australia and New Zealand

> On 29 May 2021, at 06:04, Uwe Falke <UWEFALKE at de.ibm.com> wrote:
>
> Hi, odd prefetch strategy would affect read performance, but write
latency
> is claimed to be even worse ...
> Have you simply checked what the actual IO performance of the v5k box
> under that load is and how it compares to its nominal performance and
that
> of its disks?
> how is the storage organised? how many LUNs/NSDs, what RAID code (V5k
> cannot do declustered RAID, can it?), any thin provisioning or other
> gimmicks in the game?
> what IO sizes ?
> tons of things to look at.
>
> Mit freundlichen Gr??en / Kind regards
>
> Dr. Uwe Falke
> IT Specialist
> Hybrid Cloud Infrastructure / Technology Consulting & Implementation
> Services
> +49 175 575 2877 Mobile
> Rochlitzer Str. 19, 09111 Chemnitz, Germany
> uwefalke at de.ibm.com
>
> IBM Services
>
> IBM Data Privacy Statement
>
> IBM Deutschland Business & Technology Services GmbH
> Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
> Sitz der Gesellschaft: Ehningen
> Registergericht: Amtsgericht Stuttgart, HRB 17122
>
>
>
> From:   Jan-Frode Myklebust <janfrode at tanso.net>
> To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date:   28/05/2021 19:50
> Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM
> Storwize V5030
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
>
>
>
>
> One thing to check: Storwize/SVC code will *always* guess wrong on
> prefetching for GPFS. You can see this with having a lot higher read data

> throughput on mdisk vs. on on vdisks in the webui. To fix it, disable
> cache_prefetch with "chsystem -cache_prefetch off".
>
> This being a global setting, you probably only should set it if the
system
> is only used for GPFS.
>
>
>   -jf
>
> On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
> oluwasijibomi.saula at ndsu.edu> wrote:
> Hi Folks,
>
> So, we are experiencing some very long IO waiters in our GPFS cluster:
>
> #  mmdiag --waiters
>
> === mmdiag: waiters ===
> Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread:
for
> I/O completion
> Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread:
for
> I/O completion
> Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread:
for
> I/O completion
> Waiting 15.2013 sec since 10:41:03, monitored, thread 22175
>
> However, GPFS support is pointing to our IBM Storwize V5030 disk system
as
> the source of latency. Unfortunately, we don't have paid support for the
> system so we are polling for anyone who might be able to assist.
>
> Does anyone by chance have any experience with IBM Storwize V5030 or
> possess a problem determination guide for the V5030?
>
> We've briefly reviewed the V5030 management portal, but we still haven't
> identified a cause for the increased latencies (i.e. read ~129ms, write
> ~198ms).
>
> Granted, we have some heavy client workloads, yet we seem to experience
> this drastic drop in performance every couple of months, probably
> exacerbated by heavy IO demands.
>
> Any assistance would be much appreciated.
>
>
> Thanks,
>
> (Siji) Saula
> HPC Systems Administrator  /  Information Technology
>
> Research 2 Building 220B / Fargo ND 58108-6050
> p: 701.231.7749 / www.ndsu.edu
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/49d91d85/attachment.htm>

From Philipp.Rehs at uni-duesseldorf.de  Sat May  1 13:12:05 2021
From: Philipp.Rehs at uni-duesseldorf.de (Philipp Rehs)
Date: Sat, 1 May 2021 14:12:05 +0200
Subject: [gpfsug-discuss] reinstalled gpfs gui but no admin user created
Message-ID: <7c1e1de0-3a98-e5d5-f424-cb2c71b23a77@uni-duesseldorf.de>

Hello,

today our gpfs gui crashed because of a broken hard drive and i needed
to reinstall the node. It is a legacy system still at 4.2.3.24

After the installation the gui is stuck at "Loading" and in the logs i
see the following errors:

Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]


For me it looks like it is related to this thread but it has no solution.

http://www.spectrumscale.org/pipermail/gpfsug-discuss/2017-December/004319.html

I already tried to install older versions of the gui but it did not help.

Is it possible to remove all old configurations even from ccr?

Kind regards
 Philipp

-- 
---------------------------

Zentrum f?r Informations- und Medientechnologie
Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern

Heinrich-Heine-Universit?t D?sseldorf
Universit?tsstr. 1
Raum 25.41.00.51
40225 D?sseldorf / Germany
Tel: +49-211-81-15557


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5523 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210501/18bb7e32/attachment-0001.bin>

From scale at us.ibm.com  Mon May  3 12:04:57 2021
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Mon, 3 May 2021 16:34:57 +0530
Subject: [gpfsug-discuss] reinstalled gpfs gui but no admin user created
In-Reply-To: <7c1e1de0-3a98-e5d5-f424-cb2c71b23a77@uni-duesseldorf.de>
References: <7c1e1de0-3a98-e5d5-f424-cb2c71b23a77@uni-duesseldorf.de>
Message-ID: <OFC801667D.40F5A0FE-ON852586CA.003CE36A-652586CA.003CE108@notes.na.collabserv.com>


Hi Ratan,

Can you please look into this GUI issue.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------

If you feel that your question can benefit other users of  Spectrum Scale
(GPFS), then please post it to the public IBM developerWroks Forum at
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479.


If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact
1-800-237-5511 in the United States or your local IBM Service Center in
other countries.

The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.



From:	Philipp Rehs <Philipp.Rehs at uni-duesseldorf.de>
To:	gpfsug-discuss at spectrumscale.org
Date:	01-05-2021 05.50 PM
Subject:	[EXTERNAL] [gpfsug-discuss] reinstalled gpfs gui but no admin
            user created
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hello,

today our gpfs gui crashed because of a broken hard drive and i needed
to reinstall the node. It is a legacy system still at 4.2.3.24

After the installation the gui is stuck at "Loading" and in the logs i
see the following errors:

Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings
(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin
(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet
(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service
(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter
(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter
(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings
(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin
(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet
(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service
(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter
(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter
(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]


For me it looks like it is related to this thread but it has no solution.

http://www.spectrumscale.org/pipermail/gpfsug-discuss/2017-December/004319.html


I already tried to install older versions of the gui but it did not help.

Is it possible to remove all old configurations even from ccr?

Kind regards
 Philipp

--
---------------------------

Zentrum f?r Informations- und Medientechnologie
Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern

Heinrich-Heine-Universit?t D?sseldorf
Universit?tsstr. 1
Raum 25.41.00.51
40225 D?sseldorf / Germany
Tel: +49-211-81-15557


[attachment "smime.p7s" deleted by Huzefa H Pancha/India/IBM]
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210503/66e5866c/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210503/66e5866c/attachment-0001.gif>

From committee at io500.org  Wed May  5 17:38:45 2021
From: committee at io500.org (IO500 Committee)
Date: Wed, 05 May 2021 10:38:45 -0600
Subject: [gpfsug-discuss] Call For Submissions IO500 ISC21 List
Message-ID: <561bba3f84c7ef935c1dd67004bec46a@io500.org>

https://io500.org/cfs
Stabilization Period: 05 - 14 May 2021 AoE
Submission Deadline: 11 June 2021 AoE

The IO500 is now accepting and encouraging submissions for the upcoming 
8th IO500 list. Once again, we are also accepting submissions to the 10 
Node Challenge to encourage the submission of small scale results. The 
new ranked lists will be announced via live-stream at a virtual session. 
We hope to see many new results.

What's New
Starting with ISC'21, the IO500 now follows a two-staged approach. 
First, there will be a two-week stabilization period during which we 
encourage the community to verify that the benchmark runs properly. 
During this period the benchmark will be updated based upon feedback 
from the community. The final benchmark will then be released on Monday, 
May 1st. We expect that runs compliant with the rules made during the 
stabilization period are valid as the final submission unless a 
significant defect is found.
We are now creating a more detailed schema to describe the hardware and 
software of the system under test and provide the first set of tools to 
ease capturing of this information for inclusion with the submission. 
Further details will be released on the submission page.

Background
The benchmark suite is designed to be easy to run and the community has 
multiple active support channels to help with any questions. Please note 
that submissions of all sizes are welcome; the site has customizable 
sorting, so it is possible to submit on a small system and still get a 
very good per-client score, for example. Additionally, the list is about 
much more than just the raw rank; all submissions help the community by 
collecting and publishing a wider corpus of data. More details below.

Following the success of the Top500 in collecting and analyzing 
historical trends in supercomputer technology and evolution, the IO500 
was created in 2017, published its first list at SC17, and has grown 
exponentially since then. The need for such an initiative has long been 
known within High-Performance Computing; however, defining appropriate 
benchmarks had long been challenging. Despite this challenge, the 
community, after long and spirited discussion, finally reached consensus 
on a suite of benchmarks and a metric for resolving the scores into a 
single ranking.

The multi-fold goals of the benchmark suite are as follows:

Maximizing simplicity in running the benchmark suite
Encouraging optimization and documentation of tuning parameters for 
performance
Allowing submitters to highlight their "hero run" performance numbers
Forcing submitters to simultaneously report performance for challenging 
IO patterns.

Specifically, the benchmark suite includes a hero-run of both IOR and 
mdtest configured however possible to maximize performance and establish 
an upper-bound for performance. It also includes an IOR and mdtest run 
with highly prescribed parameters in an attempt to determine a 
lower-bound. Finally, it includes a namespace search as this has been 
determined to be a highly sought-after feature in HPC storage systems 
that has historically not been well-measured. Submitters are encouraged 
to share their tuning insights for publication.

The goals of the community are also multi-fold:

Gather historical data for the sake of analysis and to aid predictions 
of storage futures
Collect tuning information to share valuable performance optimizations 
across the community
Encourage vendors and designers to optimize for workloads beyond "hero 
runs"
Establish bounded expectations for users, procurers, and administrators

10 Node I/O Challenge
The 10 Node Challenge is conducted using the regular IO500 benchmark, 
however, with the rule that exactly 10 client nodes must be used to run 
the benchmark. You may use any shared storage with, e.g., any number of 
servers. When submitting for the IO500 list, you can opt-in for 
"Participate in the 10 compute node challenge only", then we will not 
include the results into the ranked list. Other 10-node node submissions 
will be included in the full list and in the ranked list. We will 
announce the result in a separate derived list and in the full list but 
not on the ranked IO500 list at io500.org.

Birds-of-a-Feather
Once again, we encourage you to submit to join our community, and to 
attend our virtual BoF "The IO500 and the Virtual Institute of I/O" at 
ISC 2021, (time to be announced), where we will announce the new IO500 
and 10 node challenge lists. The current list includes results from 
BeeGFS, CephFS, DAOS, DataWarp, GekkoFS, GFarm, IME, Lustre, MadFS, 
Qumulo, Spectrum Scale, Vast, WekaIO, and YRCloudFile. We hope that the 
upcoming list grows even more.


-- 
The IO500 Committee


From heinrich.billich at id.ethz.ch  Mon May 10 13:16:57 2021
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Mon, 10 May 2021 12:16:57 +0000
Subject: [gpfsug-discuss] migrate to new metadata GNR systems while
 maintaining a fallback possibilty
Message-ID: <62CAB006-A4B1-4E19-9ADD-A3DA5E9C3BC2@id.ethz.ch>

Hello,

 

I need to update/replace our metadata disks but I want to keep old and new in parallel for a while before I remove the old storage: as active/active pair with double copies. This will allow an immediate fall-back if we ever need.

 

Maybe you want to comment on this procedure ? I probably found it some time ago on this mailing list, sorry if I don?t remember the author.

 

Some concerns I have

I?m a bit worried what happens when we remove the vdisks with the second copy in the last step ? will be get millions of error messages? ?Is the sequence of commands right? 

I need to change the failure groups of some vdisks while in use ? I wonder if this poses some risk? As I understand this will change the order of block allocation among the nsds (not the allocation map I guess)

This will double metadata write-io, the systems should be able to handle it. 

We will get? better metadata read-io during the transition than what we?ll finally get.

 

== Start ? ?old? ESS/GNR only

 

 -m 1 (single metadata copy), 

?-M 2

 -K whenpossible

 Metadata vdisks in failure groups 1 and 2

 

== Preparation

 

Use? mmvdisk filesystems? and move all metadata vdisk-sets to a single failuregroup (this has to be done in operation, under load)

Set ?-m 2, while we still have one failure group only. -K whenpossible will keep us running

Add the new vdisk-set with a second failure group on ?new? ESS/GNR systems

Now all new inodes have two copies, one on old and one on new

 

== Create copies on ?old? and ?new?

 

 mmrestripefs -R with ?qos maintenance and -N helper-nodes to minimize the load. This may create some locks on the filesystem/cluster and interfere with backup and snapshots??

Maybe better: use a policy ?replicate? rule to replicate all files, I can run this in small chunks and run mmrestripefs just once to crosscheck.

 

== Observe for some days, handle remaining filesystems in the same way

 

== Finally

 

Suspend ?old? disks

Run mmrestripefs -m

Remove ?old? vdisk sets with mmvidisk ? this will run another mmdeldisk

Change to -m 1

Run fix replication setting mmrestripefs -R (if needed?)

 

Thank you for reading and for any comments or suggestions.

 

Heiner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210510/bf8bb6f9/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5254 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210510/bf8bb6f9/attachment-0001.bin>

From S.J.Thompson at bham.ac.uk  Mon May 10 18:08:24 2021
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Mon, 10 May 2021 17:08:24 +0000
Subject: [gpfsug-discuss] UK UG survey
Message-ID: <A2712CC4-3877-41E3-9B9C-159D10776C74@bham.ac.uk>

I?m currently trying to gather some feedback on running an in-person SSUG event in the UK later this year ? (subject to rules etc etc). If you would normally attend the event in the UK (or you would be very likely to), I?d appreciate 3 mins of time to fill out a quick survey to gather opinion ?

https://www.surveymonkey.co.uk/r/TVWVWGC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210510/e8dabada/attachment-0001.htm>

From mutantllama at gmail.com  Wed May 12 04:40:50 2021
From: mutantllama at gmail.com (Carl)
Date: Wed, 12 May 2021 13:40:50 +1000
Subject: [gpfsug-discuss] Experience running CrowdStrike with Spectrum Scale
Message-ID: <CAGedhdf0bi0np1z-bpMgEzvumuK3j2ZbOgFV0wQfh3842Mb06w@mail.gmail.com>

Hi folks,

We are currently looking at deploying CrowdStrike on our login nodes.
Has anyone had experience with running CrowdStrike with Spectrum
Scale? Did you have any issues?

Cheers,

Carl.


From TROPPENS at de.ibm.com  Fri May 14 12:52:25 2021
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Fri, 14 May 2021 11:52:25 +0000
Subject: [gpfsug-discuss] Charts of German User Meeting are now available
Message-ID: <OF8E05C51D.D4F014E9-ON002586D5.00412744-002586D5.004139A2@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210514/b26ee9ec/attachment-0001.htm>

From chair at spectrumscale.org  Sun May 16 19:14:05 2021
From: chair at spectrumscale.org (Simon Thompson (Spectrum Scale User Group Chair))
Date: Sun, 16 May 2021 19:14:05 +0100
Subject: [gpfsug-discuss] SSUG::Digital: What is new in Spectrum Scale 5.1.1?
Message-ID: <>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210516/d3275165/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: meeting.ics
Type: text/calendar
Size: 2418 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210516/d3275165/attachment-0001.ics>

From Kritika.Gulati at ibm.com  Tue May 18 15:29:45 2021
From: Kritika.Gulati at ibm.com (Kritika Gulati)
Date: Tue, 18 May 2021 14:29:45 +0000
Subject: [gpfsug-discuss] reinstalled gpfs gui but no admin user created
Message-ID: <OF8971FFD2.1980C2A9-ON002586D9.004EED6E-002586D9.004FA113@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210518/8fc48e6a/attachment-0001.htm>

From davebond7787 at gmail.com  Thu May 20 13:56:51 2021
From: davebond7787 at gmail.com (Dave Bond)
Date: Thu, 20 May 2021 13:56:51 +0100
Subject: [gpfsug-discuss] FAL audit events not always triggered
In-Reply-To: <CAHuryA0f86GG33D9_0+YVtSnMVfBFoPcaQq6z4BjD9xB8M0T6A@mail.gmail.com>
References: <CAHuryA0f86GG33D9_0+YVtSnMVfBFoPcaQq6z4BjD9xB8M0T6A@mail.gmail.com>
Message-ID: <CAHuryA2ymNHGkLJE=20UMcpnaOq6XOHgjLnM9hEU0ZPN_r_BRw@mail.gmail.com>

Hello

I am currently testing out the FAL and watch folders auditing mechanism in
5.1.0.  I have noticed that audit events such as file access or creation
are not always recorded in the FAL log on the filesystem.  It would seem
necessary to disable and re enable FAL auditing for the same file access to
be recorded.  I have only triggered this condition twice so far, but it has
set in my mind a few questions.

1) Have others seen this?  If so what is the trigger, as I do not yet have
a reproducer.
2) Is it intended for the audit mechanism to be a soft audit?  i.e the
audit is best effort, and non blocking for file access.

This is using a single NSD and a single remote cluster doing the file
writing.  FAL is running on the NSD node.

Regards

Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/c4895935/attachment-0001.htm>

From davebond7787 at gmail.com  Thu May 20 13:58:01 2021
From: davebond7787 at gmail.com (Dave Bond)
Date: Thu, 20 May 2021 13:58:01 +0100
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <CAHuryA11v_7AWSssSoKzD8PLb8gyNeGsoZe39hZPCN=TDcJ8Pw@mail.gmail.com>
References: <CAHuryA11v_7AWSssSoKzD8PLb8gyNeGsoZe39hZPCN=TDcJ8Pw@mail.gmail.com>
Message-ID: <CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>

Hello

As part of a project I am doing I am looking if there are any de
duplication options for GPFS?  I see there is no native de dupe for the
filesystem. The scenario would be user A creates a file or folder and user
B takes a copy within the same filesystem, though separate independent
filesets.  The intention would be to store 1 copy.    So I was wondering
....

1) Is this is planned to be implemented into GPFS in the future?
2) Is anyone is using any other solutions that have good GPFS integration?

Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/71a20baa/attachment-0001.htm>

From abeattie at au1.ibm.com  Thu May 20 14:27:57 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Thu, 20 May 2021 13:27:57 +0000
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>
Message-ID: <OF162CFDAD.3050490E-ON002586DB.0049F835-1621517277192@notes.na.collabserv.com>

Dave,

Spectrum Scale does not support de-duplication, it does support compression. 
You can however use block storage that supports over subscription / thin provisioning / deduplication for data only NSD?s, we do not recommend them for metadata.

In your scenario is user B planning on making changes to the data which is why you need a copy? 

I know of customers that do this regularly with block storage such as the IBM Flashsystem product family In conjunction with IBM Spectrum Copy Data Management.  But I don?t believe CDM supports file based storage.

Regards, 

Andrew Beattie
Technical Sales Specialist
Storage for Data and AI
IBM Australia and New Zealand
P. +61 421 337 927
E. Abeattie at au1.ibm.com

> On 20 May 2021, at 22:58, Dave Bond <davebond7787 at gmail.com> wrote:
> 
> ?
> This Message Is From an External Sender
> This message came from outside your organization.
> 
> 
> Hello
> 
> As part of a project I am doing I am looking if there are any de duplication options for GPFS?  I see there is no native de dupe for the filesystem. The scenario would be user A creates a file or folder and user B takes a copy within the same filesystem, though separate independent filesets.  The intention would be to store 1 copy.    So I was wondering ....
> 
> 1) Is this is planned to be implemented into GPFS in the future?
> 2) Is anyone is using any other solutions that have good GPFS integration?
> 
> Dave

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/670d8c55/attachment-0001.htm>

From Luis.I.Teran at ibm.com  Thu May 20 23:26:36 2021
From: Luis.I.Teran at ibm.com (Luis I Teran)
Date: Thu, 20 May 2021 22:26:36 +0000
Subject: [gpfsug-discuss] FAL audit events not always triggered
Message-ID: <OF7630AC17.4C49DFA1-ON002586DB.00792014-002586DB.007B494D@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/3336ec92/attachment-0001.htm>

From ulmer at ulmer.org  Fri May 21 00:42:30 2021
From: ulmer at ulmer.org (Stephen Ulmer)
Date: Thu, 20 May 2021 19:42:30 -0400
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <OF162CFDAD.3050490E-ON002586DB.0049F835-1621517277192@notes.na.collabserv.com>
References: <OF162CFDAD.3050490E-ON002586DB.0049F835-1621517277192@notes.na.collabserv.com>
Message-ID: <16E2BBAA-3DC2-41B7-83C2-F875CDA3D57E@ulmer.org>

Do file clones meet the workflow requirement? That is, can you control from whence the second (and further) copies are made?

 -- 
Stephen


> On May 20, 2021, at 9:01 AM, Andrew Beattie <abeattie at au1.ibm.com> wrote:
> 
> ?Dave,
> 
> Spectrum Scale does not support de-duplication, it does support compression. 
> You can however use block storage that supports over subscription / thin provisioning / deduplication for data only NSD?s, we do not recommend them for metadata.
> 
> In your scenario is user B planning on making changes to the data which is why you need a copy? 
> 
> I know of customers that do this regularly with block storage such as the IBM Flashsystem product family In conjunction with IBM Spectrum Copy Data Management.  But I don?t believe CDM supports file based storage.
> 
> Regards, 
> 
> Andrew Beattie
> Technical Sales Specialist
> Storage for Data and AI
> IBM Australia and New Zealand
> P. +61 421 337 927
> E. Abeattie at au1.ibm.com
> 
>>> On 20 May 2021, at 22:58, Dave Bond <davebond7787 at gmail.com> wrote:
>>> 
>> ?
>> 
>> 
>> Hello
>> 
>> As part of a project I am doing I am looking if there are any de duplication options for GPFS?  I see there is no native de dupe for the filesystem. The scenario would be user A creates a file or folder and user B takes a copy within the same filesystem, though separate independent filesets.  The intention would be to store 1 copy.    So I was wondering ....
>> 
>> 1) Is this is planned to be implemented into GPFS in the future?
>> 2) Is anyone is using any other solutions that have good GPFS integration?
>> 
>> Dave
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/ce7a47ee/attachment-0001.htm>

From david.reynolds at tes-es.com  Fri May 21 09:42:11 2021
From: david.reynolds at tes-es.com (David Reynolds)
Date: Fri, 21 May 2021 08:42:11 +0000
Subject: [gpfsug-discuss] Spectrum Scale & S3
Message-ID: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>

When we talk about supported protocols on Scale, S3 is listed but its normally prefixed with Object. Does this mean that the S3 connectivity is used purely to connect and migrate data in object stores to parallel systems (and vice versa) or can you connect to applications/workflows using S3?

David Reynolds
TES Enterprise Solutions
david.reynolds at tes-es.com<mailto:david.reynolds at tes-es.com>
07594 524986

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210521/4d7c6b04/attachment-0001.htm>

From jonathan.buzzard at strath.ac.uk  Fri May 21 09:57:23 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Fri, 21 May 2021 09:57:23 +0100
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>
References: <CAHuryA11v_7AWSssSoKzD8PLb8gyNeGsoZe39hZPCN=TDcJ8Pw@mail.gmail.com>
	<CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>
Message-ID: <616926ad-e16b-2e7d-d45d-da3a803ae185@strath.ac.uk>

On 20/05/2021 13:58, Dave Bond wrote:

> As part of a project I am doing I am looking if there are any de 
> duplication options for GPFS?? I see there is no native de dupe for the 
> filesystem. The scenario would be user A creates a file or folder and 
> user B takes a copy within the same filesystem, though separate 
> independent filesets.? The intention would be to store 1 copy.? ? So I 
> was wondering ....
> 
> 1) Is this is planned to be implemented into GPFS in the future?
> 2) Is anyone is using any other solutions that have good GPFS integration?
> 

Disk space in 2021 is insanely cheap. With an ESS/DSS you can get many 
PB in a single rack. The complexity that dedup introduces is simple not 
worth it IMHO.

Or put another way there is better things the developers at IBM can be 
working on than dedup code.

Historically if you crunched the numbers the licensing for dedup on 
NetApp was similar to just buying more disk unless you where storing 
hundreds of copies of the same data. About the only use case scenario 
would be storing lots of virtual machines. However I refer you back to 
my original point :-)


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From abeattie at au1.ibm.com  Fri May 21 10:56:38 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Fri, 21 May 2021 09:56:38 +0000
Subject: [gpfsug-discuss] Spectrum Scale & S3
In-Reply-To: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>
Message-ID: <OF964F942C.E8EC157A-ON002586DC.00369F4C-1621590998632@notes.na.collabserv.com>

David,

You can use the CES protocol nodes to present a swift Object interface or S3 Object interface for applications to access data from the filesystem / write data into the filesystem over the object protocol. 

Check the openstack Swift -S3 compatabioity matrix to make sure that the functionality from swift aligns with your App requirements 

Sent from my iPhone

> On 21 May 2021, at 18:42, David Reynolds <david.reynolds at tes-es.com> wrote:
> 
> ?
> This Message Is From an External Sender
> This message came from outside your organization.
> When we talk about supported protocols on Scale, S3 is listed but its normally prefixed with Object. Does this mean that the S3 connectivity is used purely to connect and migrate data in object stores to parallel systems (and vice versa) or can you connect to applications/workflows using S3?
>  
> David Reynolds
> TES Enterprise Solutions
> david.reynolds at tes-es.com
> 07594 524986
>  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210521/173855a9/attachment-0001.htm>

From janfrode at tanso.net  Fri May 21 15:16:17 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 21 May 2021 16:16:17 +0200
Subject: [gpfsug-discuss] Spectrum Scale & S3
In-Reply-To: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>
References: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>
Message-ID: <CAHwPath6YvRcJhijqHoedsXXqgx6GTBvOW82rDvseM18fDU9uA@mail.gmail.com>

It has features for both being an Object store for other applications
(running openstack swift/S3), and for migrating/tiering filesystem data to
an object store like Amazon S3, IBM COS, etc...


  -jf

fre. 21. mai 2021 kl. 10:42 skrev David Reynolds <david.reynolds at tes-es.com
>:

> When we talk about supported protocols on Scale, S3 is listed but its
> normally prefixed with Object. Does this mean that the S3 connectivity is
> used purely to connect and migrate data in object stores to parallel
> systems (and vice versa) or can you connect to applications/workflows using
> S3?
>
>
>
> David Reynolds
>
> TES Enterprise Solutions
>
> david.reynolds at tes-es.com
>
> 07594 524986
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210521/a8eb7280/attachment-0001.htm>

From tomasz.rachobinski at gmail.com  Mon May 24 14:06:01 2021
From: tomasz.rachobinski at gmail.com (=?UTF-8?Q?Tomasz_Rachobi=C5=84ski?=)
Date: Mon, 24 May 2021 15:06:01 +0200
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0
Message-ID: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>

Hello everyone,
we are trying to implement a mixed linux/windows environment and we have
one thing at the top - is there any global method to avoid asynchronous I/O
and write everything in synchronous mode?
Another thing is - if there is no global sync setting, how to enforce sync
i/o from linux/windows client?

Greetings
Tom Rachobinski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210524/be2a202b/attachment-0001.htm>

From Alec.Effrat at wellsfargo.com  Mon May 24 18:23:38 2021
From: Alec.Effrat at wellsfargo.com (Alec.Effrat at wellsfargo.com)
Date: Mon, 24 May 2021 17:23:38 +0000
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0
In-Reply-To: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
References: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
Message-ID: <22a8b9b6fa474a62927ed5e215a548c6@wellsfargo.com>

In the past we?d used EMC RecoverPoint for a Spectrum Scale file system.  It isn?t officially supported, and it doesn?t provide for an online replica, but it can keep the I/O synchronous and DVR style file system playback and we had written our own integrations for it via it?s SSH interface.  It is a very competent solution and does pair well with GPFS Spectrum Scale.

Our Spectrum Scale interface basically had to do the following:

1)      Ship updates to the mmdrfs file to the target cluster.

2)      In control failover we would tag the image once GPFS was stopped

3)      Come up on the tagged image on BCP

4)      Had an awk script that would strip out all the host details from the mmdrfs file and update it to 1 host in the BCP cluster.

5)      Import the GPFS file system using the modified mmdrfs file.

Just one way to skin the cat for consideration.  Can provide more actuals on the replication scripts if interested.

Alec Effrat
SAS Lead, AVP
Business Intelligence Competency Center
SAS Administration
Cell 949-246-7713
alec.effrat at wellsfargo.com<mailto:alec.effrat at wellsfargo.com>




From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Tomasz Rachobinski
Sent: Monday, May 24, 2021 6:06 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0

Hello everyone,
we are trying to implement a mixed linux/windows environment and we have one thing at the top - is there any global method to avoid asynchronous I/O and write everything in synchronous mode?
Another thing is - if there is no global sync setting, how to enforce sync i/o from linux/windows client?

Greetings
Tom Rachobinski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210524/75bb9936/attachment-0001.htm>

From krajaram at geocomputing.net  Mon May 24 22:39:59 2021
From: krajaram at geocomputing.net (Kumaran Rajaram)
Date: Mon, 24 May 2021 21:39:59 +0000
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0
In-Reply-To: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
References: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
Message-ID: <SJ0PR18MB4011C053C80A70BEA944CACCBB269@SJ0PR18MB4011.namprd18.prod.outlook.com>

Hi Tom,

>>we are trying to implement a mixed linux/windows environment and we have one thing at the top - is there any global method to avoid asynchronous I/O and write everything in >>synchronous mode?

If the local and remote sites have good inter-site network bandwidth and low-latency, then you may consider using GPFS synchronous replication at the file-system level (-m 2 -r 2).  The Spectrum Scale documentation (link below) has further details.

https://www.ibm.com/docs/en/spectrum-scale/5.1.0?topic=data-synchronous-mirroring-gpfs-replication

Regards,
-Kums

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Tomasz Rachobinski
Sent: Monday, May 24, 2021 9:06 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0

Hello everyone,
we are trying to implement a mixed linux/windows environment and we have one thing at the top - is there any global method to avoid asynchronous I/O and write everything in synchronous mode?
Another thing is - if there is no global sync setting, how to enforce sync i/o from linux/windows client?

Greetings
Tom Rachobinski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210524/bf198bcd/attachment-0001.htm>

From coetzee.ray at gmail.com  Wed May 26 18:33:20 2021
From: coetzee.ray at gmail.com (Ray Coetzee)
Date: Wed, 26 May 2021 18:33:20 +0100
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
Message-ID: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>

Hello all

I'd be interested to know if anyone else has experienced a problem with Kernel
> 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).

We noticed that python shut.copy() is failing against a GPFS mount with:

BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
-> 'test2.file'

To reproduce the error:

```
[user at login01]$ module load python-3.8.9-gcc-9.3.0-soqwnzh

[ user at login01]$ truncate --size 640MB test.file
[ user at login01]$ python3 -c "import shutil; shutil.copy('test.file',
'test2.file')"
Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 418, in copy
 copyfile(src, dst, follow_symlinks=follow_symlinks)
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 275, in copyfile
 _fastcopy_sendfile(fsrc, fdst)
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 172, in _fastcopy_sendfile
 raise err
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 152, in _fastcopy_sendfile
 sent = os.sendfile(outfd, infd, offset, blocksize)
BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
-> 'test2.file'



 Investigating into why this is happening revealed that:


1. It is failing for python3.8 and above.
2. It is happening only a GPFS mount
3. It is happening with files whose size is multiple of 4KB (OS Page size)

Relevant links:
https://bugs.python.org/issue43743
https://www.ibm.com/support/pages/apar/IJ28891


Doing an strace revealed that at the lower level, it seems to be related to
the Linux Syscall of ?sendfile?, which seems to fail in these cases on GPFS.


Strace for a 4096 B file:

```
sendfile(4, 3, [0] => [4096], 8388608) = 4096
sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily
unavailable)

```

The same file on other disk:
```
sendfile(4, 3, [0] => [4096], 8388608) = 4096
sendfile(4, 3, [4096], 8388608) = 0


IBM's "fix" for the problem of "Do not use a file size which that is a
multiple of the page size."  sounds really blas?.


```


Kind regards

Ray Coetzee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/cf991430/attachment-0001.htm>

From knop at us.ibm.com  Wed May 26 18:45:38 2021
From: knop at us.ibm.com (Felipe Knop)
Date: Wed, 26 May 2021 17:45:38 +0000
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
References: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
Message-ID: <OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/ff9d3d0c/attachment-0001.htm>

From coetzee.ray at gmail.com  Wed May 26 19:27:10 2021
From: coetzee.ray at gmail.com (Ray Coetzee)
Date: Wed, 26 May 2021 19:27:10 +0100
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
References: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
	<OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
Message-ID: <CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>

Hi Felipe

We looked at APAR  IJ28891 & IJ29942, as both look identical but are closed
as a "program error" with a workaround of "Do not use a file size which
that is a multiple of the page size."

Kind regards

Ray Coetzee


On Wed, May 26, 2021 at 6:45 PM Felipe Knop <knop at us.ibm.com> wrote:

> Ray,
>
> I wonder if you are hitting the problem which was fixed on the following
> APAR:
>
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
>   Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
>
>
> ----- Original message -----
> From: Ray Coetzee <coetzee.ray at gmail.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
> Date: Wed, May 26, 2021 1:38 PM
>
> Hello all
>
> I'd be interested to know if anyone else has experienced a problem with Kernel
> > 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).
>
>
> We noticed that python shut.copy() is failing against a GPFS mount with:
>
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
> To reproduce the error:
>
> ```
> [user at login01]$ module load python-3.8.9-gcc-9.3.0-soqwnzh
>
> [ user at login01]$ truncate --size 640MB test.file
> [ user at login01]$ python3 -c "import shutil; shutil.copy('test.file',
> 'test2.file')"
> Traceback (most recent call last):
>  File "<string>", line 1, in <module>
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 418, in copy
>  copyfile(src, dst, follow_symlinks=follow_symlinks)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 275, in copyfile
>  _fastcopy_sendfile(fsrc, fdst)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 172, in _fastcopy_sendfile
>  raise err
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 152, in _fastcopy_sendfile
>  sent = os.sendfile(outfd, infd, offset, blocksize)
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
>
>
>  Investigating into why this is happening revealed that:
>
>
> 1. It is failing for python3.8 and above.
> 2. It is happening only a GPFS mount
> 3. It is happening with files whose size is multiple of 4KB (OS Page size)
>
> Relevant links:
> https://bugs.python.org/issue43743
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
> Doing an strace revealed that at the lower level, it seems to be related
> to the Linux Syscall of ?sendfile?, which seems to fail in these cases on
> GPFS.
>
>
> Strace for a 4096 B file:
>
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily
> unavailable)
>
> ```
>
> The same file on other disk:
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = 0
>
>
>
> IBM's "fix" for the problem of "Do not use a file size which that is a
> multiple of the page size."  sounds really blas?.
>
>
> ```
>
>
> Kind regards
>
> Ray Coetzee
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/450af1a7/attachment-0001.htm>

From knop at us.ibm.com  Wed May 26 19:36:40 2021
From: knop at us.ibm.com (Felipe Knop)
Date: Wed, 26 May 2021 18:36:40 +0000
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>
References: <CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>,
	<CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
	<OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
Message-ID: <OFC1A21264.1ACD353E-ON002586E1.0065CC6B-002586E1.00663C06@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/29fc799d/attachment-0001.htm>

From coetzee.ray at gmail.com  Wed May 26 21:21:32 2021
From: coetzee.ray at gmail.com (Ray Coetzee)
Date: Wed, 26 May 2021 21:21:32 +0100
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <OFC1A21264.1ACD353E-ON002586E1.0065CC6B-002586E1.00663C06@notes.na.collabserv.com>
References: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
	<OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
	<CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>
	<OFC1A21264.1ACD353E-ON002586E1.0065CC6B-002586E1.00663C06@notes.na.collabserv.com>
Message-ID: <CABQRq0EAO5aXKFR+UY28mjUrphOTMrH-orC2meLxRr1ReoZ5vg@mail.gmail.com>

Thank you for the info Felipe.

I'll test with 5.0.5-7 in the morning.

Kind regards

Ray Coetzee




On Wed, May 26, 2021 at 7:36 PM Felipe Knop <knop at us.ibm.com> wrote:

> Ray,
>
> Apologies; I should have added more details. My records show that there is
> a fix for IJ28891 and IJ29942, which was delivered in
>
> 5.1.0.2
> 5.0.5.5
>
> "Program error" I believe is a code that indicates "needs to be fixed in
> our product".
>
> I agree that the workaround mentioned is not "actionable" . The APAR page
> should have been clear that there is a fix available.
>
> Regards,
>
>   Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
>
>
> ----- Original message -----
> From: Ray Coetzee <coetzee.ray at gmail.com>
> To: Felipe Knop <knop at us.ibm.com>
> Cc: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
> Date: Wed, May 26, 2021 2:27 PM
>
> Hi Felipe We looked at APAR IJ28891 & IJ29942, as both look identical but
> are closed as a "program error" with a workaround of "Do not use a file
> size which that is a multiple of the page size." Kind regards ? ? ? ? ? ? ?
> ? ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
> ZjQcmQRYFpfptBannerEnd
> Hi Felipe
>
> We looked at APAR  IJ28891 & IJ29942, as both look identical but are
> closed as a "program error" with a workaround of "Do not use a file size
> which that is a multiple of the page size."
>
> Kind regards
>
> Ray Coetzee
>
>
> On Wed, May 26, 2021 at 6:45 PM Felipe Knop <knop at us.ibm.com> wrote:
>
> Ray,
>
> I wonder if you are hitting the problem which was fixed on the following
> APAR:
>
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
>   Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
>
>
> ----- Original message -----
> From: Ray Coetzee <coetzee.ray at gmail.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
> Date: Wed, May 26, 2021 1:38 PM
>
> Hello all
>
> I'd be interested to know if anyone else has experienced a problem with Kernel
> > 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).
>
>
> We noticed that python shut.copy() is failing against a GPFS mount with:
>
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
> To reproduce the error:
>
> ```
> [user at login01]$ module load python-3.8.9-gcc-9.3.0-soqwnzh
>
> [ user at login01]$ truncate --size 640MB test.file
> [ user at login01]$ python3 -c "import shutil; shutil.copy('test.file',
> 'test2.file')"
> Traceback (most recent call last):
>  File "<string>", line 1, in <module>
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 418, in copy
>  copyfile(src, dst, follow_symlinks=follow_symlinks)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 275, in copyfile
>  _fastcopy_sendfile(fsrc, fdst)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 172, in _fastcopy_sendfile
>  raise err
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 152, in _fastcopy_sendfile
>  sent = os.sendfile(outfd, infd, offset, blocksize)
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
>
>
>  Investigating into why this is happening revealed that:
>
>
> 1. It is failing for python3.8 and above.
> 2. It is happening only a GPFS mount
> 3. It is happening with files whose size is multiple of 4KB (OS Page size)
>
> Relevant links:
> https://bugs.python.org/issue43743
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
> Doing an strace revealed that at the lower level, it seems to be related
> to the Linux Syscall of ?sendfile?, which seems to fail in these cases on
> GPFS.
>
>
> Strace for a 4096 B file:
>
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily
> unavailable)
>
> ```
>
> The same file on other disk:
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = 0
>
>
>
> IBM's "fix" for the problem of "Do not use a file size which that is a
> multiple of the page size."  sounds really blas?.
>
>
> ```
>
>
> Kind regards
>
> Ray Coetzee
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/9096ca4a/attachment-0001.htm>

From pbasmaji at us.ibm.com  Thu May 27 00:59:23 2021
From: pbasmaji at us.ibm.com (Peter Basmajian)
Date: Wed, 26 May 2021 23:59:23 +0000
Subject: [gpfsug-discuss] Be a Spectrum Scale *private* reference,
	get a Spectrum Scale t-shirt!
Message-ID: <OF0163861D.E8273E15-ON002586E1.007605D4-002586E1.0083C7E7@ibm.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/06eba1f6/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Image.1622066097985.png
Type: image/png
Size: 155627 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/06eba1f6/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Image.1622066115686.png
Type: image/png
Size: 133174 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/06eba1f6/attachment-0003.png>

From henrik at morsing.cc  Thu May 27 16:10:39 2021
From: henrik at morsing.cc (Henrik Morsing)
Date: Thu, 27 May 2021 16:10:39 +0100
Subject: [gpfsug-discuss] Ransom attacks
Message-ID: <20210527151039.GC8399@morsing.cc>


Hi,

It struck me that switching a Spectrum Protect solution from tapes to a GPFS filesystem offers much less protection against ransom encryption should the SP server be compromised. Same goes really for compromising an ESS node itself, it is an awful lot of data that can be encrypted very quickly.

Is there anything that can protect the GPFS filesystem against this kind of attack?

Regards,
Henrik


From anobre at br.ibm.com  Thu May 27 16:20:08 2021
From: anobre at br.ibm.com (Anderson Ferreira Nobre)
Date: Thu, 27 May 2021 15:20:08 +0000
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <OF65FE85E0.F99DD819-ON002586E2.005411E1-002586E2.00543DD2@ibm.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210527/c5e51dc4/attachment-0001.htm>

From skylar2 at uw.edu  Thu May 27 16:23:48 2021
From: skylar2 at uw.edu (Skylar Thompson)
Date: Thu, 27 May 2021 08:23:48 -0700
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <20210527152348.3h6qmitg2cyhaqus@thargelion>

You can get clever/complicated (the interpretation could go either way)
with ACLs and SELinux but, at the end of the day, nothing beats the air-gap
of tape backups, IMHO. You might consider a belt&suspenders approach that
includes all of the above plus other controls (2FA, network security,
etc.), and in my experience combining multiple solutions gives flexibility
in that it can be easier to avoid the higher-cost aspects of one solution
taken to an extreme by having one layer mitigate the shortcomings of
another layer.

On Thu, May 27, 2021 at 04:10:39PM +0100, Henrik Morsing wrote:
> 
> Hi,
> 
> It struck me that switching a Spectrum Protect solution from tapes to a GPFS filesystem offers much less protection against ransom encryption should the SP server be compromised. Same goes really for compromising an ESS node itself, it is an awful lot of data that can be encrypted very quickly.
> 
> Is there anything that can protect the GPFS filesystem against this kind of attack?

-- 
-- Skylar Thompson (skylar2 at u.washington.edu)
-- Genome Sciences Department (UW Medicine), System Administrator
-- Foege Building S046, (206)-685-7354
-- Pronouns: He/Him/His


From jonathan.buzzard at strath.ac.uk  Thu May 27 16:49:02 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Thu, 27 May 2021 16:49:02 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527152348.3h6qmitg2cyhaqus@thargelion>
References: <20210527151039.GC8399@morsing.cc>
	<20210527152348.3h6qmitg2cyhaqus@thargelion>
Message-ID: <64fa4f59-a73a-19d4-a789-d63c9955cf64@strath.ac.uk>

On 27/05/2021 16:23, Skylar Thompson wrote:

[SNIP]

> at the end of the day, nothing beats the air-gap of tape backups, IMHO. 

Changing/deleting lots of data on tape takes time. So tape is a really 
good starting point even if you never take the tapes out the library 
except to dispose of them. Your backup is your get out of jail card. 
Protect it like it's Fort Knox.

A bit of security through obscurity by using Power and AIX will not go 
amiss. Even if it only buys you a couple of hours that can be enough to 
save the backup from harm.

Passwords on the Spectrum Protect server should be good *never* be used 
anywhere else, and only a handful of trusted people should have access 
to them.

Make sure you have a reuse delay on those tapes so even if the bastards 
do a del filespace (if they even know how to use TSM) you can roll back 
the database.

I also have the notion that you should be able to cut the power to the 
Spectrum Protect server and tape libraries such that it requires an on 
site visit to manually power them backup by flicking a nice big molly 
switch. I have a notion in my mind of tripping a residual-current 
device/ground fault circuit interrupter by using a relay to create a 
neutral earth fault. First sign of trouble disconnect the backup system :-)


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From rltodd.ml1 at gmail.com  Thu May 27 19:17:37 2021
From: rltodd.ml1 at gmail.com (Lindsay Todd)
Date: Thu, 27 May 2021 14:17:37 -0400
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>

Henrik,

Generally you need to begin with a good backup or replica, as well as
suitable air-gaps to isolate contamination.  You also need to be able to
quickly detect unusual activity - an SIEM tool like QRadar might help.
Assume that a cyber-incident will happen and plan accordingly.  Use
in-depth security.  But you are right - you lose one of the advantages of
tape - you can make duplicate copies, maybe even a WORM copy, and store it
offsite.

You might at very least want to take snapshots of the storage being used by
Spectrum Protect, and have separate administrators for the ESS and SP
server (to reduce inside risk).  If it was actually GPFS being backed up to
SP, you could have a second GPFS file system that is a point-in-time
synchronized copy of the original GPFS file system - with its own
snapshots.  It could have yet another sysadmin, and you could isolate the
second copy from the network when not actively synchronizing. See
https://www.redbooks.ibm.com/abstracts/redp5559.html?Open

That might not make sense if GPFS is holding the SP backup data, but SP can
do its own replication too - and could replicate using storage from a
second GPFS file system off-site.  Take snapshots of this second storage,
as well as SP database, and again manage with a second sysadmin team.


*Lindsay Todd, PhD*
*Spectrum Scale (GPFS) Solution Architect*
*IBM Advanced Technology Group ? Storage*
*Mobile:** 1-518-369-6108*
*E-mail:* *lindsay at us.ibm.com* <lindsay at us.ibm.com>


On Thu, May 27, 2021 at 11:10 AM Henrik Morsing <henrik at morsing.cc> wrote:

>
> Hi,
>
> It struck me that switching a Spectrum Protect solution from tapes to a
> GPFS filesystem offers much less protection against ransom encryption
> should the SP server be compromised. Same goes really for compromising an
> ESS node itself, it is an awful lot of data that can be encrypted very
> quickly.
>
> Is there anything that can protect the GPFS filesystem against this kind
> of attack?
>
> Regards,
> Henrik
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210527/966d8a32/attachment-0001.htm>

From henrik at morsing.cc  Fri May 28 07:46:35 2021
From: henrik at morsing.cc (Henrik Morsing)
Date: Fri, 28 May 2021 07:46:35 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>
References: <20210527151039.GC8399@morsing.cc>
	<CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>
Message-ID: <20210528064635.GD8399@morsing.cc>

On Thu, May 27, 2021 at 02:17:37PM -0400, Lindsay Todd wrote:
>
>That might not make sense if GPFS is holding the SP backup data, but SP can
>do its own replication too - and could replicate using storage from a
>second GPFS file system off-site.  Take snapshots of this second storage,
>as well as SP database, and again manage with a second sysadmin team.
>

Thanks all for some useful replies, something to take forward.

In this case, SP is using GPFS for storing backup data, this solution was meant to replace the tape libraries completely.

We protect the storage pools cross-site, but our solutions are identical, so if you hacked one, you have hacked both.

Regards,
Henrik


From henrik at morsing.cc  Fri May 28 08:15:37 2021
From: henrik at morsing.cc (Henrik Morsing)
Date: Fri, 28 May 2021 08:15:37 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <OF65FE85E0.F99DD819-ON002586E2.005411E1-002586E2.00543DD2@ibm.com>
References: <20210527151039.GC8399@morsing.cc>
	<OF65FE85E0.F99DD819-ON002586E2.005411E1-002586E2.00543DD2@ibm.com>
Message-ID: <20210528071537.GE8399@morsing.cc>

On Thu, May 27, 2021 at 03:20:08PM +0000, Anderson Ferreira Nobre wrote:
>   Henrik,
>   ?
>   One way would integrate Scale with QRadar. If I'm not wrong, you can
>   configure QRadar to take a snapshot when it detects there's an attack
>   happening. The details you can take from here:
>   [1]https://www.redbooks.ibm.com/redpapers/pdfs/redp5560.pdf
>   [2]https://www.youtube.com/watch?v=Zyw84dvoFR8
>   ?

Hi,

Looking at the video (not read the document yet) I'm not sure QRadar is advanced enough to detect someone encrypting a storage pool from the SP server. It's a single file pretty much access 24x7, but I will look into it further, thanks.

Regards,
Henrik


From jonathan.buzzard at strath.ac.uk  Fri May 28 09:07:12 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Fri, 28 May 2021 09:07:12 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210528064635.GD8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
	<CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>
	<20210528064635.GD8399@morsing.cc>
Message-ID: <546408b3-0617-f293-8d15-c36f776f5bd8@strath.ac.uk>

On 28/05/2021 07:46, Henrik Morsing wrote:
>>
>> That might not make sense if GPFS is holding the SP backup data, but 
>> SP can do its own replication too - and could replicate using storage from a
>> second GPFS file system off-site.? Take snapshots of this second storage,
>> as well as SP database, and again manage with a second sysadmin team.
>>
> 
> Thanks all for some useful replies, something to take forward.
> 
> In this case, SP is using GPFS for storing backup data, this solution 
> was meant to replace the tape libraries completely.
> 

If your backup is for disaster recovery that's fine. If you expand your 
disaster to include ransom attacks then disk based backups are IMHO 
inadequate simply because they can be gone forever in the blink of an eye.

> We protect the storage pools cross-site, but our solutions are 
> identical, so if you hacked one, you have hacked both.
> 

Currently we use a home grown disk based system for the backup (home 
grown because it's cheap) however we are looking to augment it with tape 
because tape is firstly ransom attack resistant, second tape is "green" 
with a very low carbon footprint.

 From a TSM perspective backup goes to the disk run as a bunch of 
sequential access files like "tapes", and the copy pool will exists on 
tape. We get the benefit of having the backup on disk aka the short 
access times to files, with the protection offered by tape should we get 
hit by a ransom attack.

JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From macthev at gmail.com  Fri May 28 09:23:08 2021
From: macthev at gmail.com (macthev at gmail.com)
Date: Fri, 28 May 2021 18:23:08 +1000
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <326E4E15-014F-4A28-8EBB-1295CF899859@gmail.com>

Take a look at IAM nodes.

Sent from my iPhone

> On 28 May 2021, at 01:10, Henrik Morsing <henrik at morsing.cc> wrote:
> 
> ?
> Hi,
> 
> It struck me that switching a Spectrum Protect solution from tapes to a GPFS filesystem offers much less protection against ransom encryption should the SP server be compromised. Same goes really for compromising an ESS node itself, it is an awful lot of data that can be encrypted very quickly.
> 
> Is there anything that can protect the GPFS filesystem against this kind of attack?
> 
> Regards,
> Henrik
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From heinrich.billich at id.ethz.ch  Fri May 28 09:25:20 2021
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Fri, 28 May 2021 08:25:20 +0000
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to estimate
 remaining execution time
Message-ID: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>

 Hello,

 

I want to estimate how much longer a running 

 

mmrestripefs -R ?metadata-only ?qos maintenance

 

job will take to finish. We did switch from ?-m 1? to ?-m 2 ? and now run mmrestripefs to get the second copy of all metadata. I know that the % values in the output are useless, but now it looks like the number of inodes processed is of no use, too? The filesystem has about 1e9 allocated inodes, but the mmrestripefs reports up to? 7e9 inodes processed? The number of inodes reported increases heavily since it passed the number of allocated inodes???

 

?? 4.41 % complete on Fri May 28 09:22:05 2021? (7219160122 inodes with total??? 8985067 MB data processed)

 

 

Should I consider the ?MB data processed? instead ? will the job finish when all used metadata disk space is processed? 

 

I see little iops on the metadata disks and wonder if something is stuck, I?m not sure if it makes sense to wait any longer. But without an estimate on how much longer the job will take I hesitate to restart. A restart will need to scan all metadata again ? Metadata is on SSD or NVMe, disks iops never were limiting. I turned qos off on the filesystem, previously I restricted the iops. But I don?t see any increase in iops.

 

Thank you. I know this is an old issue, but still I would strongly welcome any advice or comment.

 

Cheers,

 

Heiner

 

I?m aware of RFE ID:150409 mmrestripefs % complete metric is near useless. I?m not sure if it covers the metadata-only case, too.

 

Inode Information

-----------------

Total number of used inodes in all Inode spaces:????????? 735005692

Total number of free inodes in all Inode spaces:????????? 335462660

Total number of allocated inodes in all Inode spaces:??? 1070468352

Total of Maximum number of inodes in all Inode spaces:?? 1223782912

 

Mmrestripefs ?output:

 

START Mon May 24 20:27:25 CEST 2021

Scanning file system metadata, phase 1 ...

 100 % complete on Mon May 24 21:49:27 2021

Scan completed successfully.

Scanning file system metadata, phase 2 ...

 100 % complete on Mon May 24 21:49:28 2021

Scanning file system metadata for data storage pool

? 90 % complete on Mon May 24 21:49:34 2021

 100 % complete on Mon May 24 21:49:35 2021

Scanning file system metadata for Capacity storage pool

? 49 % complete on Mon May 24 21:49:41 2021

 100 % complete on Mon May 24 21:49:46 2021

Scan completed successfully.

Scanning file system metadata, phase 3 ...

Scan completed successfully.

Scanning file system metadata, phase 4 ...

 100 % complete on Mon May 24 21:49:48 2021

Scan completed successfully.

Scanning file system metadata, phase 5 ...

 100 % complete on Mon May 24 21:49:52 2021

Scan completed successfully.

Scanning user file metadata ...

?? 0.01 % complete on Mon May 24 21:50:13 2021? (??? 613108 inodes with total????? 76693 MB data processed)

?? 0.01 % complete on Mon May 24 21:50:33 2021? (?? 1090048 inodes with total????? 80495 MB data processed)

?? 0.01 % complete on Mon May 24 21:50:53 2021? (?? 1591808 inodes with total????? 84576 MB data processed)

?

? 3.97 % complete on Thu May 27 22:01:02 2021? (1048352497 inodes with total??? 8480467 MB data processed)

?? 3.99 % complete on Thu May 27 22:30:18 2021? (1050254855 inodes with total??? 8495969 MB data processed)

?? 4.01 % complete on Thu May 27 22:59:39 2021? (1052304294 inodes with total??? 8512683 MB data processed)

?? 4.03 % complete on Thu May 27 23:29:11 2021? (1055390220 inodes with total??? 8537615 MB data processed)

?? 4.05 % complete on Thu May 27 23:58:58 2021? (1059333989 inodes with total??? 8568871 MB data processed)

?? 4.07 % complete on Fri May 28 00:28:48 2021? (1064728403 inodes with total??? 8611605 MB data processed)

?? 4.09 % complete on Fri May 28 00:58:50 2021? (1067749260 inodes with total??? 8636120 MB data processed). <<< approximately number of allocated inodes

 

?? 4.11 % complete on Fri May 28 01:29:00 2021? (1488665433 inodes with total??? 8661588 MB data processed) <<< whats going??

?? 4.13 % complete on Fri May 28 01:59:23 2021? (1851124480 inodes with total??? 8682324 MB data processed)

?? 4.15 % complete on Fri May 28 02:29:55 2021? (1885948840 inodes with total??? 8700082 MB data processed)

?? 4.17 % complete on Fri May 28 03:00:38 2021? (2604503808 inodes with total?? ?8724069 MB data processed)

?? 4.19 % complete on Fri May 28 03:31:38 2021? (2877196260 inodes with total??? 8746504 MB data processed)

?? 4.21 % complete on Fri May 28 04:02:38 2021? (2933166080 inodes with total??? 8762555 MB data processed)

?? 4.23 % complete on Fri May 28 04:33:48 2021? (2956295936 inodes with total??? 8782298 MB data processed)

?? 4.25 % complete on Fri May 28 05:05:09 2021? (3628799151 inodes with total??? 8802452 MB data processed)

?? 4.27 % complete on Fri May 28 05:36:40 2021? (3970093965 inodes with total??? 8823885 MB data processed)

?? 4.29 % complete on Fri May 28 06:08:20 2021? (4012553472 inodes with total??? 8841407 MB data processed)

?? 4.31 % complete on Fri May 28 06:40:11 2021? (4029545087 inodes with total??? 8858676 MB data processed)

?? 4.33 % complete on Fri May 28 07:12:11 2021? (6080613874 inodes with total??? 8889395 MB data processed)

?? 4.35 % complete on Fri May 28 07:44:21 2021? (6146937531 inodes with total??? 8907253 MB data processed)

?? 4.37 % complete on Fri May 28 08:16:45 2021? (6167408718 inodes with total??? 8925236 MB data processed)

?? 4.39 % complete on Fri May 28 08:49:18 2021? (7151592448 inodes with total??? 8968126 MB data processed)

?? 4.41 % complete on Fri May 28 09:22:05 2021? (7219160122 inodes with total??? 8985067 MB data processed)

 

 

# mmdf fsxxxx -m --block-size M

disk??????????????? disk size? failure holds??? holds?????????? free in MB????????? free in MB

name??????????????????? in MB??? group metadata data??????? in full blocks??????? in fragments

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 32.20 TB)

fsxxxx_rg07b_1??????? 2861295? ??????3 yes????? no????????? 1684452 ( 59%)???????? 35886 ( 1%). << ?1?140?957MB used ? old nsd

fsxxxx_rg07a_1??????? 2861295??????? 3 yes????? no????????? 1685002 ( 59%)???????? 35883 ( 1%

fsxxxx_rg03b_1??????? 2861295??????? 3 yes????? no????????? 1681515 ( 59%)???????? 35627 ( 1%)

fsxxxx_rg03a_1??????? 2861295??????? 3 yes????? no????????? 1680859 ( 59%)???????? 35651 ( 1%)

 

RG003LG004VS015?????? 2046239?????? 12 yes????? no?????????? 904369 ( 44%)?????????? 114 ( 0%) << 1?141?756MB used ? newly added nsd

RG003LG003VS015?????? 2046239?????? 12 yes????? no?????????? 904291 ( 44%)?????????? 118 ( 0%)

RG003LG002VS015?????? 2046239?????? 12 yes????? no?????????? 903632 ( 44%)?????????? 114 ( 0%)

RG003LG001VS015?????? 2046239?????? 12 yes????? no?????????? 903247 ( 44%)?????????? 115 ( 0%)

??????????????? -------------??????? ?????????????????-------------------- -------------------

(pool total)???????? 19630136????????????????????????????? 10347367 ( 53%)??????? 143503 ( 1%)

 

We run spectrum scale 5.0.5.4/5.0.5.5 on Power. 

 

Filesystem:

 

-f???????????????? 32768??????????????????? Minimum fragment (subblock) size in bytes

 -i???????????????? 4096???????????????????? Inode size in bytes

 -I???????????????? 32768??????????????????? Indirect block size in bytes

 -m???????????????? 2??????????????????????? Default number of metadata replicas

 -M???????????????? 2??????????????????????? Maximum number of metadata replicas

 -r???????????????? 1??????????????????????? Default number of data replicas

 -R???????????????? 2??????????????????????? Maximum number of data replicas

 -j???????????????? scatter????????????????? Block allocation type

 

-V???????????????? 23.00 (5.0.5.0)????????? Current file system version

???????????????? ???15.01 (4.2.0.0)????????? Original file system version

 

-L???????????????? 33554432???????????????? Logfile size

 

--subblocks-per-full-block 32?????????????? Number of subblocks per full block

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/0f901305/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5254 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/0f901305/attachment-0001.bin>

From heinrich.billich at id.ethz.ch  Fri May 28 09:55:53 2021
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Fri, 28 May 2021 08:55:53 +0000
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to
 estimate remaining execution time
In-Reply-To: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>
References: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>
Message-ID: <A54D4506-01BF-4E0D-8AA6-194BA3CA89B6@id.ethz.ch>

Hello,

 

I just noticed:

 

Maybe mmrestripefs does some extra processing on snapshots? The output looks much more as expected on filesystems with no snapshots present, both number of inodes and MB data processed allow to estimate the remaining runtime. ?Unfortunately all our large production filesystems use snapshots ?

 

Cheers,

 

Heiner

 

 

?? 1.23 % complete on Tue May 18 21:07:10 2021? (? 37981765 inodes with total???? 328617 MB data processed)

?? 1.25 % complete on Tue May 18 21:17:44 2021? (? 39439584 inodes with total???? 341032 MB data processed)

 100.00 % complete on Tue May 18 21:22:44 2021? (? 41312000 inodes with total???? 356088 MB data processed).??? <<<< # of inodes matches # of allocated inodes, MB processed matches used metadata disk space

 

Scan completed successfully.

# mmdf fsxxx -F

Inode Information

-----------------

Total number of used inodes in all Inode spaces:?????????? 29007227

Total number of free inodes in all Inode spaces:?????????? 12304773

Total number of allocated inodes in all Inode spaces:????? 41312000????? <<<<<<<<<< allocated inodes

Total of Maximum number of inodes in all Inode spaces:???? 87323648

 

 

]# mmdf fsxxx -m --block-size M

disk??????????????? disk size? failure holds??? holds?????????? free in MB????????? free in MB

name??????????????????? in MB??? group metadata data??????? in full blocks??????? in fragments

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 4.13 TB)

??????????????? -------------???????????????????????? -------------------- -------------------

(pool total)????????? 2425152?????????????????????????????? 2067720 ( 85%)????????? 1342 ( 0%).? <<<<<< 356190MB used

 

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Billich Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
Reply to: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Friday, 28 May 2021 at 10:25
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to estimate remaining execution time

 

Hello,

 

I want to estimate how much longer a running 

 

mmrestripefs -R ?metadata-only ?qos maintenance

 

job will take to finish. We did switch from ?-m 1? to ?-m 2 ? and now run mmrestripefs to get the second copy of all metadata. I know that the % values in the output are useless, but now it looks like the number of inodes processed is of no use, too? The filesystem has about 1e9 allocated inodes, but the mmrestripefs reports up to  7e9 inodes processed? The number of inodes reported increases heavily since it passed the number of allocated inodes???

 

   4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with total    8985067 MB data processed)

 

 

Should I consider the ?MB data processed? instead ? will the job finish when all used metadata disk space is processed? 

 

I see little iops on the metadata disks and wonder if something is stuck, I?m not sure if it makes sense to wait any longer. But without an estimate on how much longer the job will take I hesitate to restart. A restart will need to scan all metadata again ? Metadata is on SSD or NVMe, disks iops never were limiting. I turned qos off on the filesystem, previously I restricted the iops. But I don?t see any increase in iops.

 

Thank you. I know this is an old issue, but still I would strongly welcome any advice or comment.

 

Cheers,

 

Heiner

 

I?m aware of RFE ID:150409 mmrestripefs % complete metric is near useless. I?m not sure if it covers the metadata-only case, too.

 

Inode Information

-----------------

Total number of used inodes in all Inode spaces:          735005692

Total number of free inodes in all Inode spaces:          335462660

Total number of allocated inodes in all Inode spaces:    1070468352

Total of Maximum number of inodes in all Inode spaces:   1223782912

 

Mmrestripefs  output:

 

START Mon May 24 20:27:25 CEST 2021

Scanning file system metadata, phase 1 ...

100 % complete on Mon May 24 21:49:27 2021

Scan completed successfully.

Scanning file system metadata, phase 2 ...

100 % complete on Mon May 24 21:49:28 2021

Scanning file system metadata for data storage pool

  90 % complete on Mon May 24 21:49:34 2021

100 % complete on Mon May 24 21:49:35 2021

Scanning file system metadata for Capacity storage pool

  49 % complete on Mon May 24 21:49:41 2021

100 % complete on Mon May 24 21:49:46 2021

Scan completed successfully.

Scanning file system metadata, phase 3 ...

Scan completed successfully.

Scanning file system metadata, phase 4 ...

100 % complete on Mon May 24 21:49:48 2021

Scan completed successfully.

Scanning file system metadata, phase 5 ...

100 % complete on Mon May 24 21:49:52 2021

Scan completed successfully.

Scanning user file metadata ...

   0.01 % complete on Mon May 24 21:50:13 2021  (    613108 inodes with total      76693 MB data processed)

   0.01 % complete on Mon May 24 21:50:33 2021  (   1090048 inodes with total      80495 MB data processed)

   0.01 % complete on Mon May 24 21:50:53 2021  (   1591808 inodes with total      84576 MB data processed)

?

  3.97 % complete on Thu May 27 22:01:02 2021  (1048352497 inodes with total    8480467 MB data processed)

   3.99 % complete on Thu May 27 22:30:18 2021  (1050254855 inodes with total    8495969 MB data processed)

   4.01 % complete on Thu May 27 22:59:39 2021  (1052304294 inodes with total    8512683 MB data processed)

   4.03 % complete on Thu May 27 23:29:11 2021  (1055390220 inodes with total    8537615 MB data processed)

   4.05 % complete on Thu May 27 23:58:58 2021  (1059333989 inodes with total    8568871 MB data processed)

   4.07 % complete on Fri May 28 00:28:48 2021  (1064728403 inodes with total    8611605 MB data processed)

   4.09 % complete on Fri May 28 00:58:50 2021  (1067749260 inodes with total    8636120 MB data processed). <<< approximately number of allocated inodes

 

   4.11 % complete on Fri May 28 01:29:00 2021  (1488665433 inodes with total    8661588 MB data processed) <<< whats going??

   4.13 % complete on Fri May 28 01:59:23 2021  (1851124480 inodes with total    8682324 MB data processed)

   4.15 % complete on Fri May 28 02:29:55 2021  (1885948840 inodes with total    8700082 MB data processed)

   4.17 % complete on Fri May 28 03:00:38 2021  (2604503808 inodes with total    8724069 MB data processed)

   4.19 % complete on Fri May 28 03:31:38 2021  (2877196260 inodes with total    8746504 MB data processed)

   4.21 % complete on Fri May 28 04:02:38 2021  (2933166080 inodes with total    8762555 MB data processed)

   4.23 % complete on Fri May 28 04:33:48 2021  (2956295936 inodes with total    8782298 MB data processed)

   4.25 % complete on Fri May 28 05:05:09 2021  (3628799151 inodes with total    8802452 MB data processed)

   4.27 % complete on Fri May 28 05:36:40 2021  (3970093965 inodes with total    8823885 MB data processed)

   4.29 % complete on Fri May 28 06:08:20 2021  (4012553472 inodes with total    8841407 MB data processed)

   4.31 % complete on Fri May 28 06:40:11 2021  (4029545087 inodes with total    8858676 MB data processed)

   4.33 % complete on Fri May 28 07:12:11 2021  (6080613874 inodes with total    8889395 MB data processed)

   4.35 % complete on Fri May 28 07:44:21 2021  (6146937531 inodes with total    8907253 MB data processed)

   4.37 % complete on Fri May 28 08:16:45 2021  (6167408718 inodes with total    8925236 MB data processed)

   4.39 % complete on Fri May 28 08:49:18 2021  (7151592448 inodes with total    8968126 MB data processed)

   4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with total    8985067 MB data processed)

 

 

# mmdf fsxxxx -m --block-size M

disk                disk size  failure holds    holds           free in MB          free in MB

name                    in MB    group metadata data        in full blocks        in fragments

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 32.20 TB)

fsxxxx_rg07b_1        2861295        3 yes      no          1684452 ( 59%)         35886 ( 1%). <<  1?140?957MB used ? old nsd

fsxxxx_rg07a_1        2861295        3 yes      no          1685002 ( 59%)         35883 ( 1%

fsxxxx_rg03b_1        2861295        3 yes      no          1681515 ( 59%)         35627 ( 1%)

fsxxxx_rg03a_1        2861295        3 yes      no          1680859 ( 59%)         35651 ( 1%)

 

RG003LG004VS015       2046239       12 yes      no           904369 ( 44%)           114 ( 0%) << 1?141?756MB used ? newly added nsd

RG003LG003VS015       2046239       12 yes      no           904291 ( 44%)           118 ( 0%)

RG003LG002VS015       2046239       12 yes      no           903632 ( 44%)           114 ( 0%)

RG003LG001VS015       2046239       12 yes      no           903247 ( 44%)           115 ( 0%)

                -------------                         -------------------- -------------------

(pool total)         19630136                              10347367 ( 53%)        143503 ( 1%)

 

We run spectrum scale 5.0.5.4/5.0.5.5 on Power. 

 

Filesystem:

 

-f                 32768                    Minimum fragment (subblock) size in bytes

-i                 4096                     Inode size in bytes

-I                 32768                    Indirect block size in bytes

-m                 2                        Default number of metadata replicas

-M                 2                        Maximum number of metadata replicas

-r                 1                        Default number of data replicas

-R                 2                        Maximum number of data replicas

-j                 scatter                  Block allocation type

 

-V                 23.00 (5.0.5.0)          Current file system version

                    15.01 (4.2.0.0)          Original file system version

 

-L                 33554432                 Logfile size

 

--subblocks-per-full-block 32               Number of subblocks per full block

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/ec0c5063/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5254 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/ec0c5063/attachment-0001.bin>

From oluwasijibomi.saula at ndsu.edu  Fri May 28 16:57:52 2021
From: oluwasijibomi.saula at ndsu.edu (Saula, Oluwasijibomi)
Date: Fri, 28 May 2021 15:57:52 +0000
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
Message-ID: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>

Hi Folks,

So, we are experiencing some very long IO waiters in our GPFS cluster:


#  mmdiag --waiters


=== mmdiag: waiters ===

Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for I/O completion

Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for I/O completion

Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for I/O completion

Waiting 15.2013 sec since 10:41:03, monitored, thread 22175

However, GPFS support is pointing to our IBM Storwize V5030 disk system as the source of latency. Unfortunately, we don't have paid support for the system so we are polling for anyone who might be able to assist.

Does anyone by chance have any experience with IBM Storwize V5030 or possess a problem determination guide for the V5030?

We've briefly reviewed the V5030 management portal, but we still haven't identified a cause for the increased latencies (i.e. read ~129ms, write ~198ms).

Granted, we have some heavy client workloads, yet we seem to experience this drastic drop in performance every couple of months, probably exacerbated by heavy IO demands.

Any assistance would be much appreciated.



Thanks,


Oluwasijibomi (Siji) Saula

HPC Systems Administrator  /  Information Technology



Research 2 Building 220B / Fargo ND 58108-6050

p: 701.231.7749 / www.ndsu.edu<http://www.ndsu.edu/>



[cid:image001.gif at 01D57DE0.91C300C0]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/f778d057/attachment-0001.htm>

From erich at uw.edu  Fri May 28 17:25:49 2021
From: erich at uw.edu (Eric Horst)
Date: Fri, 28 May 2021 09:25:49 -0700
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to
 estimate remaining execution time
In-Reply-To: <A54D4506-01BF-4E0D-8AA6-194BA3CA89B6@id.ethz.ch>
References: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>
	<A54D4506-01BF-4E0D-8AA6-194BA3CA89B6@id.ethz.ch>
Message-ID: <CAF-To-2FFJHvt7OzSz5gfMur9bKtpG7ixLJtgT6bvLToRRO+TQ@mail.gmail.com>

Yes Heiner, my experience is that the inode count in those operations is:
inodes * snapshots = total. I observed that as it starts processing the
snapshot inodes it moves faster as inode usage in snapshots is more sparse.

-Eric

On Fri, May 28, 2021 at 1:56 AM Billich Heinrich Rainer (ID SD) <
heinrich.billich at id.ethz.ch> wrote:

> Hello,
>
>
>
> I just noticed:
>
>
>
> Maybe mmrestripefs does some extra processing on snapshots? The output
> looks much more as expected on filesystems with no snapshots present, both
> number of inodes and MB data processed allow to estimate the remaining
> runtime.  Unfortunately all our large production filesystems use snapshots ?
>
>
>
> Cheers,
>
>
>
> Heiner
>
>
>
>
>
>    1.23 % complete on Tue May 18 21:07:10 2021  (  37981765 inodes with
> total     328617 MB data processed)
>
>    1.25 % complete on Tue May 18 21:17:44 2021  (  39439584 inodes with
> total     341032 MB data processed)
>
> 100.00 % complete on Tue May 18 21:22:44 2021  (  41312000 inodes with
> total     356088 MB data processed).    <<<< # of inodes matches # of
> allocated inodes, MB processed matches used metadata disk space
>
>
>
> Scan completed successfully.
>
> # mmdf fsxxx -F
>
> Inode Information
>
> -----------------
>
> Total number of used inodes in all Inode spaces:           29007227
>
> Total number of free inodes in all Inode spaces:           12304773
>
> Total number of allocated inodes in all Inode spaces:      41312000
> <<<<<<<<<< allocated inodes
>
> Total of Maximum number of inodes in all Inode spaces:     87323648
>
>
>
>
>
> ]# mmdf fsxxx -m --block-size M
>
> disk                disk size  failure holds    holds           free in
> MB          free in MB
>
> name                    in MB    group metadata data        in full
> blocks        in fragments
>
> --------------- ------------- -------- -------- ----- --------------------
> -------------------
>
> Disks in storage pool: system (Maximum disk size allowed is 4.13 TB)
>
>                 -------------                         --------------------
> -------------------
>
> (pool total)          2425152                               2067720 (
> 85%)          1342 ( 0%).  <<<<<< 356190MB used
>
>
>
> *From: *<gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Billich
> Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
> *Reply to: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Date: *Friday, 28 May 2021 at 10:25
> *To: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Subject: *[gpfsug-discuss] Mmrestripefs -R --metadat-only - how to
> estimate remaining execution time
>
>
>
> Hello,
>
>
>
> I want to estimate how much longer a running
>
>
>
> mmrestripefs -R ?metadata-only ?qos maintenance
>
>
>
> job will take to finish. We did switch from ?-m 1? to ?-m 2 ? and now run
> mmrestripefs to get the second copy of all metadata. I know that the %
> values in the output are useless, but now it looks like the number of
> inodes processed is of no use, too? The filesystem has about 1e9 allocated
> inodes, but the mmrestripefs reports up to  7e9 inodes processed? The
> number of inodes reported increases heavily since it passed the number of
> allocated inodes???
>
>
>
>    4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with
> total    8985067 MB data processed)
>
>
>
>
>
> Should I consider the ?MB data processed? instead ? will the job finish
> when all used metadata disk space is processed?
>
>
>
> I see little iops on the metadata disks and wonder if something is stuck,
> I?m not sure if it makes sense to wait any longer. But without an estimate
> on how much longer the job will take I hesitate to restart. A restart will
> need to scan all metadata again ? Metadata is on SSD or NVMe, disks iops
> never were limiting. I turned qos off on the filesystem, previously I
> restricted the iops. But I don?t see any increase in iops.
>
>
>
> Thank you. I know this is an old issue, but still I would strongly welcome
> any advice or comment.
>
>
>
> Cheers,
>
>
>
> Heiner
>
>
>
> I?m aware of RFE ID:150409 mmrestripefs % complete metric is near useless.
> I?m not sure if it covers the metadata-only case, too.
>
>
>
> Inode Information
>
> -----------------
>
> Total number of used inodes in all Inode spaces:          735005692
>
> Total number of free inodes in all Inode spaces:          335462660
>
> Total number of allocated inodes in all Inode spaces:    1070468352
>
> Total of Maximum number of inodes in all Inode spaces:   1223782912
>
>
>
> Mmrestripefs  output:
>
>
>
> START Mon May 24 20:27:25 CEST 2021
>
> Scanning file system metadata, phase 1 ...
>
> 100 % complete on Mon May 24 21:49:27 2021
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 2 ...
>
> 100 % complete on Mon May 24 21:49:28 2021
>
> Scanning file system metadata for data storage pool
>
>   90 % complete on Mon May 24 21:49:34 2021
>
> 100 % complete on Mon May 24 21:49:35 2021
>
> Scanning file system metadata for Capacity storage pool
>
>   49 % complete on Mon May 24 21:49:41 2021
>
> 100 % complete on Mon May 24 21:49:46 2021
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 3 ...
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 4 ...
>
> 100 % complete on Mon May 24 21:49:48 2021
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 5 ...
>
> 100 % complete on Mon May 24 21:49:52 2021
>
> Scan completed successfully.
>
> Scanning user file metadata ...
>
>    0.01 % complete on Mon May 24 21:50:13 2021  (    613108 inodes with
> total      76693 MB data processed)
>
>    0.01 % complete on Mon May 24 21:50:33 2021  (   1090048 inodes with
> total      80495 MB data processed)
>
>    0.01 % complete on Mon May 24 21:50:53 2021  (   1591808 inodes with
> total      84576 MB data processed)
>
> ?
>
>   3.97 % complete on Thu May 27 22:01:02 2021  (1048352497 inodes with
> total    8480467 MB data processed)
>
>    3.99 % complete on Thu May 27 22:30:18 2021  (1050254855 inodes with
> total    8495969 MB data processed)
>
>    4.01 % complete on Thu May 27 22:59:39 2021  (1052304294 inodes with
> total    8512683 MB data processed)
>
>    4.03 % complete on Thu May 27 23:29:11 2021  (1055390220 inodes with
> total    8537615 MB data processed)
>
>    4.05 % complete on Thu May 27 23:58:58 2021  (1059333989 inodes with
> total    8568871 MB data processed)
>
>    4.07 % complete on Fri May 28 00:28:48 2021  (1064728403 inodes with
> total    8611605 MB data processed)
>
>    4.09 % complete on Fri May 28 00:58:50 2021  (1067749260 inodes with
> total    8636120 MB data processed). <<< approximately number of allocated
> inodes
>
>
>
>    4.11 % complete on Fri May 28 01:29:00 2021  (1488665433 inodes with
> total    8661588 MB data processed) <<< whats going??
>
>    4.13 % complete on Fri May 28 01:59:23 2021  (1851124480 inodes with
> total    8682324 MB data processed)
>
>    4.15 % complete on Fri May 28 02:29:55 2021  (1885948840 inodes with
> total    8700082 MB data processed)
>
>    4.17 % complete on Fri May 28 03:00:38 2021  (2604503808 inodes with
> total    8724069 MB data processed)
>
>    4.19 % complete on Fri May 28 03:31:38 2021  (2877196260 inodes with
> total    8746504 MB data processed)
>
>    4.21 % complete on Fri May 28 04:02:38 2021  (2933166080 inodes with
> total    8762555 MB data processed)
>
>    4.23 % complete on Fri May 28 04:33:48 2021  (2956295936 inodes with
> total    8782298 MB data processed)
>
>    4.25 % complete on Fri May 28 05:05:09 2021  (3628799151 inodes with
> total    8802452 MB data processed)
>
>    4.27 % complete on Fri May 28 05:36:40 2021  (3970093965 inodes with
> total    8823885 MB data processed)
>
>    4.29 % complete on Fri May 28 06:08:20 2021  (4012553472 inodes with
> total    8841407 MB data processed)
>
>    4.31 % complete on Fri May 28 06:40:11 2021  (4029545087 inodes with
> total    8858676 MB data processed)
>
>    4.33 % complete on Fri May 28 07:12:11 2021  (6080613874 inodes with
> total    8889395 MB data processed)
>
>    4.35 % complete on Fri May 28 07:44:21 2021  (6146937531 inodes with
> total    8907253 MB data processed)
>
>    4.37 % complete on Fri May 28 08:16:45 2021  (6167408718 inodes with
> total    8925236 MB data processed)
>
>    4.39 % complete on Fri May 28 08:49:18 2021  (7151592448 inodes with
> total    8968126 MB data processed)
>
>    4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with
> total    8985067 MB data processed)
>
>
>
>
>
> # mmdf fsxxxx -m --block-size M
>
> disk                disk size  failure holds    holds           free in
> MB          free in MB
>
> name                    in MB    group metadata data        in full
> blocks        in fragments
>
> --------------- ------------- -------- -------- ----- --------------------
> -------------------
>
> Disks in storage pool: system (Maximum disk size allowed is 32.20 TB)
>
> fsxxxx_rg07b_1        2861295        3 yes      no          1684452 (
> 59%)         35886 ( 1%). <<  1?140?957MB used ? old nsd
>
> fsxxxx_rg07a_1        2861295        3 yes      no          1685002 (
> 59%)         35883 ( 1%
>
> fsxxxx_rg03b_1        2861295        3 yes      no          1681515 (
> 59%)         35627 ( 1%)
>
> fsxxxx_rg03a_1        2861295        3 yes      no          1680859 (
> 59%)         35651 ( 1%)
>
>
>
> RG003LG004VS015       2046239       12 yes      no           904369 (
> 44%)           114 ( 0%) << 1?141?756MB used ? newly added nsd
>
> RG003LG003VS015       2046239       12 yes      no           904291 (
> 44%)           118 ( 0%)
>
> RG003LG002VS015       2046239       12 yes      no           903632 (
> 44%)           114 ( 0%)
>
> RG003LG001VS015       2046239       12 yes      no           903247 (
> 44%)           115 ( 0%)
>
>                 -------------                         --------------------
> -------------------
>
> (pool total)         19630136                              10347367 (
> 53%)        143503 ( 1%)
>
>
>
> We run spectrum scale 5.0.5.4/5.0.5.5 on Power.
>
>
>
> Filesystem:
>
>
>
> -f                 32768                    Minimum fragment (subblock)
> size in bytes
>
> -i                 4096                     Inode size in bytes
>
> -I                 32768                    Indirect block size in bytes
>
> -m                 2                        Default number of metadata
> replicas
>
> -M                 2                        Maximum number of metadata
> replicas
>
> -r                 1                        Default number of data replicas
>
> -R                 2                        Maximum number of data replicas
>
> -j                 scatter                  Block allocation type
>
>
>
> -V                 23.00 (5.0.5.0)          Current file system version
>
>                     15.01 (4.2.0.0)          Original file system version
>
>
>
> -L                 33554432                 Logfile size
>
>
>
> --subblocks-per-full-block 32               Number of subblocks per full
> block
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/f46d7a39/attachment-0001.htm>

From janfrode at tanso.net  Fri May 28 18:50:21 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 28 May 2021 19:50:21 +0200
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>
References: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>
Message-ID: <CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>

One thing to check: Storwize/SVC code will *always* guess wrong on
prefetching for GPFS. You can see this with having a lot higher read data
throughput on mdisk vs. on on vdisks in the webui. To fix it, disable
cache_prefetch with "chsystem -cache_prefetch off".

This being a global setting, you probably only should set it if the system
is only used for GPFS.


   -jf

On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
oluwasijibomi.saula at ndsu.edu> wrote:

> Hi Folks,
>
> So, we are experiencing some very long IO waiters in our GPFS cluster:
>
> #  mmdiag --waiters
>
>
> === mmdiag: waiters ===
>
> Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for
> I/O completion
>
> Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for
> I/O completion
>
> Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for
> I/O completion
>
> Waiting 15.2013 sec since 10:41:03, monitored, thread 22175
>
> However, GPFS support is pointing to our IBM Storwize V5030 disk system
> as the source of latency. Unfortunately, we don't have paid support for the
> system so we are polling for anyone who might be able to assist.
>
> Does anyone by chance have any experience with IBM Storwize V5030 or
> possess a problem determination guide for the V5030?
>
> We've briefly reviewed the V5030 management portal, but we still haven't
> identified a cause for the increased latencies (i.e. read ~129ms, write
> ~198ms).
>
> Granted, we have some heavy client workloads, yet we seem to experience
> this drastic drop in performance every couple of months, probably
> exacerbated by heavy IO demands.
>
> Any assistance would be much appreciated.
>
>
> Thanks,
>
> *Oluwasijibomi (Siji) Saula*
>
> HPC Systems Administrator  /  Information Technology
>
>
>
> Research 2 Building 220B / Fargo ND 58108-6050
>
> p: 701.231.7749 / www.ndsu.edu
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/832c7e75/attachment-0001.htm>

From UWEFALKE at de.ibm.com  Fri May 28 21:04:19 2021
From: UWEFALKE at de.ibm.com (Uwe Falke)
Date: Fri, 28 May 2021 22:04:19 +0200
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>
References: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>
	<CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>
Message-ID: <OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>

Hi, odd prefetch strategy would affect read performance, but write latency 
is claimed to be even worse ...
Have you simply checked what the actual IO performance of the v5k box 
under that load is and how it compares to its nominal performance and that 
of its disks?
how is the storage organised? how many LUNs/NSDs, what RAID code (V5k 
cannot do declustered RAID, can it?), any thin provisioning or other 
gimmicks in the game?
what IO sizes ?
tons of things to look at. 

Mit freundlichen Gr??en / Kind regards

Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation 
Services
+49 175 575 2877 Mobile
Rochlitzer Str. 19, 09111 Chemnitz, Germany
uwefalke at de.ibm.com

IBM Services

IBM Data Privacy Statement

IBM Deutschland Business & Technology Services GmbH
Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122



From:   Jan-Frode Myklebust <janfrode at tanso.net>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   28/05/2021 19:50
Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM 
Storwize V5030
Sent by:        gpfsug-discuss-bounces at spectrumscale.org




One thing to check: Storwize/SVC code will *always* guess wrong on 
prefetching for GPFS. You can see this with having a lot higher read data 
throughput on mdisk vs. on on vdisks in the webui. To fix it, disable 
cache_prefetch with "chsystem -cache_prefetch off".

This being a global setting, you probably only should set it if the system 
is only used for GPFS. 


   -jf

On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
oluwasijibomi.saula at ndsu.edu> wrote:
Hi Folks,

So, we are experiencing some very long IO waiters in our GPFS cluster:

#  mmdiag --waiters 

=== mmdiag: waiters ===
Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for 
I/O completion
Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for 
I/O completion
Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for 
I/O completion
Waiting 15.2013 sec since 10:41:03, monitored, thread 22175 

However, GPFS support is pointing to our IBM Storwize V5030 disk system as 
the source of latency. Unfortunately, we don't have paid support for the 
system so we are polling for anyone who might be able to assist.

Does anyone by chance have any experience with IBM Storwize V5030 or 
possess a problem determination guide for the V5030?

We've briefly reviewed the V5030 management portal, but we still haven't 
identified a cause for the increased latencies (i.e. read ~129ms, write 
~198ms). 

Granted, we have some heavy client workloads, yet we seem to experience 
this drastic drop in performance every couple of months, probably 
exacerbated by heavy IO demands.

Any assistance would be much appreciated.


Thanks,

Oluwasijibomi (Siji) Saula
HPC Systems Administrator  /  Information Technology
 
Research 2 Building 220B / Fargo ND 58108-6050
p: 701.231.7749 / www.ndsu.edu
 



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 







From UWEFALKE at de.ibm.com  Fri May 28 21:16:12 2021
From: UWEFALKE at de.ibm.com (Uwe Falke)
Date: Fri, 28 May 2021 22:16:12 +0200
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>
References: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com><CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>
	<OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>
Message-ID: <OF7233E8BC.78902F9C-ONC12586E3.006ED69A-C12586E3.006F5924@notes.na.collabserv.com>

You say you see that every few months: does it mean with about the same 
load sometimes the system knocks out and sometimes it behaves ok?
Have you checked the v5k event log, if there is anything going on (write 
performance may suffer if write cache is off which might happen if the 
buffer batteries are low on charge - but again that does not directly 
explain the high read latency). 
Are the latencies you gave derived from GPFS, from the OS, or from the 
V5k?

Mit freundlichen Gr??en / Kind regards

Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation 
Services
+49 175 575 2877 Mobile
Rochlitzer Str. 19, 09111 Chemnitz, Germany
uwefalke at de.ibm.com

IBM Services

IBM Data Privacy Statement

IBM Deutschland Business & Technology Services GmbH
Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122



From:   "Uwe Falke" <UWEFALKE at de.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   28/05/2021 22:04
Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM 
Storwize V5030
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi, odd prefetch strategy would affect read performance, but write latency 

is claimed to be even worse ...
Have you simply checked what the actual IO performance of the v5k box 
under that load is and how it compares to its nominal performance and that 

of its disks?
how is the storage organised? how many LUNs/NSDs, what RAID code (V5k 
cannot do declustered RAID, can it?), any thin provisioning or other 
gimmicks in the game?
what IO sizes ?
tons of things to look at. 

Mit freundlichen Gr??en / Kind regards

Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation 
Services
+49 175 575 2877 Mobile
Rochlitzer Str. 19, 09111 Chemnitz, Germany
uwefalke at de.ibm.com

IBM Services

IBM Data Privacy Statement

IBM Deutschland Business & Technology Services GmbH
Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122



From:   Jan-Frode Myklebust <janfrode at tanso.net>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   28/05/2021 19:50
Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM 
Storwize V5030
Sent by:        gpfsug-discuss-bounces at spectrumscale.org




One thing to check: Storwize/SVC code will *always* guess wrong on 
prefetching for GPFS. You can see this with having a lot higher read data 
throughput on mdisk vs. on on vdisks in the webui. To fix it, disable 
cache_prefetch with "chsystem -cache_prefetch off".

This being a global setting, you probably only should set it if the system 

is only used for GPFS. 


   -jf

On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
oluwasijibomi.saula at ndsu.edu> wrote:
Hi Folks,

So, we are experiencing some very long IO waiters in our GPFS cluster:

#  mmdiag --waiters 

=== mmdiag: waiters ===
Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for 

I/O completion
Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for 

I/O completion
Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for 

I/O completion
Waiting 15.2013 sec since 10:41:03, monitored, thread 22175 

However, GPFS support is pointing to our IBM Storwize V5030 disk system as 

the source of latency. Unfortunately, we don't have paid support for the 
system so we are polling for anyone who might be able to assist.

Does anyone by chance have any experience with IBM Storwize V5030 or 
possess a problem determination guide for the V5030?

We've briefly reviewed the V5030 management portal, but we still haven't 
identified a cause for the increased latencies (i.e. read ~129ms, write 
~198ms). 

Granted, we have some heavy client workloads, yet we seem to experience 
this drastic drop in performance every couple of months, probably 
exacerbated by heavy IO demands.

Any assistance would be much appreciated.


Thanks,

Oluwasijibomi (Siji) Saula
HPC Systems Administrator  /  Information Technology
 
Research 2 Building 220B / Fargo ND 58108-6050
p: 701.231.7749 / www.ndsu.edu
 



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 





_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 








From abeattie at au1.ibm.com  Fri May 28 22:27:12 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Fri, 28 May 2021 21:27:12 +0000
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>
Message-ID: <OF09635240.267CCCEC-ON002586E3.0075D8B6-1622237232907@notes.na.collabserv.com>



Hi Oluwasijibomi,

If you set up a Storage Insights Standard account
You can monitor the performance of your 5030, and pull the performance
metrics of the block storage array when you see poor performance in your
scale cluster.
This will give you some idea as to what is happening,
but The 5030 is designed to be a backup / low IOPS Storage controller, the
processing power and memory in the controllers is very limited.

If you have significant workload happening on your file system in terms of
user access (reads / writes) I am not at all surprised your seeing
performance bottleneck from the 5030.

You could ask you local IBM Presales team to perform a StorM disk model of
the expected performance using your current configuration to show you what
you performance should look like.

Regards

Andrew Beattie
Technical Sales - Storage for Data and AI
IBM Australia and New Zealand

> On 29 May 2021, at 06:04, Uwe Falke <UWEFALKE at de.ibm.com> wrote:
>
> Hi, odd prefetch strategy would affect read performance, but write
latency
> is claimed to be even worse ...
> Have you simply checked what the actual IO performance of the v5k box
> under that load is and how it compares to its nominal performance and
that
> of its disks?
> how is the storage organised? how many LUNs/NSDs, what RAID code (V5k
> cannot do declustered RAID, can it?), any thin provisioning or other
> gimmicks in the game?
> what IO sizes ?
> tons of things to look at.
>
> Mit freundlichen Gr??en / Kind regards
>
> Dr. Uwe Falke
> IT Specialist
> Hybrid Cloud Infrastructure / Technology Consulting & Implementation
> Services
> +49 175 575 2877 Mobile
> Rochlitzer Str. 19, 09111 Chemnitz, Germany
> uwefalke at de.ibm.com
>
> IBM Services
>
> IBM Data Privacy Statement
>
> IBM Deutschland Business & Technology Services GmbH
> Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
> Sitz der Gesellschaft: Ehningen
> Registergericht: Amtsgericht Stuttgart, HRB 17122
>
>
>
> From:   Jan-Frode Myklebust <janfrode at tanso.net>
> To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date:   28/05/2021 19:50
> Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM
> Storwize V5030
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
>
>
>
>
> One thing to check: Storwize/SVC code will *always* guess wrong on
> prefetching for GPFS. You can see this with having a lot higher read data

> throughput on mdisk vs. on on vdisks in the webui. To fix it, disable
> cache_prefetch with "chsystem -cache_prefetch off".
>
> This being a global setting, you probably only should set it if the
system
> is only used for GPFS.
>
>
>   -jf
>
> On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
> oluwasijibomi.saula at ndsu.edu> wrote:
> Hi Folks,
>
> So, we are experiencing some very long IO waiters in our GPFS cluster:
>
> #  mmdiag --waiters
>
> === mmdiag: waiters ===
> Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread:
for
> I/O completion
> Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread:
for
> I/O completion
> Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread:
for
> I/O completion
> Waiting 15.2013 sec since 10:41:03, monitored, thread 22175
>
> However, GPFS support is pointing to our IBM Storwize V5030 disk system
as
> the source of latency. Unfortunately, we don't have paid support for the
> system so we are polling for anyone who might be able to assist.
>
> Does anyone by chance have any experience with IBM Storwize V5030 or
> possess a problem determination guide for the V5030?
>
> We've briefly reviewed the V5030 management portal, but we still haven't
> identified a cause for the increased latencies (i.e. read ~129ms, write
> ~198ms).
>
> Granted, we have some heavy client workloads, yet we seem to experience
> this drastic drop in performance every couple of months, probably
> exacerbated by heavy IO demands.
>
> Any assistance would be much appreciated.
>
>
> Thanks,
>
> (Siji) Saula
> HPC Systems Administrator  /  Information Technology
>
> Research 2 Building 220B / Fargo ND 58108-6050
> p: 701.231.7749 / www.ndsu.edu
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/49d91d85/attachment-0001.htm>

From Philipp.Rehs at uni-duesseldorf.de  Sat May  1 13:12:05 2021
From: Philipp.Rehs at uni-duesseldorf.de (Philipp Rehs)
Date: Sat, 1 May 2021 14:12:05 +0200
Subject: [gpfsug-discuss] reinstalled gpfs gui but no admin user created
Message-ID: <7c1e1de0-3a98-e5d5-f424-cb2c71b23a77@uni-duesseldorf.de>

Hello,

today our gpfs gui crashed because of a broken hard drive and i needed
to reinstall the node. It is a legacy system still at 4.2.3.24

After the installation the gui is stuck at "Loading" and in the logs i
see the following errors:

Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]


For me it looks like it is related to this thread but it has no solution.

http://www.spectrumscale.org/pipermail/gpfsug-discuss/2017-December/004319.html

I already tried to install older versions of the gui but it did not help.

Is it possible to remove all old configurations even from ccr?

Kind regards
 Philipp

-- 
---------------------------

Zentrum f?r Informations- und Medientechnologie
Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern

Heinrich-Heine-Universit?t D?sseldorf
Universit?tsstr. 1
Raum 25.41.00.51
40225 D?sseldorf / Germany
Tel: +49-211-81-15557


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5523 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210501/18bb7e32/attachment-0002.bin>

From scale at us.ibm.com  Mon May  3 12:04:57 2021
From: scale at us.ibm.com (IBM Spectrum Scale)
Date: Mon, 3 May 2021 16:34:57 +0530
Subject: [gpfsug-discuss] reinstalled gpfs gui but no admin user created
In-Reply-To: <7c1e1de0-3a98-e5d5-f424-cb2c71b23a77@uni-duesseldorf.de>
References: <7c1e1de0-3a98-e5d5-f424-cb2c71b23a77@uni-duesseldorf.de>
Message-ID: <OFC801667D.40F5A0FE-ON852586CA.003CE36A-652586CA.003CE108@notes.na.collabserv.com>


Hi Ratan,

Can you please look into this GUI issue.

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------

If you feel that your question can benefit other users of  Spectrum Scale
(GPFS), then please post it to the public IBM developerWroks Forum at
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479.


If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact
1-800-237-5511 in the United States or your local IBM Service Center in
other countries.

The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.



From:	Philipp Rehs <Philipp.Rehs at uni-duesseldorf.de>
To:	gpfsug-discuss at spectrumscale.org
Date:	01-05-2021 05.50 PM
Subject:	[EXTERNAL] [gpfsug-discuss] reinstalled gpfs gui but no admin
            user created
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Hello,

today our gpfs gui crashed because of a broken hard drive and i needed
to reinstall the node. It is a legacy system still at 4.2.3.24

After the installation the gui is stuck at "Loading" and in the logs i
see the following errors:

Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings
(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin
(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet
(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service
(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter
(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter
(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: [ERROR   ] SRVE0777E:
Exception thrown by application class
'com.ibm.gss.gui.beans.PasswordPolicyBean.<init>:73'
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: java.lang.NullPointerException
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.beans.PasswordPolicyBean.<init>(PasswordPolicyBean.java:73)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicy(AccessRPC.java:166)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.logic.AccessRPC.getPasswordPolicySettings
(AccessRPC.java:199)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm._jsp._login._jspService(_login.java:193)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:100)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.forwardToLogin
(AuthenticationHandler.java:502)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.AuthenticationHandler.doGet
(AuthenticationHandler.java:146)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service
(ServletWrapper.java:1255)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.evo.servlets.filters.ServletFilter.doFilter(ServletFilter.java:99)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.gss.gui.servlets.filters.ServletFilter.doFilter
(ServletFilter.java:50)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter
(FilterInstanceWrapper.java:201)
Mai 01 14:10:18 hpc-gpfs-gui numactl[7769]: at [internal classes]


For me it looks like it is related to this thread but it has no solution.

http://www.spectrumscale.org/pipermail/gpfsug-discuss/2017-December/004319.html


I already tried to install older versions of the gui but it did not help.

Is it possible to remove all old configurations even from ccr?

Kind regards
 Philipp

--
---------------------------

Zentrum f?r Informations- und Medientechnologie
Kompetenzzentrum f?r wissenschaftliches Rechnen und Speichern

Heinrich-Heine-Universit?t D?sseldorf
Universit?tsstr. 1
Raum 25.41.00.51
40225 D?sseldorf / Germany
Tel: +49-211-81-15557


[attachment "smime.p7s" deleted by Huzefa H Pancha/India/IBM]
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210503/66e5866c/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210503/66e5866c/attachment-0002.gif>

From committee at io500.org  Wed May  5 17:38:45 2021
From: committee at io500.org (IO500 Committee)
Date: Wed, 05 May 2021 10:38:45 -0600
Subject: [gpfsug-discuss] Call For Submissions IO500 ISC21 List
Message-ID: <561bba3f84c7ef935c1dd67004bec46a@io500.org>

https://io500.org/cfs
Stabilization Period: 05 - 14 May 2021 AoE
Submission Deadline: 11 June 2021 AoE

The IO500 is now accepting and encouraging submissions for the upcoming 
8th IO500 list. Once again, we are also accepting submissions to the 10 
Node Challenge to encourage the submission of small scale results. The 
new ranked lists will be announced via live-stream at a virtual session. 
We hope to see many new results.

What's New
Starting with ISC'21, the IO500 now follows a two-staged approach. 
First, there will be a two-week stabilization period during which we 
encourage the community to verify that the benchmark runs properly. 
During this period the benchmark will be updated based upon feedback 
from the community. The final benchmark will then be released on Monday, 
May 1st. We expect that runs compliant with the rules made during the 
stabilization period are valid as the final submission unless a 
significant defect is found.
We are now creating a more detailed schema to describe the hardware and 
software of the system under test and provide the first set of tools to 
ease capturing of this information for inclusion with the submission. 
Further details will be released on the submission page.

Background
The benchmark suite is designed to be easy to run and the community has 
multiple active support channels to help with any questions. Please note 
that submissions of all sizes are welcome; the site has customizable 
sorting, so it is possible to submit on a small system and still get a 
very good per-client score, for example. Additionally, the list is about 
much more than just the raw rank; all submissions help the community by 
collecting and publishing a wider corpus of data. More details below.

Following the success of the Top500 in collecting and analyzing 
historical trends in supercomputer technology and evolution, the IO500 
was created in 2017, published its first list at SC17, and has grown 
exponentially since then. The need for such an initiative has long been 
known within High-Performance Computing; however, defining appropriate 
benchmarks had long been challenging. Despite this challenge, the 
community, after long and spirited discussion, finally reached consensus 
on a suite of benchmarks and a metric for resolving the scores into a 
single ranking.

The multi-fold goals of the benchmark suite are as follows:

Maximizing simplicity in running the benchmark suite
Encouraging optimization and documentation of tuning parameters for 
performance
Allowing submitters to highlight their "hero run" performance numbers
Forcing submitters to simultaneously report performance for challenging 
IO patterns.

Specifically, the benchmark suite includes a hero-run of both IOR and 
mdtest configured however possible to maximize performance and establish 
an upper-bound for performance. It also includes an IOR and mdtest run 
with highly prescribed parameters in an attempt to determine a 
lower-bound. Finally, it includes a namespace search as this has been 
determined to be a highly sought-after feature in HPC storage systems 
that has historically not been well-measured. Submitters are encouraged 
to share their tuning insights for publication.

The goals of the community are also multi-fold:

Gather historical data for the sake of analysis and to aid predictions 
of storage futures
Collect tuning information to share valuable performance optimizations 
across the community
Encourage vendors and designers to optimize for workloads beyond "hero 
runs"
Establish bounded expectations for users, procurers, and administrators

10 Node I/O Challenge
The 10 Node Challenge is conducted using the regular IO500 benchmark, 
however, with the rule that exactly 10 client nodes must be used to run 
the benchmark. You may use any shared storage with, e.g., any number of 
servers. When submitting for the IO500 list, you can opt-in for 
"Participate in the 10 compute node challenge only", then we will not 
include the results into the ranked list. Other 10-node node submissions 
will be included in the full list and in the ranked list. We will 
announce the result in a separate derived list and in the full list but 
not on the ranked IO500 list at io500.org.

Birds-of-a-Feather
Once again, we encourage you to submit to join our community, and to 
attend our virtual BoF "The IO500 and the Virtual Institute of I/O" at 
ISC 2021, (time to be announced), where we will announce the new IO500 
and 10 node challenge lists. The current list includes results from 
BeeGFS, CephFS, DAOS, DataWarp, GekkoFS, GFarm, IME, Lustre, MadFS, 
Qumulo, Spectrum Scale, Vast, WekaIO, and YRCloudFile. We hope that the 
upcoming list grows even more.


-- 
The IO500 Committee


From heinrich.billich at id.ethz.ch  Mon May 10 13:16:57 2021
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Mon, 10 May 2021 12:16:57 +0000
Subject: [gpfsug-discuss] migrate to new metadata GNR systems while
 maintaining a fallback possibilty
Message-ID: <62CAB006-A4B1-4E19-9ADD-A3DA5E9C3BC2@id.ethz.ch>

Hello,

 

I need to update/replace our metadata disks but I want to keep old and new in parallel for a while before I remove the old storage: as active/active pair with double copies. This will allow an immediate fall-back if we ever need.

 

Maybe you want to comment on this procedure ? I probably found it some time ago on this mailing list, sorry if I don?t remember the author.

 

Some concerns I have

I?m a bit worried what happens when we remove the vdisks with the second copy in the last step ? will be get millions of error messages? ?Is the sequence of commands right? 

I need to change the failure groups of some vdisks while in use ? I wonder if this poses some risk? As I understand this will change the order of block allocation among the nsds (not the allocation map I guess)

This will double metadata write-io, the systems should be able to handle it. 

We will get? better metadata read-io during the transition than what we?ll finally get.

 

== Start ? ?old? ESS/GNR only

 

 -m 1 (single metadata copy), 

?-M 2

 -K whenpossible

 Metadata vdisks in failure groups 1 and 2

 

== Preparation

 

Use? mmvdisk filesystems? and move all metadata vdisk-sets to a single failuregroup (this has to be done in operation, under load)

Set ?-m 2, while we still have one failure group only. -K whenpossible will keep us running

Add the new vdisk-set with a second failure group on ?new? ESS/GNR systems

Now all new inodes have two copies, one on old and one on new

 

== Create copies on ?old? and ?new?

 

 mmrestripefs -R with ?qos maintenance and -N helper-nodes to minimize the load. This may create some locks on the filesystem/cluster and interfere with backup and snapshots??

Maybe better: use a policy ?replicate? rule to replicate all files, I can run this in small chunks and run mmrestripefs just once to crosscheck.

 

== Observe for some days, handle remaining filesystems in the same way

 

== Finally

 

Suspend ?old? disks

Run mmrestripefs -m

Remove ?old? vdisk sets with mmvidisk ? this will run another mmdeldisk

Change to -m 1

Run fix replication setting mmrestripefs -R (if needed?)

 

Thank you for reading and for any comments or suggestions.

 

Heiner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210510/bf8bb6f9/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5254 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210510/bf8bb6f9/attachment-0002.bin>

From S.J.Thompson at bham.ac.uk  Mon May 10 18:08:24 2021
From: S.J.Thompson at bham.ac.uk (Simon Thompson)
Date: Mon, 10 May 2021 17:08:24 +0000
Subject: [gpfsug-discuss] UK UG survey
Message-ID: <A2712CC4-3877-41E3-9B9C-159D10776C74@bham.ac.uk>

I?m currently trying to gather some feedback on running an in-person SSUG event in the UK later this year ? (subject to rules etc etc). If you would normally attend the event in the UK (or you would be very likely to), I?d appreciate 3 mins of time to fill out a quick survey to gather opinion ?

https://www.surveymonkey.co.uk/r/TVWVWGC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210510/e8dabada/attachment-0002.htm>

From mutantllama at gmail.com  Wed May 12 04:40:50 2021
From: mutantllama at gmail.com (Carl)
Date: Wed, 12 May 2021 13:40:50 +1000
Subject: [gpfsug-discuss] Experience running CrowdStrike with Spectrum Scale
Message-ID: <CAGedhdf0bi0np1z-bpMgEzvumuK3j2ZbOgFV0wQfh3842Mb06w@mail.gmail.com>

Hi folks,

We are currently looking at deploying CrowdStrike on our login nodes.
Has anyone had experience with running CrowdStrike with Spectrum
Scale? Did you have any issues?

Cheers,

Carl.


From TROPPENS at de.ibm.com  Fri May 14 12:52:25 2021
From: TROPPENS at de.ibm.com (Ulf Troppens)
Date: Fri, 14 May 2021 11:52:25 +0000
Subject: [gpfsug-discuss] Charts of German User Meeting are now available
Message-ID: <OF8E05C51D.D4F014E9-ON002586D5.00412744-002586D5.004139A2@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210514/b26ee9ec/attachment-0002.htm>

From chair at spectrumscale.org  Sun May 16 19:14:05 2021
From: chair at spectrumscale.org (Simon Thompson (Spectrum Scale User Group Chair))
Date: Sun, 16 May 2021 19:14:05 +0100
Subject: [gpfsug-discuss] SSUG::Digital: What is new in Spectrum Scale 5.1.1?
Message-ID: <>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210516/d3275165/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: meeting.ics
Type: text/calendar
Size: 2418 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210516/d3275165/attachment-0002.ics>

From Kritika.Gulati at ibm.com  Tue May 18 15:29:45 2021
From: Kritika.Gulati at ibm.com (Kritika Gulati)
Date: Tue, 18 May 2021 14:29:45 +0000
Subject: [gpfsug-discuss] reinstalled gpfs gui but no admin user created
Message-ID: <OF8971FFD2.1980C2A9-ON002586D9.004EED6E-002586D9.004FA113@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210518/8fc48e6a/attachment-0002.htm>

From davebond7787 at gmail.com  Thu May 20 13:56:51 2021
From: davebond7787 at gmail.com (Dave Bond)
Date: Thu, 20 May 2021 13:56:51 +0100
Subject: [gpfsug-discuss] FAL audit events not always triggered
In-Reply-To: <CAHuryA0f86GG33D9_0+YVtSnMVfBFoPcaQq6z4BjD9xB8M0T6A@mail.gmail.com>
References: <CAHuryA0f86GG33D9_0+YVtSnMVfBFoPcaQq6z4BjD9xB8M0T6A@mail.gmail.com>
Message-ID: <CAHuryA2ymNHGkLJE=20UMcpnaOq6XOHgjLnM9hEU0ZPN_r_BRw@mail.gmail.com>

Hello

I am currently testing out the FAL and watch folders auditing mechanism in
5.1.0.  I have noticed that audit events such as file access or creation
are not always recorded in the FAL log on the filesystem.  It would seem
necessary to disable and re enable FAL auditing for the same file access to
be recorded.  I have only triggered this condition twice so far, but it has
set in my mind a few questions.

1) Have others seen this?  If so what is the trigger, as I do not yet have
a reproducer.
2) Is it intended for the audit mechanism to be a soft audit?  i.e the
audit is best effort, and non blocking for file access.

This is using a single NSD and a single remote cluster doing the file
writing.  FAL is running on the NSD node.

Regards

Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/c4895935/attachment-0002.htm>

From davebond7787 at gmail.com  Thu May 20 13:58:01 2021
From: davebond7787 at gmail.com (Dave Bond)
Date: Thu, 20 May 2021 13:58:01 +0100
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <CAHuryA11v_7AWSssSoKzD8PLb8gyNeGsoZe39hZPCN=TDcJ8Pw@mail.gmail.com>
References: <CAHuryA11v_7AWSssSoKzD8PLb8gyNeGsoZe39hZPCN=TDcJ8Pw@mail.gmail.com>
Message-ID: <CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>

Hello

As part of a project I am doing I am looking if there are any de
duplication options for GPFS?  I see there is no native de dupe for the
filesystem. The scenario would be user A creates a file or folder and user
B takes a copy within the same filesystem, though separate independent
filesets.  The intention would be to store 1 copy.    So I was wondering
....

1) Is this is planned to be implemented into GPFS in the future?
2) Is anyone is using any other solutions that have good GPFS integration?

Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/71a20baa/attachment-0002.htm>

From abeattie at au1.ibm.com  Thu May 20 14:27:57 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Thu, 20 May 2021 13:27:57 +0000
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>
Message-ID: <OF162CFDAD.3050490E-ON002586DB.0049F835-1621517277192@notes.na.collabserv.com>

Dave,

Spectrum Scale does not support de-duplication, it does support compression. 
You can however use block storage that supports over subscription / thin provisioning / deduplication for data only NSD?s, we do not recommend them for metadata.

In your scenario is user B planning on making changes to the data which is why you need a copy? 

I know of customers that do this regularly with block storage such as the IBM Flashsystem product family In conjunction with IBM Spectrum Copy Data Management.  But I don?t believe CDM supports file based storage.

Regards, 

Andrew Beattie
Technical Sales Specialist
Storage for Data and AI
IBM Australia and New Zealand
P. +61 421 337 927
E. Abeattie at au1.ibm.com

> On 20 May 2021, at 22:58, Dave Bond <davebond7787 at gmail.com> wrote:
> 
> ?
> This Message Is From an External Sender
> This message came from outside your organization.
> 
> 
> Hello
> 
> As part of a project I am doing I am looking if there are any de duplication options for GPFS?  I see there is no native de dupe for the filesystem. The scenario would be user A creates a file or folder and user B takes a copy within the same filesystem, though separate independent filesets.  The intention would be to store 1 copy.    So I was wondering ....
> 
> 1) Is this is planned to be implemented into GPFS in the future?
> 2) Is anyone is using any other solutions that have good GPFS integration?
> 
> Dave

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/670d8c55/attachment-0002.htm>

From Luis.I.Teran at ibm.com  Thu May 20 23:26:36 2021
From: Luis.I.Teran at ibm.com (Luis I Teran)
Date: Thu, 20 May 2021 22:26:36 +0000
Subject: [gpfsug-discuss] FAL audit events not always triggered
Message-ID: <OF7630AC17.4C49DFA1-ON002586DB.00792014-002586DB.007B494D@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/3336ec92/attachment-0002.htm>

From ulmer at ulmer.org  Fri May 21 00:42:30 2021
From: ulmer at ulmer.org (Stephen Ulmer)
Date: Thu, 20 May 2021 19:42:30 -0400
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <OF162CFDAD.3050490E-ON002586DB.0049F835-1621517277192@notes.na.collabserv.com>
References: <OF162CFDAD.3050490E-ON002586DB.0049F835-1621517277192@notes.na.collabserv.com>
Message-ID: <16E2BBAA-3DC2-41B7-83C2-F875CDA3D57E@ulmer.org>

Do file clones meet the workflow requirement? That is, can you control from whence the second (and further) copies are made?

 -- 
Stephen


> On May 20, 2021, at 9:01 AM, Andrew Beattie <abeattie at au1.ibm.com> wrote:
> 
> ?Dave,
> 
> Spectrum Scale does not support de-duplication, it does support compression. 
> You can however use block storage that supports over subscription / thin provisioning / deduplication for data only NSD?s, we do not recommend them for metadata.
> 
> In your scenario is user B planning on making changes to the data which is why you need a copy? 
> 
> I know of customers that do this regularly with block storage such as the IBM Flashsystem product family In conjunction with IBM Spectrum Copy Data Management.  But I don?t believe CDM supports file based storage.
> 
> Regards, 
> 
> Andrew Beattie
> Technical Sales Specialist
> Storage for Data and AI
> IBM Australia and New Zealand
> P. +61 421 337 927
> E. Abeattie at au1.ibm.com
> 
>>> On 20 May 2021, at 22:58, Dave Bond <davebond7787 at gmail.com> wrote:
>>> 
>> ?
>> 
>> 
>> Hello
>> 
>> As part of a project I am doing I am looking if there are any de duplication options for GPFS?  I see there is no native de dupe for the filesystem. The scenario would be user A creates a file or folder and user B takes a copy within the same filesystem, though separate independent filesets.  The intention would be to store 1 copy.    So I was wondering ....
>> 
>> 1) Is this is planned to be implemented into GPFS in the future?
>> 2) Is anyone is using any other solutions that have good GPFS integration?
>> 
>> Dave
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210520/ce7a47ee/attachment-0002.htm>

From david.reynolds at tes-es.com  Fri May 21 09:42:11 2021
From: david.reynolds at tes-es.com (David Reynolds)
Date: Fri, 21 May 2021 08:42:11 +0000
Subject: [gpfsug-discuss] Spectrum Scale & S3
Message-ID: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>

When we talk about supported protocols on Scale, S3 is listed but its normally prefixed with Object. Does this mean that the S3 connectivity is used purely to connect and migrate data in object stores to parallel systems (and vice versa) or can you connect to applications/workflows using S3?

David Reynolds
TES Enterprise Solutions
david.reynolds at tes-es.com<mailto:david.reynolds at tes-es.com>
07594 524986

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210521/4d7c6b04/attachment-0002.htm>

From jonathan.buzzard at strath.ac.uk  Fri May 21 09:57:23 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Fri, 21 May 2021 09:57:23 +0100
Subject: [gpfsug-discuss] GPFS de duplication
In-Reply-To: <CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>
References: <CAHuryA11v_7AWSssSoKzD8PLb8gyNeGsoZe39hZPCN=TDcJ8Pw@mail.gmail.com>
	<CAHuryA0i0RGv43Oo025B3f6=+hN0AjdpHuQ4pwFAONN6FPoc9A@mail.gmail.com>
Message-ID: <616926ad-e16b-2e7d-d45d-da3a803ae185@strath.ac.uk>

On 20/05/2021 13:58, Dave Bond wrote:

> As part of a project I am doing I am looking if there are any de 
> duplication options for GPFS?? I see there is no native de dupe for the 
> filesystem. The scenario would be user A creates a file or folder and 
> user B takes a copy within the same filesystem, though separate 
> independent filesets.? The intention would be to store 1 copy.? ? So I 
> was wondering ....
> 
> 1) Is this is planned to be implemented into GPFS in the future?
> 2) Is anyone is using any other solutions that have good GPFS integration?
> 

Disk space in 2021 is insanely cheap. With an ESS/DSS you can get many 
PB in a single rack. The complexity that dedup introduces is simple not 
worth it IMHO.

Or put another way there is better things the developers at IBM can be 
working on than dedup code.

Historically if you crunched the numbers the licensing for dedup on 
NetApp was similar to just buying more disk unless you where storing 
hundreds of copies of the same data. About the only use case scenario 
would be storing lots of virtual machines. However I refer you back to 
my original point :-)


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From abeattie at au1.ibm.com  Fri May 21 10:56:38 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Fri, 21 May 2021 09:56:38 +0000
Subject: [gpfsug-discuss] Spectrum Scale & S3
In-Reply-To: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>
Message-ID: <OF964F942C.E8EC157A-ON002586DC.00369F4C-1621590998632@notes.na.collabserv.com>

David,

You can use the CES protocol nodes to present a swift Object interface or S3 Object interface for applications to access data from the filesystem / write data into the filesystem over the object protocol. 

Check the openstack Swift -S3 compatabioity matrix to make sure that the functionality from swift aligns with your App requirements 

Sent from my iPhone

> On 21 May 2021, at 18:42, David Reynolds <david.reynolds at tes-es.com> wrote:
> 
> ?
> This Message Is From an External Sender
> This message came from outside your organization.
> When we talk about supported protocols on Scale, S3 is listed but its normally prefixed with Object. Does this mean that the S3 connectivity is used purely to connect and migrate data in object stores to parallel systems (and vice versa) or can you connect to applications/workflows using S3?
>  
> David Reynolds
> TES Enterprise Solutions
> david.reynolds at tes-es.com
> 07594 524986
>  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210521/173855a9/attachment-0002.htm>

From janfrode at tanso.net  Fri May 21 15:16:17 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 21 May 2021 16:16:17 +0200
Subject: [gpfsug-discuss] Spectrum Scale & S3
In-Reply-To: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>
References: <LO3P123MB31777341A2303753A85E8FE5A8299@LO3P123MB3177.GBRP123.PROD.OUTLOOK.COM>
Message-ID: <CAHwPath6YvRcJhijqHoedsXXqgx6GTBvOW82rDvseM18fDU9uA@mail.gmail.com>

It has features for both being an Object store for other applications
(running openstack swift/S3), and for migrating/tiering filesystem data to
an object store like Amazon S3, IBM COS, etc...


  -jf

fre. 21. mai 2021 kl. 10:42 skrev David Reynolds <david.reynolds at tes-es.com
>:

> When we talk about supported protocols on Scale, S3 is listed but its
> normally prefixed with Object. Does this mean that the S3 connectivity is
> used purely to connect and migrate data in object stores to parallel
> systems (and vice versa) or can you connect to applications/workflows using
> S3?
>
>
>
> David Reynolds
>
> TES Enterprise Solutions
>
> david.reynolds at tes-es.com
>
> 07594 524986
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210521/a8eb7280/attachment-0002.htm>

From tomasz.rachobinski at gmail.com  Mon May 24 14:06:01 2021
From: tomasz.rachobinski at gmail.com (=?UTF-8?Q?Tomasz_Rachobi=C5=84ski?=)
Date: Mon, 24 May 2021 15:06:01 +0200
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0
Message-ID: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>

Hello everyone,
we are trying to implement a mixed linux/windows environment and we have
one thing at the top - is there any global method to avoid asynchronous I/O
and write everything in synchronous mode?
Another thing is - if there is no global sync setting, how to enforce sync
i/o from linux/windows client?

Greetings
Tom Rachobinski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210524/be2a202b/attachment-0002.htm>

From Alec.Effrat at wellsfargo.com  Mon May 24 18:23:38 2021
From: Alec.Effrat at wellsfargo.com (Alec.Effrat at wellsfargo.com)
Date: Mon, 24 May 2021 17:23:38 +0000
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0
In-Reply-To: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
References: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
Message-ID: <22a8b9b6fa474a62927ed5e215a548c6@wellsfargo.com>

In the past we?d used EMC RecoverPoint for a Spectrum Scale file system.  It isn?t officially supported, and it doesn?t provide for an online replica, but it can keep the I/O synchronous and DVR style file system playback and we had written our own integrations for it via it?s SSH interface.  It is a very competent solution and does pair well with GPFS Spectrum Scale.

Our Spectrum Scale interface basically had to do the following:

1)      Ship updates to the mmdrfs file to the target cluster.

2)      In control failover we would tag the image once GPFS was stopped

3)      Come up on the tagged image on BCP

4)      Had an awk script that would strip out all the host details from the mmdrfs file and update it to 1 host in the BCP cluster.

5)      Import the GPFS file system using the modified mmdrfs file.

Just one way to skin the cat for consideration.  Can provide more actuals on the replication scripts if interested.

Alec Effrat
SAS Lead, AVP
Business Intelligence Competency Center
SAS Administration
Cell 949-246-7713
alec.effrat at wellsfargo.com<mailto:alec.effrat at wellsfargo.com>




From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Tomasz Rachobinski
Sent: Monday, May 24, 2021 6:06 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0

Hello everyone,
we are trying to implement a mixed linux/windows environment and we have one thing at the top - is there any global method to avoid asynchronous I/O and write everything in synchronous mode?
Another thing is - if there is no global sync setting, how to enforce sync i/o from linux/windows client?

Greetings
Tom Rachobinski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210524/75bb9936/attachment-0002.htm>

From krajaram at geocomputing.net  Mon May 24 22:39:59 2021
From: krajaram at geocomputing.net (Kumaran Rajaram)
Date: Mon, 24 May 2021 21:39:59 +0000
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0
In-Reply-To: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
References: <CAOnndBM9aQEQAD6Uso8J8MEUDVmTaD24ffR=eRnfcoxJu_oupA@mail.gmail.com>
Message-ID: <SJ0PR18MB4011C053C80A70BEA944CACCBB269@SJ0PR18MB4011.namprd18.prod.outlook.com>

Hi Tom,

>>we are trying to implement a mixed linux/windows environment and we have one thing at the top - is there any global method to avoid asynchronous I/O and write everything in >>synchronous mode?

If the local and remote sites have good inter-site network bandwidth and low-latency, then you may consider using GPFS synchronous replication at the file-system level (-m 2 -r 2).  The Spectrum Scale documentation (link below) has further details.

https://www.ibm.com/docs/en/spectrum-scale/5.1.0?topic=data-synchronous-mirroring-gpfs-replication

Regards,
-Kums

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Tomasz Rachobinski
Sent: Monday, May 24, 2021 9:06 AM
To: gpfsug-discuss at spectrumscale.org
Subject: [gpfsug-discuss] Spectrum Scale - how to get RPO=0

Hello everyone,
we are trying to implement a mixed linux/windows environment and we have one thing at the top - is there any global method to avoid asynchronous I/O and write everything in synchronous mode?
Another thing is - if there is no global sync setting, how to enforce sync i/o from linux/windows client?

Greetings
Tom Rachobinski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210524/bf198bcd/attachment-0002.htm>

From coetzee.ray at gmail.com  Wed May 26 18:33:20 2021
From: coetzee.ray at gmail.com (Ray Coetzee)
Date: Wed, 26 May 2021 18:33:20 +0100
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
Message-ID: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>

Hello all

I'd be interested to know if anyone else has experienced a problem with Kernel
> 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).

We noticed that python shut.copy() is failing against a GPFS mount with:

BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
-> 'test2.file'

To reproduce the error:

```
[user at login01]$ module load python-3.8.9-gcc-9.3.0-soqwnzh

[ user at login01]$ truncate --size 640MB test.file
[ user at login01]$ python3 -c "import shutil; shutil.copy('test.file',
'test2.file')"
Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 418, in copy
 copyfile(src, dst, follow_symlinks=follow_symlinks)
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 275, in copyfile
 _fastcopy_sendfile(fsrc, fdst)
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 172, in _fastcopy_sendfile
 raise err
 File
"/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
line 152, in _fastcopy_sendfile
 sent = os.sendfile(outfd, infd, offset, blocksize)
BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
-> 'test2.file'



 Investigating into why this is happening revealed that:


1. It is failing for python3.8 and above.
2. It is happening only a GPFS mount
3. It is happening with files whose size is multiple of 4KB (OS Page size)

Relevant links:
https://bugs.python.org/issue43743
https://www.ibm.com/support/pages/apar/IJ28891


Doing an strace revealed that at the lower level, it seems to be related to
the Linux Syscall of ?sendfile?, which seems to fail in these cases on GPFS.


Strace for a 4096 B file:

```
sendfile(4, 3, [0] => [4096], 8388608) = 4096
sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily
unavailable)

```

The same file on other disk:
```
sendfile(4, 3, [0] => [4096], 8388608) = 4096
sendfile(4, 3, [4096], 8388608) = 0


IBM's "fix" for the problem of "Do not use a file size which that is a
multiple of the page size."  sounds really blas?.


```


Kind regards

Ray Coetzee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/cf991430/attachment-0002.htm>

From knop at us.ibm.com  Wed May 26 18:45:38 2021
From: knop at us.ibm.com (Felipe Knop)
Date: Wed, 26 May 2021 17:45:38 +0000
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
References: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
Message-ID: <OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/ff9d3d0c/attachment-0002.htm>

From coetzee.ray at gmail.com  Wed May 26 19:27:10 2021
From: coetzee.ray at gmail.com (Ray Coetzee)
Date: Wed, 26 May 2021 19:27:10 +0100
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
References: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
	<OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
Message-ID: <CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>

Hi Felipe

We looked at APAR  IJ28891 & IJ29942, as both look identical but are closed
as a "program error" with a workaround of "Do not use a file size which
that is a multiple of the page size."

Kind regards

Ray Coetzee


On Wed, May 26, 2021 at 6:45 PM Felipe Knop <knop at us.ibm.com> wrote:

> Ray,
>
> I wonder if you are hitting the problem which was fixed on the following
> APAR:
>
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
>   Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
>
>
> ----- Original message -----
> From: Ray Coetzee <coetzee.ray at gmail.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
> Date: Wed, May 26, 2021 1:38 PM
>
> Hello all
>
> I'd be interested to know if anyone else has experienced a problem with Kernel
> > 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).
>
>
> We noticed that python shut.copy() is failing against a GPFS mount with:
>
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
> To reproduce the error:
>
> ```
> [user at login01]$ module load python-3.8.9-gcc-9.3.0-soqwnzh
>
> [ user at login01]$ truncate --size 640MB test.file
> [ user at login01]$ python3 -c "import shutil; shutil.copy('test.file',
> 'test2.file')"
> Traceback (most recent call last):
>  File "<string>", line 1, in <module>
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 418, in copy
>  copyfile(src, dst, follow_symlinks=follow_symlinks)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 275, in copyfile
>  _fastcopy_sendfile(fsrc, fdst)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 172, in _fastcopy_sendfile
>  raise err
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 152, in _fastcopy_sendfile
>  sent = os.sendfile(outfd, infd, offset, blocksize)
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
>
>
>  Investigating into why this is happening revealed that:
>
>
> 1. It is failing for python3.8 and above.
> 2. It is happening only a GPFS mount
> 3. It is happening with files whose size is multiple of 4KB (OS Page size)
>
> Relevant links:
> https://bugs.python.org/issue43743
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
> Doing an strace revealed that at the lower level, it seems to be related
> to the Linux Syscall of ?sendfile?, which seems to fail in these cases on
> GPFS.
>
>
> Strace for a 4096 B file:
>
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily
> unavailable)
>
> ```
>
> The same file on other disk:
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = 0
>
>
>
> IBM's "fix" for the problem of "Do not use a file size which that is a
> multiple of the page size."  sounds really blas?.
>
>
> ```
>
>
> Kind regards
>
> Ray Coetzee
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/450af1a7/attachment-0002.htm>

From knop at us.ibm.com  Wed May 26 19:36:40 2021
From: knop at us.ibm.com (Felipe Knop)
Date: Wed, 26 May 2021 18:36:40 +0000
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>
References: <CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>,
	<CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
	<OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
Message-ID: <OFC1A21264.1ACD353E-ON002586E1.0065CC6B-002586E1.00663C06@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/29fc799d/attachment-0002.htm>

From coetzee.ray at gmail.com  Wed May 26 21:21:32 2021
From: coetzee.ray at gmail.com (Ray Coetzee)
Date: Wed, 26 May 2021 21:21:32 +0100
Subject: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
In-Reply-To: <OFC1A21264.1ACD353E-ON002586E1.0065CC6B-002586E1.00663C06@notes.na.collabserv.com>
References: <CABQRq0Hqpk0Xumdg2R-fSZY6pxmyhd7CKJUf8iNfQODdjg4s3w@mail.gmail.com>
	<OFEB88B7BA.2599F325-ON002586E1.00615B47-002586E1.00618FE7@notes.na.collabserv.com>
	<CABQRq0H1POyHhfQ57Lnhku4L+QgPsatDEQd+Nk5UPKOqRCpcOw@mail.gmail.com>
	<OFC1A21264.1ACD353E-ON002586E1.0065CC6B-002586E1.00663C06@notes.na.collabserv.com>
Message-ID: <CABQRq0EAO5aXKFR+UY28mjUrphOTMrH-orC2meLxRr1ReoZ5vg@mail.gmail.com>

Thank you for the info Felipe.

I'll test with 5.0.5-7 in the morning.

Kind regards

Ray Coetzee




On Wed, May 26, 2021 at 7:36 PM Felipe Knop <knop at us.ibm.com> wrote:

> Ray,
>
> Apologies; I should have added more details. My records show that there is
> a fix for IJ28891 and IJ29942, which was delivered in
>
> 5.1.0.2
> 5.0.5.5
>
> "Program error" I believe is a code that indicates "needs to be fixed in
> our product".
>
> I agree that the workaround mentioned is not "actionable" . The APAR page
> should have been clear that there is a fix available.
>
> Regards,
>
>   Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
>
>
> ----- Original message -----
> From: Ray Coetzee <coetzee.ray at gmail.com>
> To: Felipe Knop <knop at us.ibm.com>
> Cc: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
> Date: Wed, May 26, 2021 2:27 PM
>
> Hi Felipe We looked at APAR IJ28891 & IJ29942, as both look identical but
> are closed as a "program error" with a workaround of "Do not use a file
> size which that is a multiple of the page size." Kind regards ? ? ? ? ? ? ?
> ? ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
> ZjQcmQRYFpfptBannerEnd
> Hi Felipe
>
> We looked at APAR  IJ28891 & IJ29942, as both look identical but are
> closed as a "program error" with a workaround of "Do not use a file size
> which that is a multiple of the page size."
>
> Kind regards
>
> Ray Coetzee
>
>
> On Wed, May 26, 2021 at 6:45 PM Felipe Knop <knop at us.ibm.com> wrote:
>
> Ray,
>
> I wonder if you are hitting the problem which was fixed on the following
> APAR:
>
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
>   Felipe
>
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>
>
>
>
> ----- Original message -----
> From: Ray Coetzee <coetzee.ray at gmail.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] Kernel > 4.10, python >= 3.8 issue
> Date: Wed, May 26, 2021 1:38 PM
>
> Hello all
>
> I'd be interested to know if anyone else has experienced a problem with Kernel
> > 4.10, python >= 3.8 and Spectrum Scale (5.0.5-2).
>
>
> We noticed that python shut.copy() is failing against a GPFS mount with:
>
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
> To reproduce the error:
>
> ```
> [user at login01]$ module load python-3.8.9-gcc-9.3.0-soqwnzh
>
> [ user at login01]$ truncate --size 640MB test.file
> [ user at login01]$ python3 -c "import shutil; shutil.copy('test.file',
> 'test2.file')"
> Traceback (most recent call last):
>  File "<string>", line 1, in <module>
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 418, in copy
>  copyfile(src, dst, follow_symlinks=follow_symlinks)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 275, in copyfile
>  _fastcopy_sendfile(fsrc, fdst)
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 172, in _fastcopy_sendfile
>  raise err
>  File
> "/hps/software/spack/opt/spack/linux-centos8-sandybridge/gcc-9.3.0/python-3.8.9-soqwnzhndvqpk3mly3w6z6zx6cdv45sn/lib/python3.8/shutil.py",
> line 152, in _fastcopy_sendfile
>  sent = os.sendfile(outfd, infd, offset, blocksize)
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'test.file'
> -> 'test2.file'
>
>
>
>  Investigating into why this is happening revealed that:
>
>
> 1. It is failing for python3.8 and above.
> 2. It is happening only a GPFS mount
> 3. It is happening with files whose size is multiple of 4KB (OS Page size)
>
> Relevant links:
> https://bugs.python.org/issue43743
> https://www.ibm.com/support/pages/apar/IJ28891
>
>
> Doing an strace revealed that at the lower level, it seems to be related
> to the Linux Syscall of ?sendfile?, which seems to fail in these cases on
> GPFS.
>
>
> Strace for a 4096 B file:
>
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = -1 EAGAIN (Resource temporarily
> unavailable)
>
> ```
>
> The same file on other disk:
> ```
> sendfile(4, 3, [0] => [4096], 8388608) = 4096
> sendfile(4, 3, [4096], 8388608) = 0
>
>
>
> IBM's "fix" for the problem of "Do not use a file size which that is a
> multiple of the page size."  sounds really blas?.
>
>
> ```
>
>
> Kind regards
>
> Ray Coetzee
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/9096ca4a/attachment-0002.htm>

From pbasmaji at us.ibm.com  Thu May 27 00:59:23 2021
From: pbasmaji at us.ibm.com (Peter Basmajian)
Date: Wed, 26 May 2021 23:59:23 +0000
Subject: [gpfsug-discuss] Be a Spectrum Scale *private* reference,
	get a Spectrum Scale t-shirt!
Message-ID: <OF0163861D.E8273E15-ON002586E1.007605D4-002586E1.0083C7E7@ibm.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/06eba1f6/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Image.1622066097985.png
Type: image/png
Size: 155627 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/06eba1f6/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Image.1622066115686.png
Type: image/png
Size: 133174 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210526/06eba1f6/attachment-0005.png>

From henrik at morsing.cc  Thu May 27 16:10:39 2021
From: henrik at morsing.cc (Henrik Morsing)
Date: Thu, 27 May 2021 16:10:39 +0100
Subject: [gpfsug-discuss] Ransom attacks
Message-ID: <20210527151039.GC8399@morsing.cc>


Hi,

It struck me that switching a Spectrum Protect solution from tapes to a GPFS filesystem offers much less protection against ransom encryption should the SP server be compromised. Same goes really for compromising an ESS node itself, it is an awful lot of data that can be encrypted very quickly.

Is there anything that can protect the GPFS filesystem against this kind of attack?

Regards,
Henrik


From anobre at br.ibm.com  Thu May 27 16:20:08 2021
From: anobre at br.ibm.com (Anderson Ferreira Nobre)
Date: Thu, 27 May 2021 15:20:08 +0000
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <OF65FE85E0.F99DD819-ON002586E2.005411E1-002586E2.00543DD2@ibm.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210527/c5e51dc4/attachment-0002.htm>

From skylar2 at uw.edu  Thu May 27 16:23:48 2021
From: skylar2 at uw.edu (Skylar Thompson)
Date: Thu, 27 May 2021 08:23:48 -0700
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <20210527152348.3h6qmitg2cyhaqus@thargelion>

You can get clever/complicated (the interpretation could go either way)
with ACLs and SELinux but, at the end of the day, nothing beats the air-gap
of tape backups, IMHO. You might consider a belt&suspenders approach that
includes all of the above plus other controls (2FA, network security,
etc.), and in my experience combining multiple solutions gives flexibility
in that it can be easier to avoid the higher-cost aspects of one solution
taken to an extreme by having one layer mitigate the shortcomings of
another layer.

On Thu, May 27, 2021 at 04:10:39PM +0100, Henrik Morsing wrote:
> 
> Hi,
> 
> It struck me that switching a Spectrum Protect solution from tapes to a GPFS filesystem offers much less protection against ransom encryption should the SP server be compromised. Same goes really for compromising an ESS node itself, it is an awful lot of data that can be encrypted very quickly.
> 
> Is there anything that can protect the GPFS filesystem against this kind of attack?

-- 
-- Skylar Thompson (skylar2 at u.washington.edu)
-- Genome Sciences Department (UW Medicine), System Administrator
-- Foege Building S046, (206)-685-7354
-- Pronouns: He/Him/His


From jonathan.buzzard at strath.ac.uk  Thu May 27 16:49:02 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Thu, 27 May 2021 16:49:02 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527152348.3h6qmitg2cyhaqus@thargelion>
References: <20210527151039.GC8399@morsing.cc>
	<20210527152348.3h6qmitg2cyhaqus@thargelion>
Message-ID: <64fa4f59-a73a-19d4-a789-d63c9955cf64@strath.ac.uk>

On 27/05/2021 16:23, Skylar Thompson wrote:

[SNIP]

> at the end of the day, nothing beats the air-gap of tape backups, IMHO. 

Changing/deleting lots of data on tape takes time. So tape is a really 
good starting point even if you never take the tapes out the library 
except to dispose of them. Your backup is your get out of jail card. 
Protect it like it's Fort Knox.

A bit of security through obscurity by using Power and AIX will not go 
amiss. Even if it only buys you a couple of hours that can be enough to 
save the backup from harm.

Passwords on the Spectrum Protect server should be good *never* be used 
anywhere else, and only a handful of trusted people should have access 
to them.

Make sure you have a reuse delay on those tapes so even if the bastards 
do a del filespace (if they even know how to use TSM) you can roll back 
the database.

I also have the notion that you should be able to cut the power to the 
Spectrum Protect server and tape libraries such that it requires an on 
site visit to manually power them backup by flicking a nice big molly 
switch. I have a notion in my mind of tripping a residual-current 
device/ground fault circuit interrupter by using a relay to create a 
neutral earth fault. First sign of trouble disconnect the backup system :-)


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From rltodd.ml1 at gmail.com  Thu May 27 19:17:37 2021
From: rltodd.ml1 at gmail.com (Lindsay Todd)
Date: Thu, 27 May 2021 14:17:37 -0400
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>

Henrik,

Generally you need to begin with a good backup or replica, as well as
suitable air-gaps to isolate contamination.  You also need to be able to
quickly detect unusual activity - an SIEM tool like QRadar might help.
Assume that a cyber-incident will happen and plan accordingly.  Use
in-depth security.  But you are right - you lose one of the advantages of
tape - you can make duplicate copies, maybe even a WORM copy, and store it
offsite.

You might at very least want to take snapshots of the storage being used by
Spectrum Protect, and have separate administrators for the ESS and SP
server (to reduce inside risk).  If it was actually GPFS being backed up to
SP, you could have a second GPFS file system that is a point-in-time
synchronized copy of the original GPFS file system - with its own
snapshots.  It could have yet another sysadmin, and you could isolate the
second copy from the network when not actively synchronizing. See
https://www.redbooks.ibm.com/abstracts/redp5559.html?Open

That might not make sense if GPFS is holding the SP backup data, but SP can
do its own replication too - and could replicate using storage from a
second GPFS file system off-site.  Take snapshots of this second storage,
as well as SP database, and again manage with a second sysadmin team.


*Lindsay Todd, PhD*
*Spectrum Scale (GPFS) Solution Architect*
*IBM Advanced Technology Group ? Storage*
*Mobile:** 1-518-369-6108*
*E-mail:* *lindsay at us.ibm.com* <lindsay at us.ibm.com>


On Thu, May 27, 2021 at 11:10 AM Henrik Morsing <henrik at morsing.cc> wrote:

>
> Hi,
>
> It struck me that switching a Spectrum Protect solution from tapes to a
> GPFS filesystem offers much less protection against ransom encryption
> should the SP server be compromised. Same goes really for compromising an
> ESS node itself, it is an awful lot of data that can be encrypted very
> quickly.
>
> Is there anything that can protect the GPFS filesystem against this kind
> of attack?
>
> Regards,
> Henrik
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210527/966d8a32/attachment-0002.htm>

From henrik at morsing.cc  Fri May 28 07:46:35 2021
From: henrik at morsing.cc (Henrik Morsing)
Date: Fri, 28 May 2021 07:46:35 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>
References: <20210527151039.GC8399@morsing.cc>
	<CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>
Message-ID: <20210528064635.GD8399@morsing.cc>

On Thu, May 27, 2021 at 02:17:37PM -0400, Lindsay Todd wrote:
>
>That might not make sense if GPFS is holding the SP backup data, but SP can
>do its own replication too - and could replicate using storage from a
>second GPFS file system off-site.  Take snapshots of this second storage,
>as well as SP database, and again manage with a second sysadmin team.
>

Thanks all for some useful replies, something to take forward.

In this case, SP is using GPFS for storing backup data, this solution was meant to replace the tape libraries completely.

We protect the storage pools cross-site, but our solutions are identical, so if you hacked one, you have hacked both.

Regards,
Henrik


From henrik at morsing.cc  Fri May 28 08:15:37 2021
From: henrik at morsing.cc (Henrik Morsing)
Date: Fri, 28 May 2021 08:15:37 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <OF65FE85E0.F99DD819-ON002586E2.005411E1-002586E2.00543DD2@ibm.com>
References: <20210527151039.GC8399@morsing.cc>
	<OF65FE85E0.F99DD819-ON002586E2.005411E1-002586E2.00543DD2@ibm.com>
Message-ID: <20210528071537.GE8399@morsing.cc>

On Thu, May 27, 2021 at 03:20:08PM +0000, Anderson Ferreira Nobre wrote:
>   Henrik,
>   ?
>   One way would integrate Scale with QRadar. If I'm not wrong, you can
>   configure QRadar to take a snapshot when it detects there's an attack
>   happening. The details you can take from here:
>   [1]https://www.redbooks.ibm.com/redpapers/pdfs/redp5560.pdf
>   [2]https://www.youtube.com/watch?v=Zyw84dvoFR8
>   ?

Hi,

Looking at the video (not read the document yet) I'm not sure QRadar is advanced enough to detect someone encrypting a storage pool from the SP server. It's a single file pretty much access 24x7, but I will look into it further, thanks.

Regards,
Henrik


From jonathan.buzzard at strath.ac.uk  Fri May 28 09:07:12 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Fri, 28 May 2021 09:07:12 +0100
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210528064635.GD8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
	<CA+Ah3z=LQYA78QTTXPkO=We3xzQ0zN_SupKoiBfP48Q15ZgA-w@mail.gmail.com>
	<20210528064635.GD8399@morsing.cc>
Message-ID: <546408b3-0617-f293-8d15-c36f776f5bd8@strath.ac.uk>

On 28/05/2021 07:46, Henrik Morsing wrote:
>>
>> That might not make sense if GPFS is holding the SP backup data, but 
>> SP can do its own replication too - and could replicate using storage from a
>> second GPFS file system off-site.? Take snapshots of this second storage,
>> as well as SP database, and again manage with a second sysadmin team.
>>
> 
> Thanks all for some useful replies, something to take forward.
> 
> In this case, SP is using GPFS for storing backup data, this solution 
> was meant to replace the tape libraries completely.
> 

If your backup is for disaster recovery that's fine. If you expand your 
disaster to include ransom attacks then disk based backups are IMHO 
inadequate simply because they can be gone forever in the blink of an eye.

> We protect the storage pools cross-site, but our solutions are 
> identical, so if you hacked one, you have hacked both.
> 

Currently we use a home grown disk based system for the backup (home 
grown because it's cheap) however we are looking to augment it with tape 
because tape is firstly ransom attack resistant, second tape is "green" 
with a very low carbon footprint.

 From a TSM perspective backup goes to the disk run as a bunch of 
sequential access files like "tapes", and the copy pool will exists on 
tape. We get the benefit of having the backup on disk aka the short 
access times to files, with the protection offered by tape should we get 
hit by a ransom attack.

JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From macthev at gmail.com  Fri May 28 09:23:08 2021
From: macthev at gmail.com (macthev at gmail.com)
Date: Fri, 28 May 2021 18:23:08 +1000
Subject: [gpfsug-discuss] Ransom attacks
In-Reply-To: <20210527151039.GC8399@morsing.cc>
References: <20210527151039.GC8399@morsing.cc>
Message-ID: <326E4E15-014F-4A28-8EBB-1295CF899859@gmail.com>

Take a look at IAM nodes.

Sent from my iPhone

> On 28 May 2021, at 01:10, Henrik Morsing <henrik at morsing.cc> wrote:
> 
> ?
> Hi,
> 
> It struck me that switching a Spectrum Protect solution from tapes to a GPFS filesystem offers much less protection against ransom encryption should the SP server be compromised. Same goes really for compromising an ESS node itself, it is an awful lot of data that can be encrypted very quickly.
> 
> Is there anything that can protect the GPFS filesystem against this kind of attack?
> 
> Regards,
> Henrik
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From heinrich.billich at id.ethz.ch  Fri May 28 09:25:20 2021
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Fri, 28 May 2021 08:25:20 +0000
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to estimate
 remaining execution time
Message-ID: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>

 Hello,

 

I want to estimate how much longer a running 

 

mmrestripefs -R ?metadata-only ?qos maintenance

 

job will take to finish. We did switch from ?-m 1? to ?-m 2 ? and now run mmrestripefs to get the second copy of all metadata. I know that the % values in the output are useless, but now it looks like the number of inodes processed is of no use, too? The filesystem has about 1e9 allocated inodes, but the mmrestripefs reports up to? 7e9 inodes processed? The number of inodes reported increases heavily since it passed the number of allocated inodes???

 

?? 4.41 % complete on Fri May 28 09:22:05 2021? (7219160122 inodes with total??? 8985067 MB data processed)

 

 

Should I consider the ?MB data processed? instead ? will the job finish when all used metadata disk space is processed? 

 

I see little iops on the metadata disks and wonder if something is stuck, I?m not sure if it makes sense to wait any longer. But without an estimate on how much longer the job will take I hesitate to restart. A restart will need to scan all metadata again ? Metadata is on SSD or NVMe, disks iops never were limiting. I turned qos off on the filesystem, previously I restricted the iops. But I don?t see any increase in iops.

 

Thank you. I know this is an old issue, but still I would strongly welcome any advice or comment.

 

Cheers,

 

Heiner

 

I?m aware of RFE ID:150409 mmrestripefs % complete metric is near useless. I?m not sure if it covers the metadata-only case, too.

 

Inode Information

-----------------

Total number of used inodes in all Inode spaces:????????? 735005692

Total number of free inodes in all Inode spaces:????????? 335462660

Total number of allocated inodes in all Inode spaces:??? 1070468352

Total of Maximum number of inodes in all Inode spaces:?? 1223782912

 

Mmrestripefs ?output:

 

START Mon May 24 20:27:25 CEST 2021

Scanning file system metadata, phase 1 ...

 100 % complete on Mon May 24 21:49:27 2021

Scan completed successfully.

Scanning file system metadata, phase 2 ...

 100 % complete on Mon May 24 21:49:28 2021

Scanning file system metadata for data storage pool

? 90 % complete on Mon May 24 21:49:34 2021

 100 % complete on Mon May 24 21:49:35 2021

Scanning file system metadata for Capacity storage pool

? 49 % complete on Mon May 24 21:49:41 2021

 100 % complete on Mon May 24 21:49:46 2021

Scan completed successfully.

Scanning file system metadata, phase 3 ...

Scan completed successfully.

Scanning file system metadata, phase 4 ...

 100 % complete on Mon May 24 21:49:48 2021

Scan completed successfully.

Scanning file system metadata, phase 5 ...

 100 % complete on Mon May 24 21:49:52 2021

Scan completed successfully.

Scanning user file metadata ...

?? 0.01 % complete on Mon May 24 21:50:13 2021? (??? 613108 inodes with total????? 76693 MB data processed)

?? 0.01 % complete on Mon May 24 21:50:33 2021? (?? 1090048 inodes with total????? 80495 MB data processed)

?? 0.01 % complete on Mon May 24 21:50:53 2021? (?? 1591808 inodes with total????? 84576 MB data processed)

?

? 3.97 % complete on Thu May 27 22:01:02 2021? (1048352497 inodes with total??? 8480467 MB data processed)

?? 3.99 % complete on Thu May 27 22:30:18 2021? (1050254855 inodes with total??? 8495969 MB data processed)

?? 4.01 % complete on Thu May 27 22:59:39 2021? (1052304294 inodes with total??? 8512683 MB data processed)

?? 4.03 % complete on Thu May 27 23:29:11 2021? (1055390220 inodes with total??? 8537615 MB data processed)

?? 4.05 % complete on Thu May 27 23:58:58 2021? (1059333989 inodes with total??? 8568871 MB data processed)

?? 4.07 % complete on Fri May 28 00:28:48 2021? (1064728403 inodes with total??? 8611605 MB data processed)

?? 4.09 % complete on Fri May 28 00:58:50 2021? (1067749260 inodes with total??? 8636120 MB data processed). <<< approximately number of allocated inodes

 

?? 4.11 % complete on Fri May 28 01:29:00 2021? (1488665433 inodes with total??? 8661588 MB data processed) <<< whats going??

?? 4.13 % complete on Fri May 28 01:59:23 2021? (1851124480 inodes with total??? 8682324 MB data processed)

?? 4.15 % complete on Fri May 28 02:29:55 2021? (1885948840 inodes with total??? 8700082 MB data processed)

?? 4.17 % complete on Fri May 28 03:00:38 2021? (2604503808 inodes with total?? ?8724069 MB data processed)

?? 4.19 % complete on Fri May 28 03:31:38 2021? (2877196260 inodes with total??? 8746504 MB data processed)

?? 4.21 % complete on Fri May 28 04:02:38 2021? (2933166080 inodes with total??? 8762555 MB data processed)

?? 4.23 % complete on Fri May 28 04:33:48 2021? (2956295936 inodes with total??? 8782298 MB data processed)

?? 4.25 % complete on Fri May 28 05:05:09 2021? (3628799151 inodes with total??? 8802452 MB data processed)

?? 4.27 % complete on Fri May 28 05:36:40 2021? (3970093965 inodes with total??? 8823885 MB data processed)

?? 4.29 % complete on Fri May 28 06:08:20 2021? (4012553472 inodes with total??? 8841407 MB data processed)

?? 4.31 % complete on Fri May 28 06:40:11 2021? (4029545087 inodes with total??? 8858676 MB data processed)

?? 4.33 % complete on Fri May 28 07:12:11 2021? (6080613874 inodes with total??? 8889395 MB data processed)

?? 4.35 % complete on Fri May 28 07:44:21 2021? (6146937531 inodes with total??? 8907253 MB data processed)

?? 4.37 % complete on Fri May 28 08:16:45 2021? (6167408718 inodes with total??? 8925236 MB data processed)

?? 4.39 % complete on Fri May 28 08:49:18 2021? (7151592448 inodes with total??? 8968126 MB data processed)

?? 4.41 % complete on Fri May 28 09:22:05 2021? (7219160122 inodes with total??? 8985067 MB data processed)

 

 

# mmdf fsxxxx -m --block-size M

disk??????????????? disk size? failure holds??? holds?????????? free in MB????????? free in MB

name??????????????????? in MB??? group metadata data??????? in full blocks??????? in fragments

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 32.20 TB)

fsxxxx_rg07b_1??????? 2861295? ??????3 yes????? no????????? 1684452 ( 59%)???????? 35886 ( 1%). << ?1?140?957MB used ? old nsd

fsxxxx_rg07a_1??????? 2861295??????? 3 yes????? no????????? 1685002 ( 59%)???????? 35883 ( 1%

fsxxxx_rg03b_1??????? 2861295??????? 3 yes????? no????????? 1681515 ( 59%)???????? 35627 ( 1%)

fsxxxx_rg03a_1??????? 2861295??????? 3 yes????? no????????? 1680859 ( 59%)???????? 35651 ( 1%)

 

RG003LG004VS015?????? 2046239?????? 12 yes????? no?????????? 904369 ( 44%)?????????? 114 ( 0%) << 1?141?756MB used ? newly added nsd

RG003LG003VS015?????? 2046239?????? 12 yes????? no?????????? 904291 ( 44%)?????????? 118 ( 0%)

RG003LG002VS015?????? 2046239?????? 12 yes????? no?????????? 903632 ( 44%)?????????? 114 ( 0%)

RG003LG001VS015?????? 2046239?????? 12 yes????? no?????????? 903247 ( 44%)?????????? 115 ( 0%)

??????????????? -------------??????? ?????????????????-------------------- -------------------

(pool total)???????? 19630136????????????????????????????? 10347367 ( 53%)??????? 143503 ( 1%)

 

We run spectrum scale 5.0.5.4/5.0.5.5 on Power. 

 

Filesystem:

 

-f???????????????? 32768??????????????????? Minimum fragment (subblock) size in bytes

 -i???????????????? 4096???????????????????? Inode size in bytes

 -I???????????????? 32768??????????????????? Indirect block size in bytes

 -m???????????????? 2??????????????????????? Default number of metadata replicas

 -M???????????????? 2??????????????????????? Maximum number of metadata replicas

 -r???????????????? 1??????????????????????? Default number of data replicas

 -R???????????????? 2??????????????????????? Maximum number of data replicas

 -j???????????????? scatter????????????????? Block allocation type

 

-V???????????????? 23.00 (5.0.5.0)????????? Current file system version

???????????????? ???15.01 (4.2.0.0)????????? Original file system version

 

-L???????????????? 33554432???????????????? Logfile size

 

--subblocks-per-full-block 32?????????????? Number of subblocks per full block

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/0f901305/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5254 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/0f901305/attachment-0002.bin>

From heinrich.billich at id.ethz.ch  Fri May 28 09:55:53 2021
From: heinrich.billich at id.ethz.ch (Billich  Heinrich Rainer (ID SD))
Date: Fri, 28 May 2021 08:55:53 +0000
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to
 estimate remaining execution time
In-Reply-To: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>
References: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>
Message-ID: <A54D4506-01BF-4E0D-8AA6-194BA3CA89B6@id.ethz.ch>

Hello,

 

I just noticed:

 

Maybe mmrestripefs does some extra processing on snapshots? The output looks much more as expected on filesystems with no snapshots present, both number of inodes and MB data processed allow to estimate the remaining runtime. ?Unfortunately all our large production filesystems use snapshots ?

 

Cheers,

 

Heiner

 

 

?? 1.23 % complete on Tue May 18 21:07:10 2021? (? 37981765 inodes with total???? 328617 MB data processed)

?? 1.25 % complete on Tue May 18 21:17:44 2021? (? 39439584 inodes with total???? 341032 MB data processed)

 100.00 % complete on Tue May 18 21:22:44 2021? (? 41312000 inodes with total???? 356088 MB data processed).??? <<<< # of inodes matches # of allocated inodes, MB processed matches used metadata disk space

 

Scan completed successfully.

# mmdf fsxxx -F

Inode Information

-----------------

Total number of used inodes in all Inode spaces:?????????? 29007227

Total number of free inodes in all Inode spaces:?????????? 12304773

Total number of allocated inodes in all Inode spaces:????? 41312000????? <<<<<<<<<< allocated inodes

Total of Maximum number of inodes in all Inode spaces:???? 87323648

 

 

]# mmdf fsxxx -m --block-size M

disk??????????????? disk size? failure holds??? holds?????????? free in MB????????? free in MB

name??????????????????? in MB??? group metadata data??????? in full blocks??????? in fragments

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 4.13 TB)

??????????????? -------------???????????????????????? -------------------- -------------------

(pool total)????????? 2425152?????????????????????????????? 2067720 ( 85%)????????? 1342 ( 0%).? <<<<<< 356190MB used

 

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Billich Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
Reply to: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Friday, 28 May 2021 at 10:25
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to estimate remaining execution time

 

Hello,

 

I want to estimate how much longer a running 

 

mmrestripefs -R ?metadata-only ?qos maintenance

 

job will take to finish. We did switch from ?-m 1? to ?-m 2 ? and now run mmrestripefs to get the second copy of all metadata. I know that the % values in the output are useless, but now it looks like the number of inodes processed is of no use, too? The filesystem has about 1e9 allocated inodes, but the mmrestripefs reports up to  7e9 inodes processed? The number of inodes reported increases heavily since it passed the number of allocated inodes???

 

   4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with total    8985067 MB data processed)

 

 

Should I consider the ?MB data processed? instead ? will the job finish when all used metadata disk space is processed? 

 

I see little iops on the metadata disks and wonder if something is stuck, I?m not sure if it makes sense to wait any longer. But without an estimate on how much longer the job will take I hesitate to restart. A restart will need to scan all metadata again ? Metadata is on SSD or NVMe, disks iops never were limiting. I turned qos off on the filesystem, previously I restricted the iops. But I don?t see any increase in iops.

 

Thank you. I know this is an old issue, but still I would strongly welcome any advice or comment.

 

Cheers,

 

Heiner

 

I?m aware of RFE ID:150409 mmrestripefs % complete metric is near useless. I?m not sure if it covers the metadata-only case, too.

 

Inode Information

-----------------

Total number of used inodes in all Inode spaces:          735005692

Total number of free inodes in all Inode spaces:          335462660

Total number of allocated inodes in all Inode spaces:    1070468352

Total of Maximum number of inodes in all Inode spaces:   1223782912

 

Mmrestripefs  output:

 

START Mon May 24 20:27:25 CEST 2021

Scanning file system metadata, phase 1 ...

100 % complete on Mon May 24 21:49:27 2021

Scan completed successfully.

Scanning file system metadata, phase 2 ...

100 % complete on Mon May 24 21:49:28 2021

Scanning file system metadata for data storage pool

  90 % complete on Mon May 24 21:49:34 2021

100 % complete on Mon May 24 21:49:35 2021

Scanning file system metadata for Capacity storage pool

  49 % complete on Mon May 24 21:49:41 2021

100 % complete on Mon May 24 21:49:46 2021

Scan completed successfully.

Scanning file system metadata, phase 3 ...

Scan completed successfully.

Scanning file system metadata, phase 4 ...

100 % complete on Mon May 24 21:49:48 2021

Scan completed successfully.

Scanning file system metadata, phase 5 ...

100 % complete on Mon May 24 21:49:52 2021

Scan completed successfully.

Scanning user file metadata ...

   0.01 % complete on Mon May 24 21:50:13 2021  (    613108 inodes with total      76693 MB data processed)

   0.01 % complete on Mon May 24 21:50:33 2021  (   1090048 inodes with total      80495 MB data processed)

   0.01 % complete on Mon May 24 21:50:53 2021  (   1591808 inodes with total      84576 MB data processed)

?

  3.97 % complete on Thu May 27 22:01:02 2021  (1048352497 inodes with total    8480467 MB data processed)

   3.99 % complete on Thu May 27 22:30:18 2021  (1050254855 inodes with total    8495969 MB data processed)

   4.01 % complete on Thu May 27 22:59:39 2021  (1052304294 inodes with total    8512683 MB data processed)

   4.03 % complete on Thu May 27 23:29:11 2021  (1055390220 inodes with total    8537615 MB data processed)

   4.05 % complete on Thu May 27 23:58:58 2021  (1059333989 inodes with total    8568871 MB data processed)

   4.07 % complete on Fri May 28 00:28:48 2021  (1064728403 inodes with total    8611605 MB data processed)

   4.09 % complete on Fri May 28 00:58:50 2021  (1067749260 inodes with total    8636120 MB data processed). <<< approximately number of allocated inodes

 

   4.11 % complete on Fri May 28 01:29:00 2021  (1488665433 inodes with total    8661588 MB data processed) <<< whats going??

   4.13 % complete on Fri May 28 01:59:23 2021  (1851124480 inodes with total    8682324 MB data processed)

   4.15 % complete on Fri May 28 02:29:55 2021  (1885948840 inodes with total    8700082 MB data processed)

   4.17 % complete on Fri May 28 03:00:38 2021  (2604503808 inodes with total    8724069 MB data processed)

   4.19 % complete on Fri May 28 03:31:38 2021  (2877196260 inodes with total    8746504 MB data processed)

   4.21 % complete on Fri May 28 04:02:38 2021  (2933166080 inodes with total    8762555 MB data processed)

   4.23 % complete on Fri May 28 04:33:48 2021  (2956295936 inodes with total    8782298 MB data processed)

   4.25 % complete on Fri May 28 05:05:09 2021  (3628799151 inodes with total    8802452 MB data processed)

   4.27 % complete on Fri May 28 05:36:40 2021  (3970093965 inodes with total    8823885 MB data processed)

   4.29 % complete on Fri May 28 06:08:20 2021  (4012553472 inodes with total    8841407 MB data processed)

   4.31 % complete on Fri May 28 06:40:11 2021  (4029545087 inodes with total    8858676 MB data processed)

   4.33 % complete on Fri May 28 07:12:11 2021  (6080613874 inodes with total    8889395 MB data processed)

   4.35 % complete on Fri May 28 07:44:21 2021  (6146937531 inodes with total    8907253 MB data processed)

   4.37 % complete on Fri May 28 08:16:45 2021  (6167408718 inodes with total    8925236 MB data processed)

   4.39 % complete on Fri May 28 08:49:18 2021  (7151592448 inodes with total    8968126 MB data processed)

   4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with total    8985067 MB data processed)

 

 

# mmdf fsxxxx -m --block-size M

disk                disk size  failure holds    holds           free in MB          free in MB

name                    in MB    group metadata data        in full blocks        in fragments

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 32.20 TB)

fsxxxx_rg07b_1        2861295        3 yes      no          1684452 ( 59%)         35886 ( 1%). <<  1?140?957MB used ? old nsd

fsxxxx_rg07a_1        2861295        3 yes      no          1685002 ( 59%)         35883 ( 1%

fsxxxx_rg03b_1        2861295        3 yes      no          1681515 ( 59%)         35627 ( 1%)

fsxxxx_rg03a_1        2861295        3 yes      no          1680859 ( 59%)         35651 ( 1%)

 

RG003LG004VS015       2046239       12 yes      no           904369 ( 44%)           114 ( 0%) << 1?141?756MB used ? newly added nsd

RG003LG003VS015       2046239       12 yes      no           904291 ( 44%)           118 ( 0%)

RG003LG002VS015       2046239       12 yes      no           903632 ( 44%)           114 ( 0%)

RG003LG001VS015       2046239       12 yes      no           903247 ( 44%)           115 ( 0%)

                -------------                         -------------------- -------------------

(pool total)         19630136                              10347367 ( 53%)        143503 ( 1%)

 

We run spectrum scale 5.0.5.4/5.0.5.5 on Power. 

 

Filesystem:

 

-f                 32768                    Minimum fragment (subblock) size in bytes

-i                 4096                     Inode size in bytes

-I                 32768                    Indirect block size in bytes

-m                 2                        Default number of metadata replicas

-M                 2                        Maximum number of metadata replicas

-r                 1                        Default number of data replicas

-R                 2                        Maximum number of data replicas

-j                 scatter                  Block allocation type

 

-V                 23.00 (5.0.5.0)          Current file system version

                    15.01 (4.2.0.0)          Original file system version

 

-L                 33554432                 Logfile size

 

--subblocks-per-full-block 32               Number of subblocks per full block

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/ec0c5063/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5254 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/ec0c5063/attachment-0002.bin>

From oluwasijibomi.saula at ndsu.edu  Fri May 28 16:57:52 2021
From: oluwasijibomi.saula at ndsu.edu (Saula, Oluwasijibomi)
Date: Fri, 28 May 2021 15:57:52 +0000
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
Message-ID: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>

Hi Folks,

So, we are experiencing some very long IO waiters in our GPFS cluster:


#  mmdiag --waiters


=== mmdiag: waiters ===

Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for I/O completion

Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for I/O completion

Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for I/O completion

Waiting 15.2013 sec since 10:41:03, monitored, thread 22175

However, GPFS support is pointing to our IBM Storwize V5030 disk system as the source of latency. Unfortunately, we don't have paid support for the system so we are polling for anyone who might be able to assist.

Does anyone by chance have any experience with IBM Storwize V5030 or possess a problem determination guide for the V5030?

We've briefly reviewed the V5030 management portal, but we still haven't identified a cause for the increased latencies (i.e. read ~129ms, write ~198ms).

Granted, we have some heavy client workloads, yet we seem to experience this drastic drop in performance every couple of months, probably exacerbated by heavy IO demands.

Any assistance would be much appreciated.



Thanks,


Oluwasijibomi (Siji) Saula

HPC Systems Administrator  /  Information Technology



Research 2 Building 220B / Fargo ND 58108-6050

p: 701.231.7749 / www.ndsu.edu<http://www.ndsu.edu/>



[cid:image001.gif at 01D57DE0.91C300C0]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/f778d057/attachment-0002.htm>

From erich at uw.edu  Fri May 28 17:25:49 2021
From: erich at uw.edu (Eric Horst)
Date: Fri, 28 May 2021 09:25:49 -0700
Subject: [gpfsug-discuss] Mmrestripefs -R --metadat-only - how to
 estimate remaining execution time
In-Reply-To: <A54D4506-01BF-4E0D-8AA6-194BA3CA89B6@id.ethz.ch>
References: <17F8A237-CAC2-4BE1-8239-780BAEEDF265@id.ethz.ch>
	<A54D4506-01BF-4E0D-8AA6-194BA3CA89B6@id.ethz.ch>
Message-ID: <CAF-To-2FFJHvt7OzSz5gfMur9bKtpG7ixLJtgT6bvLToRRO+TQ@mail.gmail.com>

Yes Heiner, my experience is that the inode count in those operations is:
inodes * snapshots = total. I observed that as it starts processing the
snapshot inodes it moves faster as inode usage in snapshots is more sparse.

-Eric

On Fri, May 28, 2021 at 1:56 AM Billich Heinrich Rainer (ID SD) <
heinrich.billich at id.ethz.ch> wrote:

> Hello,
>
>
>
> I just noticed:
>
>
>
> Maybe mmrestripefs does some extra processing on snapshots? The output
> looks much more as expected on filesystems with no snapshots present, both
> number of inodes and MB data processed allow to estimate the remaining
> runtime.  Unfortunately all our large production filesystems use snapshots ?
>
>
>
> Cheers,
>
>
>
> Heiner
>
>
>
>
>
>    1.23 % complete on Tue May 18 21:07:10 2021  (  37981765 inodes with
> total     328617 MB data processed)
>
>    1.25 % complete on Tue May 18 21:17:44 2021  (  39439584 inodes with
> total     341032 MB data processed)
>
> 100.00 % complete on Tue May 18 21:22:44 2021  (  41312000 inodes with
> total     356088 MB data processed).    <<<< # of inodes matches # of
> allocated inodes, MB processed matches used metadata disk space
>
>
>
> Scan completed successfully.
>
> # mmdf fsxxx -F
>
> Inode Information
>
> -----------------
>
> Total number of used inodes in all Inode spaces:           29007227
>
> Total number of free inodes in all Inode spaces:           12304773
>
> Total number of allocated inodes in all Inode spaces:      41312000
> <<<<<<<<<< allocated inodes
>
> Total of Maximum number of inodes in all Inode spaces:     87323648
>
>
>
>
>
> ]# mmdf fsxxx -m --block-size M
>
> disk                disk size  failure holds    holds           free in
> MB          free in MB
>
> name                    in MB    group metadata data        in full
> blocks        in fragments
>
> --------------- ------------- -------- -------- ----- --------------------
> -------------------
>
> Disks in storage pool: system (Maximum disk size allowed is 4.13 TB)
>
>                 -------------                         --------------------
> -------------------
>
> (pool total)          2425152                               2067720 (
> 85%)          1342 ( 0%).  <<<<<< 356190MB used
>
>
>
> *From: *<gpfsug-discuss-bounces at spectrumscale.org> on behalf of "Billich
> Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
> *Reply to: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Date: *Friday, 28 May 2021 at 10:25
> *To: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Subject: *[gpfsug-discuss] Mmrestripefs -R --metadat-only - how to
> estimate remaining execution time
>
>
>
> Hello,
>
>
>
> I want to estimate how much longer a running
>
>
>
> mmrestripefs -R ?metadata-only ?qos maintenance
>
>
>
> job will take to finish. We did switch from ?-m 1? to ?-m 2 ? and now run
> mmrestripefs to get the second copy of all metadata. I know that the %
> values in the output are useless, but now it looks like the number of
> inodes processed is of no use, too? The filesystem has about 1e9 allocated
> inodes, but the mmrestripefs reports up to  7e9 inodes processed? The
> number of inodes reported increases heavily since it passed the number of
> allocated inodes???
>
>
>
>    4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with
> total    8985067 MB data processed)
>
>
>
>
>
> Should I consider the ?MB data processed? instead ? will the job finish
> when all used metadata disk space is processed?
>
>
>
> I see little iops on the metadata disks and wonder if something is stuck,
> I?m not sure if it makes sense to wait any longer. But without an estimate
> on how much longer the job will take I hesitate to restart. A restart will
> need to scan all metadata again ? Metadata is on SSD or NVMe, disks iops
> never were limiting. I turned qos off on the filesystem, previously I
> restricted the iops. But I don?t see any increase in iops.
>
>
>
> Thank you. I know this is an old issue, but still I would strongly welcome
> any advice or comment.
>
>
>
> Cheers,
>
>
>
> Heiner
>
>
>
> I?m aware of RFE ID:150409 mmrestripefs % complete metric is near useless.
> I?m not sure if it covers the metadata-only case, too.
>
>
>
> Inode Information
>
> -----------------
>
> Total number of used inodes in all Inode spaces:          735005692
>
> Total number of free inodes in all Inode spaces:          335462660
>
> Total number of allocated inodes in all Inode spaces:    1070468352
>
> Total of Maximum number of inodes in all Inode spaces:   1223782912
>
>
>
> Mmrestripefs  output:
>
>
>
> START Mon May 24 20:27:25 CEST 2021
>
> Scanning file system metadata, phase 1 ...
>
> 100 % complete on Mon May 24 21:49:27 2021
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 2 ...
>
> 100 % complete on Mon May 24 21:49:28 2021
>
> Scanning file system metadata for data storage pool
>
>   90 % complete on Mon May 24 21:49:34 2021
>
> 100 % complete on Mon May 24 21:49:35 2021
>
> Scanning file system metadata for Capacity storage pool
>
>   49 % complete on Mon May 24 21:49:41 2021
>
> 100 % complete on Mon May 24 21:49:46 2021
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 3 ...
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 4 ...
>
> 100 % complete on Mon May 24 21:49:48 2021
>
> Scan completed successfully.
>
> Scanning file system metadata, phase 5 ...
>
> 100 % complete on Mon May 24 21:49:52 2021
>
> Scan completed successfully.
>
> Scanning user file metadata ...
>
>    0.01 % complete on Mon May 24 21:50:13 2021  (    613108 inodes with
> total      76693 MB data processed)
>
>    0.01 % complete on Mon May 24 21:50:33 2021  (   1090048 inodes with
> total      80495 MB data processed)
>
>    0.01 % complete on Mon May 24 21:50:53 2021  (   1591808 inodes with
> total      84576 MB data processed)
>
> ?
>
>   3.97 % complete on Thu May 27 22:01:02 2021  (1048352497 inodes with
> total    8480467 MB data processed)
>
>    3.99 % complete on Thu May 27 22:30:18 2021  (1050254855 inodes with
> total    8495969 MB data processed)
>
>    4.01 % complete on Thu May 27 22:59:39 2021  (1052304294 inodes with
> total    8512683 MB data processed)
>
>    4.03 % complete on Thu May 27 23:29:11 2021  (1055390220 inodes with
> total    8537615 MB data processed)
>
>    4.05 % complete on Thu May 27 23:58:58 2021  (1059333989 inodes with
> total    8568871 MB data processed)
>
>    4.07 % complete on Fri May 28 00:28:48 2021  (1064728403 inodes with
> total    8611605 MB data processed)
>
>    4.09 % complete on Fri May 28 00:58:50 2021  (1067749260 inodes with
> total    8636120 MB data processed). <<< approximately number of allocated
> inodes
>
>
>
>    4.11 % complete on Fri May 28 01:29:00 2021  (1488665433 inodes with
> total    8661588 MB data processed) <<< whats going??
>
>    4.13 % complete on Fri May 28 01:59:23 2021  (1851124480 inodes with
> total    8682324 MB data processed)
>
>    4.15 % complete on Fri May 28 02:29:55 2021  (1885948840 inodes with
> total    8700082 MB data processed)
>
>    4.17 % complete on Fri May 28 03:00:38 2021  (2604503808 inodes with
> total    8724069 MB data processed)
>
>    4.19 % complete on Fri May 28 03:31:38 2021  (2877196260 inodes with
> total    8746504 MB data processed)
>
>    4.21 % complete on Fri May 28 04:02:38 2021  (2933166080 inodes with
> total    8762555 MB data processed)
>
>    4.23 % complete on Fri May 28 04:33:48 2021  (2956295936 inodes with
> total    8782298 MB data processed)
>
>    4.25 % complete on Fri May 28 05:05:09 2021  (3628799151 inodes with
> total    8802452 MB data processed)
>
>    4.27 % complete on Fri May 28 05:36:40 2021  (3970093965 inodes with
> total    8823885 MB data processed)
>
>    4.29 % complete on Fri May 28 06:08:20 2021  (4012553472 inodes with
> total    8841407 MB data processed)
>
>    4.31 % complete on Fri May 28 06:40:11 2021  (4029545087 inodes with
> total    8858676 MB data processed)
>
>    4.33 % complete on Fri May 28 07:12:11 2021  (6080613874 inodes with
> total    8889395 MB data processed)
>
>    4.35 % complete on Fri May 28 07:44:21 2021  (6146937531 inodes with
> total    8907253 MB data processed)
>
>    4.37 % complete on Fri May 28 08:16:45 2021  (6167408718 inodes with
> total    8925236 MB data processed)
>
>    4.39 % complete on Fri May 28 08:49:18 2021  (7151592448 inodes with
> total    8968126 MB data processed)
>
>    4.41 % complete on Fri May 28 09:22:05 2021  (7219160122 inodes with
> total    8985067 MB data processed)
>
>
>
>
>
> # mmdf fsxxxx -m --block-size M
>
> disk                disk size  failure holds    holds           free in
> MB          free in MB
>
> name                    in MB    group metadata data        in full
> blocks        in fragments
>
> --------------- ------------- -------- -------- ----- --------------------
> -------------------
>
> Disks in storage pool: system (Maximum disk size allowed is 32.20 TB)
>
> fsxxxx_rg07b_1        2861295        3 yes      no          1684452 (
> 59%)         35886 ( 1%). <<  1?140?957MB used ? old nsd
>
> fsxxxx_rg07a_1        2861295        3 yes      no          1685002 (
> 59%)         35883 ( 1%
>
> fsxxxx_rg03b_1        2861295        3 yes      no          1681515 (
> 59%)         35627 ( 1%)
>
> fsxxxx_rg03a_1        2861295        3 yes      no          1680859 (
> 59%)         35651 ( 1%)
>
>
>
> RG003LG004VS015       2046239       12 yes      no           904369 (
> 44%)           114 ( 0%) << 1?141?756MB used ? newly added nsd
>
> RG003LG003VS015       2046239       12 yes      no           904291 (
> 44%)           118 ( 0%)
>
> RG003LG002VS015       2046239       12 yes      no           903632 (
> 44%)           114 ( 0%)
>
> RG003LG001VS015       2046239       12 yes      no           903247 (
> 44%)           115 ( 0%)
>
>                 -------------                         --------------------
> -------------------
>
> (pool total)         19630136                              10347367 (
> 53%)        143503 ( 1%)
>
>
>
> We run spectrum scale 5.0.5.4/5.0.5.5 on Power.
>
>
>
> Filesystem:
>
>
>
> -f                 32768                    Minimum fragment (subblock)
> size in bytes
>
> -i                 4096                     Inode size in bytes
>
> -I                 32768                    Indirect block size in bytes
>
> -m                 2                        Default number of metadata
> replicas
>
> -M                 2                        Maximum number of metadata
> replicas
>
> -r                 1                        Default number of data replicas
>
> -R                 2                        Maximum number of data replicas
>
> -j                 scatter                  Block allocation type
>
>
>
> -V                 23.00 (5.0.5.0)          Current file system version
>
>                     15.01 (4.2.0.0)          Original file system version
>
>
>
> -L                 33554432                 Logfile size
>
>
>
> --subblocks-per-full-block 32               Number of subblocks per full
> block
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/f46d7a39/attachment-0002.htm>

From janfrode at tanso.net  Fri May 28 18:50:21 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Fri, 28 May 2021 19:50:21 +0200
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>
References: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>
Message-ID: <CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>

One thing to check: Storwize/SVC code will *always* guess wrong on
prefetching for GPFS. You can see this with having a lot higher read data
throughput on mdisk vs. on on vdisks in the webui. To fix it, disable
cache_prefetch with "chsystem -cache_prefetch off".

This being a global setting, you probably only should set it if the system
is only used for GPFS.


   -jf

On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
oluwasijibomi.saula at ndsu.edu> wrote:

> Hi Folks,
>
> So, we are experiencing some very long IO waiters in our GPFS cluster:
>
> #  mmdiag --waiters
>
>
> === mmdiag: waiters ===
>
> Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for
> I/O completion
>
> Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for
> I/O completion
>
> Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for
> I/O completion
>
> Waiting 15.2013 sec since 10:41:03, monitored, thread 22175
>
> However, GPFS support is pointing to our IBM Storwize V5030 disk system
> as the source of latency. Unfortunately, we don't have paid support for the
> system so we are polling for anyone who might be able to assist.
>
> Does anyone by chance have any experience with IBM Storwize V5030 or
> possess a problem determination guide for the V5030?
>
> We've briefly reviewed the V5030 management portal, but we still haven't
> identified a cause for the increased latencies (i.e. read ~129ms, write
> ~198ms).
>
> Granted, we have some heavy client workloads, yet we seem to experience
> this drastic drop in performance every couple of months, probably
> exacerbated by heavy IO demands.
>
> Any assistance would be much appreciated.
>
>
> Thanks,
>
> *Oluwasijibomi (Siji) Saula*
>
> HPC Systems Administrator  /  Information Technology
>
>
>
> Research 2 Building 220B / Fargo ND 58108-6050
>
> p: 701.231.7749 / www.ndsu.edu
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/832c7e75/attachment-0002.htm>

From UWEFALKE at de.ibm.com  Fri May 28 21:04:19 2021
From: UWEFALKE at de.ibm.com (Uwe Falke)
Date: Fri, 28 May 2021 22:04:19 +0200
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>
References: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com>
	<CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>
Message-ID: <OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>

Hi, odd prefetch strategy would affect read performance, but write latency 
is claimed to be even worse ...
Have you simply checked what the actual IO performance of the v5k box 
under that load is and how it compares to its nominal performance and that 
of its disks?
how is the storage organised? how many LUNs/NSDs, what RAID code (V5k 
cannot do declustered RAID, can it?), any thin provisioning or other 
gimmicks in the game?
what IO sizes ?
tons of things to look at. 

Mit freundlichen Gr??en / Kind regards

Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation 
Services
+49 175 575 2877 Mobile
Rochlitzer Str. 19, 09111 Chemnitz, Germany
uwefalke at de.ibm.com

IBM Services

IBM Data Privacy Statement

IBM Deutschland Business & Technology Services GmbH
Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122



From:   Jan-Frode Myklebust <janfrode at tanso.net>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   28/05/2021 19:50
Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM 
Storwize V5030
Sent by:        gpfsug-discuss-bounces at spectrumscale.org




One thing to check: Storwize/SVC code will *always* guess wrong on 
prefetching for GPFS. You can see this with having a lot higher read data 
throughput on mdisk vs. on on vdisks in the webui. To fix it, disable 
cache_prefetch with "chsystem -cache_prefetch off".

This being a global setting, you probably only should set it if the system 
is only used for GPFS. 


   -jf

On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
oluwasijibomi.saula at ndsu.edu> wrote:
Hi Folks,

So, we are experiencing some very long IO waiters in our GPFS cluster:

#  mmdiag --waiters 

=== mmdiag: waiters ===
Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for 
I/O completion
Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for 
I/O completion
Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for 
I/O completion
Waiting 15.2013 sec since 10:41:03, monitored, thread 22175 

However, GPFS support is pointing to our IBM Storwize V5030 disk system as 
the source of latency. Unfortunately, we don't have paid support for the 
system so we are polling for anyone who might be able to assist.

Does anyone by chance have any experience with IBM Storwize V5030 or 
possess a problem determination guide for the V5030?

We've briefly reviewed the V5030 management portal, but we still haven't 
identified a cause for the increased latencies (i.e. read ~129ms, write 
~198ms). 

Granted, we have some heavy client workloads, yet we seem to experience 
this drastic drop in performance every couple of months, probably 
exacerbated by heavy IO demands.

Any assistance would be much appreciated.


Thanks,

Oluwasijibomi (Siji) Saula
HPC Systems Administrator  /  Information Technology
 
Research 2 Building 220B / Fargo ND 58108-6050
p: 701.231.7749 / www.ndsu.edu
 



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 







From UWEFALKE at de.ibm.com  Fri May 28 21:16:12 2021
From: UWEFALKE at de.ibm.com (Uwe Falke)
Date: Fri, 28 May 2021 22:16:12 +0200
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>
References: <PH0PR08MB6598396C19D9D51BAA81197598229@PH0PR08MB6598.namprd08.prod.outlook.com><CAHwPatj_KRQTD0azdtOth=dqErHAu0twzxRsuLTv7RnZG7uswA@mail.gmail.com>
	<OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>
Message-ID: <OF7233E8BC.78902F9C-ONC12586E3.006ED69A-C12586E3.006F5924@notes.na.collabserv.com>

You say you see that every few months: does it mean with about the same 
load sometimes the system knocks out and sometimes it behaves ok?
Have you checked the v5k event log, if there is anything going on (write 
performance may suffer if write cache is off which might happen if the 
buffer batteries are low on charge - but again that does not directly 
explain the high read latency). 
Are the latencies you gave derived from GPFS, from the OS, or from the 
V5k?

Mit freundlichen Gr??en / Kind regards

Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation 
Services
+49 175 575 2877 Mobile
Rochlitzer Str. 19, 09111 Chemnitz, Germany
uwefalke at de.ibm.com

IBM Services

IBM Data Privacy Statement

IBM Deutschland Business & Technology Services GmbH
Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122



From:   "Uwe Falke" <UWEFALKE at de.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   28/05/2021 22:04
Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM 
Storwize V5030
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi, odd prefetch strategy would affect read performance, but write latency 

is claimed to be even worse ...
Have you simply checked what the actual IO performance of the v5k box 
under that load is and how it compares to its nominal performance and that 

of its disks?
how is the storage organised? how many LUNs/NSDs, what RAID code (V5k 
cannot do declustered RAID, can it?), any thin provisioning or other 
gimmicks in the game?
what IO sizes ?
tons of things to look at. 

Mit freundlichen Gr??en / Kind regards

Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation 
Services
+49 175 575 2877 Mobile
Rochlitzer Str. 19, 09111 Chemnitz, Germany
uwefalke at de.ibm.com

IBM Services

IBM Data Privacy Statement

IBM Deutschland Business & Technology Services GmbH
Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122



From:   Jan-Frode Myklebust <janfrode at tanso.net>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   28/05/2021 19:50
Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM 
Storwize V5030
Sent by:        gpfsug-discuss-bounces at spectrumscale.org




One thing to check: Storwize/SVC code will *always* guess wrong on 
prefetching for GPFS. You can see this with having a lot higher read data 
throughput on mdisk vs. on on vdisks in the webui. To fix it, disable 
cache_prefetch with "chsystem -cache_prefetch off".

This being a global setting, you probably only should set it if the system 

is only used for GPFS. 


   -jf

On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
oluwasijibomi.saula at ndsu.edu> wrote:
Hi Folks,

So, we are experiencing some very long IO waiters in our GPFS cluster:

#  mmdiag --waiters 

=== mmdiag: waiters ===
Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for 

I/O completion
Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for 

I/O completion
Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for 

I/O completion
Waiting 15.2013 sec since 10:41:03, monitored, thread 22175 

However, GPFS support is pointing to our IBM Storwize V5030 disk system as 

the source of latency. Unfortunately, we don't have paid support for the 
system so we are polling for anyone who might be able to assist.

Does anyone by chance have any experience with IBM Storwize V5030 or 
possess a problem determination guide for the V5030?

We've briefly reviewed the V5030 management portal, but we still haven't 
identified a cause for the increased latencies (i.e. read ~129ms, write 
~198ms). 

Granted, we have some heavy client workloads, yet we seem to experience 
this drastic drop in performance every couple of months, probably 
exacerbated by heavy IO demands.

Any assistance would be much appreciated.


Thanks,

Oluwasijibomi (Siji) Saula
HPC Systems Administrator  /  Information Technology
 
Research 2 Building 220B / Fargo ND 58108-6050
p: 701.231.7749 / www.ndsu.edu
 



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 





_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 








From abeattie at au1.ibm.com  Fri May 28 22:27:12 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Fri, 28 May 2021 21:27:12 +0000
Subject: [gpfsug-discuss] Long IO waiters and IBM Storwize V5030
In-Reply-To: <OFB56D056E.B4AA64DF-ONC12586E3.006DCD71-C12586E3.006E426D@notes.na.collabserv.com>
Message-ID: <OF09635240.267CCCEC-ON002586E3.0075D8B6-1622237232907@notes.na.collabserv.com>



Hi Oluwasijibomi,

If you set up a Storage Insights Standard account
You can monitor the performance of your 5030, and pull the performance
metrics of the block storage array when you see poor performance in your
scale cluster.
This will give you some idea as to what is happening,
but The 5030 is designed to be a backup / low IOPS Storage controller, the
processing power and memory in the controllers is very limited.

If you have significant workload happening on your file system in terms of
user access (reads / writes) I am not at all surprised your seeing
performance bottleneck from the 5030.

You could ask you local IBM Presales team to perform a StorM disk model of
the expected performance using your current configuration to show you what
you performance should look like.

Regards

Andrew Beattie
Technical Sales - Storage for Data and AI
IBM Australia and New Zealand

> On 29 May 2021, at 06:04, Uwe Falke <UWEFALKE at de.ibm.com> wrote:
>
> Hi, odd prefetch strategy would affect read performance, but write
latency
> is claimed to be even worse ...
> Have you simply checked what the actual IO performance of the v5k box
> under that load is and how it compares to its nominal performance and
that
> of its disks?
> how is the storage organised? how many LUNs/NSDs, what RAID code (V5k
> cannot do declustered RAID, can it?), any thin provisioning or other
> gimmicks in the game?
> what IO sizes ?
> tons of things to look at.
>
> Mit freundlichen Gr??en / Kind regards
>
> Dr. Uwe Falke
> IT Specialist
> Hybrid Cloud Infrastructure / Technology Consulting & Implementation
> Services
> +49 175 575 2877 Mobile
> Rochlitzer Str. 19, 09111 Chemnitz, Germany
> uwefalke at de.ibm.com
>
> IBM Services
>
> IBM Data Privacy Statement
>
> IBM Deutschland Business & Technology Services GmbH
> Gesch?ftsf?hrung: Sven Schooss, Stefan Hierl
> Sitz der Gesellschaft: Ehningen
> Registergericht: Amtsgericht Stuttgart, HRB 17122
>
>
>
> From:   Jan-Frode Myklebust <janfrode at tanso.net>
> To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date:   28/05/2021 19:50
> Subject:        [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM
> Storwize V5030
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
>
>
>
>
> One thing to check: Storwize/SVC code will *always* guess wrong on
> prefetching for GPFS. You can see this with having a lot higher read data

> throughput on mdisk vs. on on vdisks in the webui. To fix it, disable
> cache_prefetch with "chsystem -cache_prefetch off".
>
> This being a global setting, you probably only should set it if the
system
> is only used for GPFS.
>
>
>   -jf
>
> On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <
> oluwasijibomi.saula at ndsu.edu> wrote:
> Hi Folks,
>
> So, we are experiencing some very long IO waiters in our GPFS cluster:
>
> #  mmdiag --waiters
>
> === mmdiag: waiters ===
> Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread:
for
> I/O completion
> Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread:
for
> I/O completion
> Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread:
for
> I/O completion
> Waiting 15.2013 sec since 10:41:03, monitored, thread 22175
>
> However, GPFS support is pointing to our IBM Storwize V5030 disk system
as
> the source of latency. Unfortunately, we don't have paid support for the
> system so we are polling for anyone who might be able to assist.
>
> Does anyone by chance have any experience with IBM Storwize V5030 or
> possess a problem determination guide for the V5030?
>
> We've briefly reviewed the V5030 management portal, but we still haven't
> identified a cause for the increased latencies (i.e. read ~129ms, write
> ~198ms).
>
> Granted, we have some heavy client workloads, yet we seem to experience
> this drastic drop in performance every couple of months, probably
> exacerbated by heavy IO demands.
>
> Any assistance would be much appreciated.
>
>
> Thanks,
>
> (Siji) Saula
> HPC Systems Administrator  /  Information Technology
>
> Research 2 Building 220B / Fargo ND 58108-6050
> p: 701.231.7749 / www.ndsu.edu
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210528/49d91d85/attachment-0002.htm>