From david_johnson at brown.edu  Wed Apr  7 18:10:56 2021
From: david_johnson at brown.edu (David Johnson)
Date: Wed, 7 Apr 2021 13:10:56 -0400
Subject: [gpfsug-discuss] Strategies for keeping GPFS copy for Disaster
	Recovery
Message-ID: <6612F282-02AD-47B0-ABC9-5E26427B7446@brown.edu>

We plan to use rsync to keep a DR copy of our filesystem. The production filesystem contains
hundreds of dependent fillets and a much smaller number of independent filesets.

A question for any of you folks out there with a similar situation:
do you synchronize filesets in parallel on the DR copy?  If so, how do you handle day-to-day fileset create/delete?
I looked to see if there was a user exit script that could be called on fileset create, but did not see any.

Thanks,
 -- ddj
Dave Johnson

From bbanister at jumptrading.com  Thu Apr  8 18:43:03 2021
From: bbanister at jumptrading.com (Bryan Banister)
Date: Thu, 8 Apr 2021 17:43:03 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes case
 number and uploads snap data to IBM automatically
Message-ID: <BN7PR14MB233825B11340FAC935A7F7F2A2749@BN7PR14MB2338.namprd14.prod.outlook.com>

Hey All,

Hope you'll up vote this RFE I just made: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=149787

Just think of the collective time saved from having to manually upload data once the gpfs.snap is collected for support.

Hope all your clusters are healthy,
-Bryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210408/7600f989/attachment.htm>

From MDIETZ at de.ibm.com  Fri Apr  9 21:57:30 2021
From: MDIETZ at de.ibm.com (Mathias Dietz)
Date: Fri, 9 Apr 2021 20:57:30 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes
 case number and uploads snap data to IBM automatically
In-Reply-To: <BN7PR14MB233825B11340FAC935A7F7F2A2749@BN7PR14MB2338.namprd14.prod.outlook.com>
Message-ID: <OF09C93E4E.9826481B-ON002586B2.0072328B-002586B2.007320DB@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210409/0edc4802/attachment.htm>

From novosirj at rutgers.edu  Fri Apr  9 22:02:01 2021
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Fri, 9 Apr 2021 21:02:01 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes
 case number and uploads snap data to IBM automatically
In-Reply-To: <OF09C93E4E.9826481B-ON002586B2.0072328B-002586B2.007320DB@notes.na.collabserv.com>
References: <OF09C93E4E.9826481B-ON002586B2.0072328B-002586B2.007320DB@notes.na.collabserv.com>
Message-ID: <42F8BF21-5727-4860-9B22-4E907F761C35@rutgers.edu>

If I?m not mistaken, you can?t use Spectrum Scale Call Home at all except on ESS, is that right? If so, that excludes anyone on Lenovo GSS/DSS-G, though they do have IBM support.

--
#BlackLivesMatter
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Apr 9, 2021, at 4:57 PM, Mathias Dietz <MDIETZ at de.ibm.com> wrote:
> 
> Hi Bryan,
>  
> Spectrum Scale already has a feature to upload gpfs.snap to IBM support and attach it to a ticket using a single command.
>  
> To upload the gpfs.snap to IBM Service run the following command (requires Spectrum Scale Call Home to be enabled) :
> mmcallhome run SendFile --file <snap file> --pmr  <Support ticket number TSxxxxxxxxx>
>  
> When using the Spectrum Scale GUI you can collect a gpfs.snap and upload it to IBM support through a single GUI panel too:
> https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=issues-collecting-diagnostic-data-through-gui
>  
>  
> Mit freundlichen Gr??en / Kind regards
>  
> Mathias Dietz
>  
> Spectrum Scale RAS Architect
> Senior Technical Staff Member (STSM)
> Dept. M925 - IBM Spectrum Scale Software Development
>  
> Phone:
> +49-15152801035	
> Am Weiher 24  
> 
> Email:	mdietz at de.ibm.com        	65451 Kelsterbach
>  	 	 
> IBM Data Privacy Statement
> IBM Deutschland Research & Development GmbH
> Vorsitzender des Aufsichtsrats: Gregor Pillen
> Gesch?ftsf?hrung: Dirk Wittkopp
> Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
>  
>  
> ----- Original message -----
> From: Bryan Banister <bbanister at jumptrading.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes case number and uploads snap data to IBM automatically
> Date: Thu, Apr 8, 2021 20:01
>  
> Hey All,
>  
> Hope you?ll up vote this RFE I just made: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=149787
>  
> Just think of the collective time saved from having to manually upload data once the gpfs.snap is collected for support.
>  
> Hope all your clusters are healthy,
> -Bryan
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
>  
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From abeattie at au1.ibm.com  Sat Apr 10 02:02:24 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Sat, 10 Apr 2021 01:02:24 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes
 case number and uploads snap data to IBM automatically
In-Reply-To: <42F8BF21-5727-4860-9B22-4E907F761C35@rutgers.edu>
Message-ID: <OF1143CB1F.687C0BB8-ON002586B3.0005B62E-1618016543986@notes.na.collabserv.com>


Ryan,

There are 2 parts to call home,
The spectrum scale Software call home and the IBM ESS hardware call home l.

The software call home should work regardless of hardware platform.
However it will not pull the detail of the hardware, you will need to
upload hardware logs separately.

Sent from my iPhone

> On 10 Apr 2021, at 07:02, Ryan Novosielski <novosirj at rutgers.edu> wrote:
>
> ?If I?m not mistaken, you can?t use Spectrum Scale Call Home at all except
on ESS, is that right? If so, that excludes anyone on Lenovo GSS/DSS-G,
though they do have IBM support.
>
> --
> #BlackLivesMatter
> ____
> || \\UTGERS,
|---------------------------*O*---------------------------
> ||_// the State     |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS
Campus
> ||  \\    of NJ     | Office of Advanced Research Computing - MSB C630,
Newark
>     `'
>
>> On Apr 9, 2021, at 4:57 PM, Mathias Dietz <MDIETZ at de.ibm.com> wrote:
>>
>> Hi Bryan,
>>
>> Spectrum Scale already has a feature to upload gpfs.snap to IBM support
and attach it to a ticket using a single command.
>>
>> To upload the gpfs.snap to IBM Service run the following command
(requires Spectrum Scale Call Home to be enabled) :
>> mmcallhome run SendFile --file <snap file> --pmr  <Support ticket number
TSxxxxxxxxx>
>>
>> When using the Spectrum Scale GUI you can collect a gpfs.snap and upload
it to IBM support through a single GUI panel too:
>>
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=issues-collecting-diagnostic-data-through-gui

>>
>>
>> Mit freundlichen Gr??en / Kind regards
>>
>> Mathias Dietz
>>
>> Spectrum Scale RAS Architect
>> Senior Technical Staff Member (STSM)
>> Dept. M925 - IBM Spectrum Scale Software Development
>>
>> Phone:
>> +49-15152801035
>> Am Weiher 24
>>
>> Email:    mdietz at de.ibm.com            65451 Kelsterbach
>>
>> IBM Data Privacy Statement
>> IBM Deutschland Research & Development GmbH
>> Vorsitzender des Aufsichtsrats: Gregor Pillen
>> Gesch?ftsf?hrung: Dirk Wittkopp
>> Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht
Stuttgart, HRB 243294
>>
>>
>> ----- Original message -----
>> From: Bryan Banister <bbanister at jumptrading.com>
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> Cc:
>> Subject: [EXTERNAL] [gpfsug-discuss] New RFE! Add option to gpfs.snap
that takes case number and uploads snap data to IBM automatically
>> Date: Thu, Apr 8, 2021 20:01
>>
>> Hey All,
>>
>> Hope you?ll up vote this RFE I just made:
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=149787
>>
>> Just think of the collective time saved from having to manually upload
data once the gpfs.snap is collected for support.
>>
>> Hope all your clusters are healthy,
>> -Bryan
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210410/23ecc2d1/attachment.htm>

From carlz at us.ibm.com  Tue Apr 13 17:05:22 2021
From: carlz at us.ibm.com (Carl Zetie)
Date: Tue, 13 Apr 2021 16:05:22 +0000
Subject: [gpfsug-discuss] Licensing fun: opportunity for Sockets customers
Message-ID: <0D85EF34-2A25-45C9-A33E-75E7523EDB4A@us.ibm.com>

Everybody loves licensing conversations?

Is there anybody who is

  1.  On Socket licensing, and
  2.  Potentially interested in moving from Perpetual to Subscription licensing model?

Please note that this is purely an investigation at this point, trying to see if it is worth struggling with the inevitable IBM processes?

If interested, please contact me off-list.


Regards,



Carl Zetie
Program Director
Offering Management
Spectrum Scale
----
(919) 473 3318 ][ Research Triangle Park
carlz at us.ibm.com

[signature_1627441165]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210413/4970a477/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 69558 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210413/4970a477/attachment.png>

From alvise.dorigo at psi.ch  Tue Apr 20 11:12:20 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 10:12:20 +0000
Subject: [gpfsug-discuss] NFSIO metrics absent in pmcollector
Message-ID: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>

Dear Community,

I've activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
        name = "NFSIO"
        period = 5
},
{
        name = "SMBGlobalStats"
        period = 5
},
{
        name = "SMBStats"
        period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[...]

Any clue ? Is the activation of NFSIO enough in the mmperfmon's config file (this is the first time I try to configure CES monitoring) ?

Thank you,

   Alvise
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/0fa14786/attachment.htm>

From janfrode at tanso.net  Tue Apr 20 11:31:01 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Tue, 20 Apr 2021 12:31:01 +0200
Subject: [gpfsug-discuss] NFSIO metrics absent in pmcollector
In-Reply-To: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
Message-ID: <CAHwPatiiPro-62OEbmL2ta_Ycp=ZyJCoUCjJ0YBwomihDvh9yA@mail.gmail.com>

Have you installed the gpfs.pm-ganesha package, and do you have any active
NFS exports/clients ?


  -jf

On Tue, Apr 20, 2021 at 12:19 PM Dorigo Alvise (PSI) <alvise.dorigo at psi.ch>
wrote:

> Dear Community,
>
>
>
> I?ve activated CES-related metrics by simply doing:
>
> [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
>
>         name = "NFSIO"
>
>         period = 5
>
> },
>
> {
>
>         name = "SMBGlobalStats"
>
>         period = 5
>
> },
>
> {
>
>         name = "SMBStats"
>
>         period = 5
>
> }
>
>
>
> Despite that, I get no result for any nfs* metrics:
>
>
>
> --------------------------------------------
>
> [root at xbl-ces-91 ~]# telnet localhost 9084
>
> Trying 127.0.0.1...
>
> Connected to localhost.
>
> Escape character is '^]'.
>
> get -j metrics nfs_read bucket_size 5 last 1
>
> Error: no data available for query
>
> .
>
> Connection closed by foreign host.
>
>
>
> [root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
>
> Error: no data available for query
>
> .
>
>
>
> mmperfmon: Command failed. Examine previous error messages to determine
> cause.
>
> --------------------------------------------
>
>
>
>
>
> I am pretty sure my collector is working correctly as it returns data for
> a different metrics:
>
>
>
> [root at xbl-ces-91 ~]# telnet localhost 9084
>
> Trying 127.0.0.1...
>
> Connected to localhost.
>
> Escape character is '^]'.
>
> get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
>
> {"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" :
> 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" :
> 2,"aggrType" : "value difference between start and end of the
> bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|
> lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
> "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
> start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch
> |GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption"
> : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
> start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch
> |GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
> "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
> start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch
> |GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
>
> [?]
>
>
>
> Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config
> file (this is the first time I try to configure CES monitoring) ?
>
>
>
> Thank you,
>
>
>
>    Alvise
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/076cc591/attachment.htm>

From rohwedder at de.ibm.com  Tue Apr 20 11:34:59 2021
From: rohwedder at de.ibm.com (Markus Rohwedder)
Date: Tue, 20 Apr 2021 12:34:59 +0200
Subject: [gpfsug-discuss] NFSIO metrics absent in pmcollector
In-Reply-To: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
Message-ID: <OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>

Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors
rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
        name = "NFSIO"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics


 /var/log/zimon/ZIMonSensors.log may provides some additional clues as
well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center
                                                                                             
                                                                                             
                                                                                             
                                                                                             
                                                                                             
 Phone:            +49 162 4159920       IBM Deutschland GmbH                                
                                                                                             
 E-Mail:           rohwedder at de.ibm.com  Am Weiher 24                                        
                                                                                             
                                         65451 Kelsterbach                                   
                                                                                             
                                         Germany                                             
                                                                                             
                                                                                             
                                                                                             
 IBM Deutschland                                                                             
 GmbH /                                                                                      
 Vorsitzender des                                                                            
 Aufsichtsrats:                                                                              
 Sebastian                                                                                   
 Krause /                                                                                    
 Gesch?ftsf?hrung:                                                                           
 Gregor Pillen                                                                               
 (Vorsitzender),                                                                             
 Agnes Heftberger,                                                                           
 Norbert Janzen,                                                                             
 Markus Koerner,                                                                             
 Christian Noll,                                                                             
 Nicole Reimer /                                                                             
 Sitz der                                                                                    
 Gesellschaft:                                                                               
 71139 Ehningen,                                                                             
 IBM-Allee 1 /                                                                               
 Registergericht:                                                                            
 Amtsgericht                                                                                 
 Stuttgart,                                                                                  
 HRB14562                                                                                    
                                                                                             
                                                                                             
                                                                                             





From:	"Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch>
To:	"'gpfsug-discuss at spectrumscale.org'"
            <gpfsug-discuss at spectrumscale.org>
Cc:	"'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch>
Date:	20.04.2021 12:19
Subject:	[EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Dear Community, I?ve activated CES-related metrics by simply doing:
[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name =
"NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
        name = "NFSIO"
        period = 5
},
{
        name = "SMBGlobalStats"
        period = 5
},
{
        name = "SMBStats"
        period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine
cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a
different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" :
1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" :
2,"aggrType" : "value difference between start and end of the
bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|
lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
"gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|
GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" :
"gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|
GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
"gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|
GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file
(this is the first time I try to configure CES monitoring) ?

Thank you,

   Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1A548529.gif
Type: image/gif
Size: 4659 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment-0002.gif>

From u.sibiller at science-computing.de  Tue Apr 20 12:18:15 2021
From: u.sibiller at science-computing.de (Ulrich Sibiller)
Date: Tue, 20 Apr 2021 13:18:15 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
Message-ID: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>


Hello *,

I have to delete a subtree of about ~50 million files in thousands of subdirs, ~14TB of data. 
Running a recursive rm is very slow so I setup a simple policy file:

RULE 'delstuff' DELETE
     DIRECTORIES_PLUS
     WHERE PATH_NAME LIKE '/mypath/%'

This kinda works but is not really fast, either. It even requires a second run because files and 
directories within the tree will be processed in arbitrary order so it will happen quite frequently 
that a directory is going to be deleted before its content has been removed completely. For those 
dirs I see an error message and have to delete afterwards.

I am wondering if there's a quicker way. Given the fact that this is a whole tree I think there's 
should be a quick way to unlink the complete inode hierachy.

Unfortunately we are not using a fileset for that tree...

So are there any ideas how to solve that more efficiently?

Uli
-- 
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196

From jonathan.buzzard at strath.ac.uk  Tue Apr 20 12:47:44 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Tue, 20 Apr 2021 12:47:44 +0100
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
Message-ID: <70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>

On 20/04/2021 12:18, Ulrich Sibiller wrote:
> 
> Hello *,
> 
> I have to delete a subtree of about ~50 million files in thousands of
>  subdirs, ~14TB of data. Running a recursive rm is very slow so I
> setup a simple policy file:
> 
> RULE 'delstuff' DELETE
>    DIRECTORIES_PLUS >    WHERE PATH_NAME LIKE '/mypath/%'
> 
> This kinda works but is not really fast, either. It even requires a 
> second run because files and directories within the tree will be
> processed in arbitrary order so it will happen quite frequently that
> a directory is going to be deleted before its content has been 
> removed completely. For those dirs I see an error message and have to
> delete afterwards.
> 

You are going to have to remove all the inodes and hence the speed is 
going to be dependant on your metadata performance no matter how you 
approach the problem. I doubt that you are going to get much better than 
a recursive rm.

You could try running a series of parallel rm's attacking subsection of 
the directory tree on different nodes, but I suspect that it will make 
very little difference. You could my sync/restore script to split the 
problem up between different nodes

https://github.com/digitalcabbage/syncrestore

> I am wondering if there's a quicker way. Given the fact that this is
> a whole tree I think there's should be a quick way to unlink the
> complete inode hierachy.
> 
> Unfortunately we are not using a fileset for that tree...
> 
> So are there any ideas how to solve that more efficiently?
> 

Does it matter that it takes a long time? Use screen or tmux to run the 
command in the background safe from accidental detaches and forget 
about. Check on it now and then to see how it's going or just forget 
about it, it will finish. Something like

screen -S filedel -L -Logfile /tmp/filedel.log -d -m /bin/rm -rf <tree>

Consider using mv to move it out the way or hide it while the delete is 
in progress. If you do that think carefully about backups, you don't 
want to back it all up again while it is being deleted :-)


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From alvise.dorigo at psi.ch  Tue Apr 20 12:50:08 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 11:50:08 +0000
Subject: [gpfsug-discuss] R: [sls-htc-admins] NFSIO metrics absent in
	pmcollector
In-Reply-To: <CAHwPatiiPro-62OEbmL2ta_Ycp=ZyJCoUCjJ0YBwomihDvh9yA@mail.gmail.com>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<CAHwPatiiPro-62OEbmL2ta_Ycp=ZyJCoUCjJ0YBwomihDvh9yA@mail.gmail.com>
Message-ID: <0f96be99814648bd8e405389cafe5ba9@psi.ch>

Hello, Jan

The answer is definitely yes to both questions ?

  A

Da: sls-htc-admins-request at lists.psi.ch <sls-htc-admins-request at lists.psi.ch> Per conto di Jan-Frode Myklebust
Inviato: marted? 20 aprile 2021 12:31
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: sls-htc-admins at lists.psi.ch
Oggetto: Re: [sls-htc-admins] [gpfsug-discuss] NFSIO metrics absent in pmcollector

Have you installed the gpfs.pm-ganesha package, and do you have any active NFS exports/clients ?


  -jf

On Tue, Apr 20, 2021 at 12:19 PM Dorigo Alvise (PSI) <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>> wrote:
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
        name = "NFSIO"
        period = 5
},
{
        name = "SMBGlobalStats"
        period = 5
},
{
        name = "SMBStats"
        period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch<http://xbl-ces-91.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch<http://xbl-ces-91.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch<http://xbl-ces-92.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch<http://xbl-ces-92.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

   Alvise
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e9f2934a/attachment.htm>

From alvise.dorigo at psi.ch  Tue Apr 20 12:51:19 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 11:51:19 +0000
Subject: [gpfsug-discuss] R:  NFSIO metrics absent in pmcollector
In-Reply-To: <OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
Message-ID: <7dfae5dee31a46068457c8398ba154ae@psi.ch>

Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss-bounces at spectrumscale.org; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________



Phone:

+49 162 4159920

IBM Deutschland GmbH

[cid:image002.png at 01D735EC.3C210570]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________

IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562




[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 4659 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.gif
Type: image/gif
Size: 105 bytes
Desc: image003.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 167 bytes
Desc: image004.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment-0001.png>

From janfrode at tanso.net  Tue Apr 20 12:52:50 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Tue, 20 Apr 2021 13:52:50 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
Message-ID: <CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>

A couple of ideas.

The KC recommends adding WEIGHT(DIRECTORY_HASH) to group deletions within a
directory. Then maybe also do it as a 2-step process, in the same policy
run. Where you delete all non-directories first, and then deletes the
directories in a depth-first order using WEIGTH(Length(PATH_NAME)):


RULE 'delnondir' DELETE
     WEIGHT(DIRECTORY_HASH)
     DIRECTORIES_PLUS
     WHERE PATH_NAME LIKE '/mypath/%' AND NOT MISC_ATTRIBUTES LIKE '%D%'

RULE 'deldir' DELETE
     DIRECTORIES_PLUS
    WEIGHT(Length(PATH_NAME))
     WHERE PATH_NAME LIKE '/mypath/%'  AND MISC_ATTRIBUTES LIKE '%D%'

HTH


On Tue, Apr 20, 2021 at 1:18 PM Ulrich Sibiller <
u.sibiller at science-computing.de> wrote:

>
> Hello *,
>
> I have to delete a subtree of about ~50 million files in thousands of
> subdirs, ~14TB of data.
> Running a recursive rm is very slow so I setup a simple policy file:
>
> RULE 'delstuff' DELETE
>      DIRECTORIES_PLUS
>      WHERE PATH_NAME LIKE '/mypath/%'
>
> This kinda works but is not really fast, either. It even requires a second
> run because files and
> directories within the tree will be processed in arbitrary order so it
> will happen quite frequently
> that a directory is going to be deleted before its content has been
> removed completely. For those
> dirs I see an error message and have to delete afterwards.
>
> I am wondering if there's a quicker way. Given the fact that this is a
> whole tree I think there's
> should be a quick way to unlink the complete inode hierachy.
>
> Unfortunately we are not using a fileset for that tree...
>
> So are there any ideas how to solve that more efficiently?
>
> Uli
> --
> Science + Computing AG
> Vorstandsvorsitzender/Chairman of the board of management:
> Dr. Martin Matzke
> Vorstand/Board of Management:
> Matthias Schempp, Sabine Hohenstein
> Vorsitzender des Aufsichtsrats/
> Chairman of the Supervisory Board:
> Philippe Miltin
> Aufsichtsrat/Supervisory Board:
> Martin Wibbe, Ursula Morgenstern
> Sitz/Registered Office: Tuebingen
> Registergericht/Registration Court: Stuttgart
> Registernummer/Commercial Register No.: HRB 382196
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/af8e2b54/attachment.htm>

From u.sibiller at science-computing.de  Tue Apr 20 13:09:42 2021
From: u.sibiller at science-computing.de (Ulrich Sibiller)
Date: Tue, 20 Apr 2021 14:09:42 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
Message-ID: <d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>

On 4/20/21 1:47 PM, Jonathan Buzzard wrote:
> You are going to have to remove all the inodes and hence the speed is
> going to be dependant on your metadata performance no matter how you
> approach the problem. I doubt that you are going to get much better than
> a recursive rm.
> 
> You could try running a series of parallel rm's attacking subsection of
> the directory tree on different nodes, but I suspect that it will make
> very little difference. You could my sync/restore script to split the
> problem up between different nodes
> 
> https://github.com/digitalcabbage/syncrestore

might be worth a try, thanks.

> Does it matter that it takes a long time? Use screen or tmux to run the

Unfortunately it makes sense since I need the sapce for something else...

> command in the background safe from accidental detaches and forget
> about. Check on it now and then to see how it's going or just forget
> about it, it will finish. Something like
> 
> screen -S filedel -L -Logfile /tmp/filedel.log -d -m /bin/rm -rf <tree>
> 
> Consider using mv to move it out the way or hide it while the delete is
> in progress. If you do that think carefully about backups, you don't
> want to back it all up again while it is being deleted :-)

;-) Yeah, that's why I did not the do the mv in the first place ;-)

Thanks,

Uli


-- 
Dipl.-Inf. Ulrich Sibiller           science + computing ag
System Administration                    Hagellocher Weg 73
Hotline +49 7071 9457 681          72070 Tuebingen, Germany
                           https://atos.net/de/deutschland/sc
-- 
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196

From u.sibiller at science-computing.de  Tue Apr 20 13:09:51 2021
From: u.sibiller at science-computing.de (Ulrich Sibiller)
Date: Tue, 20 Apr 2021 14:09:51 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>
Message-ID: <084d86ed-38b9-8b7f-a840-863dc6686bb5@science-computing.de>

On 4/20/21 1:52 PM, Jan-Frode Myklebust wrote:
> A couple of ideas.
> 
> The KC recommends adding WEIGHT(DIRECTORY_HASH) to group?deletions within a directory. Then maybe > also do it as a 2-step process, in the same policy run. Where you delete all non-directories first,
> and then deletes the directories in a depth-first order using WEIGTH(Length(PATH_NAME)):
> 
> 
> RULE 'delnondir' DELETE
> WEIGHT(DIRECTORY_HASH)
>  ? ? ?DIRECTORIES_PLUS
>  ? ? ?WHERE PATH_NAME LIKE '/mypath/%' AND NOT MISC_ATTRIBUTES LIKE '%D%'
> 
> RULE 'deldir' DELETE
>  ? ? ?DIRECTORIES_PLUS
>  ? ? WEIGHT(Length(PATH_NAME))
>  ? ? ?WHERE PATH_NAME LIKE '/mypath/%' ?AND MISC_ATTRIBUTES LIKE '%D%'


Thanks, I am aware of that but it will not really help with my speed concerns.

Uli


-- 
Dipl.-Inf. Ulrich Sibiller           science + computing ag
System Administration                    Hagellocher Weg 73
Hotline +49 7071 9457 681          72070 Tuebingen, Germany
                           https://atos.net/de/deutschland/sc
-- 
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196

From alvise.dorigo at psi.ch  Tue Apr 20 13:37:57 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 12:37:57 +0000
Subject: [gpfsug-discuss] R: [EXT] R: NFSIO metrics absent in pmcollector
In-Reply-To: <SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
	<7dfae5dee31a46068457c8398ba154ae@psi.ch>
	<SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>
Message-ID: <8f597b3c85cd4f6ea8448f3fc49ce5da@psi.ch>

Hello Larry,
I?ve implemented your suggestion to my perfmon configuration (proxy, restriction to relevat node, type).
But the problem is still there.
I guess I need to open a PMR to IBM, if not other clue?

    Alvise

Da: sls-htc-admins-request at lists.psi.ch <sls-htc-admins-request at lists.psi.ch> Per conto di Henson Jr.,Larry J
Inviato: marted? 20 aprile 2021 14:22
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss-bounces at spectrumscale.org; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>
Oggetto: RE: [sls-htc-admins] [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

Hello Dorigo,

This is what I have on the end of our ZIMonSensors.cfg file.  Without it the GUI shows no NFS/SMB metrics.

{
        name = "NFSIO"
        period = 10
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBGlobalStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Regards,
Larry Henson
IT Operations Storage Team
Office (832) 750-1403
Cell (713) 702-4896
[cid:image001.png at 01D735F2.BFA83A70]
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of Dorigo Alvise (PSI)
Sent: Tuesday, April 20, 2021 6:51 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Subject: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

WARNING: This email originated from outside of MD Anderson. Please validate the sender's email address before clicking on links or attachments as they may not be safe.


Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYLEsnxBNw$>

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________



Phone:

+49 162 4159920

IBM Deutschland GmbH

[cid:image003.png at 01D735F2.BFA83A70]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________

IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562




[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYI-Ipx8NA$>
The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 19425 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 166 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 4659 bytes
Desc: image003.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.gif
Type: image/gif
Size: 105 bytes
Desc: image004.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment.gif>

From stockf at us.ibm.com  Tue Apr 20 14:00:50 2021
From: stockf at us.ibm.com (Frederick Stock)
Date: Tue, 20 Apr 2021 13:00:50 +0000
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <084d86ed-38b9-8b7f-a840-863dc6686bb5@science-computing.de>
References: <084d86ed-38b9-8b7f-a840-863dc6686bb5@science-computing.de>,
	<f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de><CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>
Message-ID: <OFB96766C9.C77426E0-ON002586BD.0046E2E0-002586BD.00477CF1@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/74ce6742/attachment.htm>

From LJHenson at mdanderson.org  Tue Apr 20 13:21:34 2021
From: LJHenson at mdanderson.org (Henson Jr.,Larry J)
Date: Tue, 20 Apr 2021 12:21:34 +0000
Subject: [gpfsug-discuss] [EXT] R: NFSIO metrics absent in pmcollector
In-Reply-To: <7dfae5dee31a46068457c8398ba154ae@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
	<7dfae5dee31a46068457c8398ba154ae@psi.ch>
Message-ID: <SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>

Hello Dorigo,

This is what I have on the end of our ZIMonSensors.cfg file.  Without it the GUI shows no NFS/SMB metrics.

{
        name = "NFSIO"
        period = 10
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBGlobalStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Regards,
Larry Henson
IT Operations Storage Team
Office (832) 750-1403
Cell (713) 702-4896
[cid:image001.png at 01D735B5.C8961270]
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Dorigo Alvise (PSI)
Sent: Tuesday, April 20, 2021 6:51 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss-bounces at spectrumscale.org; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>
Subject: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

WARNING: This email originated from outside of MD Anderson. Please validate the sender's email address before clicking on links or attachments as they may not be safe.

Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYLEsnxBNw$>

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________

Phone:
+49 162 4159920
IBM Deutschland GmbH
[cid:image006.png at 01D735B5.C8961270]
E-Mail:
rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>
Am Weiher 24


65451 Kelsterbach


Germany
________________________________
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562


[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYI-Ipx8NA$>


The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 19425 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 4659 bytes
Desc: image006.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.gif
Type: image/gif
Size: 105 bytes
Desc: image007.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 166 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0002.png>

From alvise.dorigo at psi.ch  Tue Apr 20 14:39:03 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 13:39:03 +0000
Subject: [gpfsug-discuss] R: [EXT] R: NFSIO metrics absent in pmcollector
In-Reply-To: <8f597b3c85cd4f6ea8448f3fc49ce5da@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
	<7dfae5dee31a46068457c8398ba154ae@psi.ch>
	<SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>
	<8f597b3c85cd4f6ea8448f3fc49ce5da@psi.ch>
Message-ID: <c0853ab6defc496692d8c26246562003@psi.ch>

Hello again,
found the reason: no client had already generated any traffic (because this is just a test cluster).
Now, some read/write has been done, and the related table was created in the collector?s DB.

Thanks and sorry, I probably missed this detail.

   Alvise

Da: sls-htc-admins-request at lists.psi.ch <sls-htc-admins-request at lists.psi.ch> Per conto di Dorigo Alvise (PSI)
Inviato: marted? 20 aprile 2021 14:38
A: 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>; 'Henson Jr.,Larry J' <LJHenson at mdanderson.org>; 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org>
Cc: 'gpfsug-discuss-bounces at spectrumscale.org' <gpfsug-discuss-bounces at spectrumscale.org>
Oggetto: [sls-htc-admins] R: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

Hello Larry,
I?ve implemented your suggestion to my perfmon configuration (proxy, restriction to relevat node, type).
But the problem is still there.
I guess I need to open a PMR to IBM, if not other clue?

    Alvise

Da: sls-htc-admins-request at lists.psi.ch<mailto:sls-htc-admins-request at lists.psi.ch> <sls-htc-admins-request at lists.psi.ch<mailto:sls-htc-admins-request at lists.psi.ch>> Per conto di Henson Jr.,Larry J
Inviato: marted? 20 aprile 2021 14:22
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: RE: [sls-htc-admins] [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

Hello Dorigo,

This is what I have on the end of our ZIMonSensors.cfg file.  Without it the GUI shows no NFS/SMB metrics.

{
        name = "NFSIO"
        period = 10
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBGlobalStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Regards,
Larry Henson
IT Operations Storage Team
Office (832) 750-1403
Cell (713) 702-4896
[cid:image001.png at 01D735FB.48EDDAD0]
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of Dorigo Alvise (PSI)
Sent: Tuesday, April 20, 2021 6:51 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Subject: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

WARNING: This email originated from outside of MD Anderson. Please validate the sender's email address before clicking on links or attachments as they may not be safe.


Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYLEsnxBNw$>

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________



Phone:

+49 162 4159920

IBM Deutschland GmbH

[cid:image003.png at 01D735FB.48EDDAD0]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________

IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562




[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYI-Ipx8NA$>
The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 19425 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 166 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 4659 bytes
Desc: image003.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.gif
Type: image/gif
Size: 105 bytes
Desc: image004.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment.gif>

From jonathan.buzzard at strath.ac.uk  Tue Apr 20 14:51:35 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Tue, 20 Apr 2021 14:51:35 +0100
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
	<d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>
Message-ID: <c19dc2fa-448c-d6a5-2a17-56f7fd307a41@strath.ac.uk>

On 20/04/2021 13:09, Ulrich Sibiller wrote:

>>
>> Consider using mv to move it out the way or hide it while the delete is
>> in progress. If you do that think carefully about backups, you don't
>> want to back it all up again while it is being deleted :-)
> 
> ;-) Yeah, that's why I did not the do the mv in the first place ;-)
> 

I would estimate (based on my experience) is that you should be able to 
delete that amount of data/files in under 24 hours anyway with a simple 
rm -rf. Which is why I question trying to find faster methods. You have 
already wasted a significant amount of that time :-)

If your using TSM for the backup then just exclude it from the backup in 
your dsm.opts file

exclude.dir  <path to directory>

We have a NOBACK option that allows users to select if they don't want 
something backing up. Helpful if your job generates lots of temporary 
files or data that are junked as soon as the job finishes. Anything 
under a directory called NOBACK does not get backed up.

exclude.dir  /.../NOBACK/


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From anacreo at gmail.com  Tue Apr 20 17:44:53 2021
From: anacreo at gmail.com (Alec)
Date: Tue, 20 Apr 2021 09:44:53 -0700
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <c19dc2fa-448c-d6a5-2a17-56f7fd307a41@strath.ac.uk>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
	<d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>
	<c19dc2fa-448c-d6a5-2a17-56f7fd307a41@strath.ac.uk>
Message-ID: <CAGhSTwhiLq_uk1f1NKMyfL_RHjGeHfjhM5eLz2HeXU9fkOmujA@mail.gmail.com>

I would start with the mv to hide it, and then allow the delete to progress
in the background.  I would seperate out the delete of files from
directories... And i would try using mmxargs with a rm command to get
parallelism plus reduce the number of execs in one policy, followed by a
simple rm -r of the directory tree.

Maybe it's cheaper to make a new filesystem and just retain the data you
want though...

Alec

On Tue, Apr 20, 2021, 6:51 AM Jonathan Buzzard <
jonathan.buzzard at strath.ac.uk> wrote:

> On 20/04/2021 13:09, Ulrich Sibiller wrote:
>
> >>
> >> Consider using mv to move it out the way or hide it while the delete is
> >> in progress. If you do that think carefully about backups, you don't
> >> want to back it all up again while it is being deleted :-)
> >
> > ;-) Yeah, that's why I did not the do the mv in the first place ;-)
> >
>
> I would estimate (based on my experience) is that you should be able to
> delete that amount of data/files in under 24 hours anyway with a simple
> rm -rf. Which is why I question trying to find faster methods. You have
> already wasted a significant amount of that time :-)
>
> If your using TSM for the backup then just exclude it from the backup in
> your dsm.opts file
>
> exclude.dir  <path to directory>
>
> We have a NOBACK option that allows users to select if they don't want
> something backing up. Helpful if your job generates lots of temporary
> files or data that are junked as soon as the job finishes. Anything
> under a directory called NOBACK does not get backed up.
>
> exclude.dir  /.../NOBACK/
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                         Tel: +44141-5483420
> HPC System Administrator, ARCHIE-WeSt.
> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/86cec6ae/attachment.htm>

From tomasz.rachobinski at gmail.com  Mon Apr 26 11:10:04 2021
From: tomasz.rachobinski at gmail.com (=?UTF-8?Q?Tomasz_Rachobi=C5=84ski?=)
Date: Mon, 26 Apr 2021 12:10:04 +0200
Subject: [gpfsug-discuss] Hello from Poland
Message-ID: <CAOnndBP6Db087Z8Gd=h8Kud743dBMZuM=7Lgh31uoAdPQhOjYQ@mail.gmail.com>

Hello All,
My name is Tom, I'm from Warsaw, Poland. I work for polish Bank PKO,
usually I'm a mainframe sysprog but for 1 year I'm learning a lot of gpfs -
in my case it's Spectrum Scale Data Access due to mixed Linux/Windows env.
I have many questions in my head, but I'll try not to bore You too much :)

Greetings
Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210426/2eee42e4/attachment.htm>

From david_johnson at brown.edu  Wed Apr  7 18:10:56 2021
From: david_johnson at brown.edu (David Johnson)
Date: Wed, 7 Apr 2021 13:10:56 -0400
Subject: [gpfsug-discuss] Strategies for keeping GPFS copy for Disaster
	Recovery
Message-ID: <6612F282-02AD-47B0-ABC9-5E26427B7446@brown.edu>

We plan to use rsync to keep a DR copy of our filesystem. The production filesystem contains
hundreds of dependent fillets and a much smaller number of independent filesets.

A question for any of you folks out there with a similar situation:
do you synchronize filesets in parallel on the DR copy?  If so, how do you handle day-to-day fileset create/delete?
I looked to see if there was a user exit script that could be called on fileset create, but did not see any.

Thanks,
 -- ddj
Dave Johnson

From bbanister at jumptrading.com  Thu Apr  8 18:43:03 2021
From: bbanister at jumptrading.com (Bryan Banister)
Date: Thu, 8 Apr 2021 17:43:03 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes case
 number and uploads snap data to IBM automatically
Message-ID: <BN7PR14MB233825B11340FAC935A7F7F2A2749@BN7PR14MB2338.namprd14.prod.outlook.com>

Hey All,

Hope you'll up vote this RFE I just made: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=149787

Just think of the collective time saved from having to manually upload data once the gpfs.snap is collected for support.

Hope all your clusters are healthy,
-Bryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210408/7600f989/attachment-0001.htm>

From MDIETZ at de.ibm.com  Fri Apr  9 21:57:30 2021
From: MDIETZ at de.ibm.com (Mathias Dietz)
Date: Fri, 9 Apr 2021 20:57:30 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes
 case number and uploads snap data to IBM automatically
In-Reply-To: <BN7PR14MB233825B11340FAC935A7F7F2A2749@BN7PR14MB2338.namprd14.prod.outlook.com>
Message-ID: <OF09C93E4E.9826481B-ON002586B2.0072328B-002586B2.007320DB@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210409/0edc4802/attachment-0001.htm>

From novosirj at rutgers.edu  Fri Apr  9 22:02:01 2021
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Fri, 9 Apr 2021 21:02:01 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes
 case number and uploads snap data to IBM automatically
In-Reply-To: <OF09C93E4E.9826481B-ON002586B2.0072328B-002586B2.007320DB@notes.na.collabserv.com>
References: <OF09C93E4E.9826481B-ON002586B2.0072328B-002586B2.007320DB@notes.na.collabserv.com>
Message-ID: <42F8BF21-5727-4860-9B22-4E907F761C35@rutgers.edu>

If I?m not mistaken, you can?t use Spectrum Scale Call Home at all except on ESS, is that right? If so, that excludes anyone on Lenovo GSS/DSS-G, though they do have IBM support.

--
#BlackLivesMatter
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Apr 9, 2021, at 4:57 PM, Mathias Dietz <MDIETZ at de.ibm.com> wrote:
> 
> Hi Bryan,
>  
> Spectrum Scale already has a feature to upload gpfs.snap to IBM support and attach it to a ticket using a single command.
>  
> To upload the gpfs.snap to IBM Service run the following command (requires Spectrum Scale Call Home to be enabled) :
> mmcallhome run SendFile --file <snap file> --pmr  <Support ticket number TSxxxxxxxxx>
>  
> When using the Spectrum Scale GUI you can collect a gpfs.snap and upload it to IBM support through a single GUI panel too:
> https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=issues-collecting-diagnostic-data-through-gui
>  
>  
> Mit freundlichen Gr??en / Kind regards
>  
> Mathias Dietz
>  
> Spectrum Scale RAS Architect
> Senior Technical Staff Member (STSM)
> Dept. M925 - IBM Spectrum Scale Software Development
>  
> Phone:
> +49-15152801035	
> Am Weiher 24  
> 
> Email:	mdietz at de.ibm.com        	65451 Kelsterbach
>  	 	 
> IBM Data Privacy Statement
> IBM Deutschland Research & Development GmbH
> Vorsitzender des Aufsichtsrats: Gregor Pillen
> Gesch?ftsf?hrung: Dirk Wittkopp
> Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
>  
>  
> ----- Original message -----
> From: Bryan Banister <bbanister at jumptrading.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes case number and uploads snap data to IBM automatically
> Date: Thu, Apr 8, 2021 20:01
>  
> Hey All,
>  
> Hope you?ll up vote this RFE I just made: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=149787
>  
> Just think of the collective time saved from having to manually upload data once the gpfs.snap is collected for support.
>  
> Hope all your clusters are healthy,
> -Bryan
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
>  
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From abeattie at au1.ibm.com  Sat Apr 10 02:02:24 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Sat, 10 Apr 2021 01:02:24 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes
 case number and uploads snap data to IBM automatically
In-Reply-To: <42F8BF21-5727-4860-9B22-4E907F761C35@rutgers.edu>
Message-ID: <OF1143CB1F.687C0BB8-ON002586B3.0005B62E-1618016543986@notes.na.collabserv.com>


Ryan,

There are 2 parts to call home,
The spectrum scale Software call home and the IBM ESS hardware call home l.

The software call home should work regardless of hardware platform.
However it will not pull the detail of the hardware, you will need to
upload hardware logs separately.

Sent from my iPhone

> On 10 Apr 2021, at 07:02, Ryan Novosielski <novosirj at rutgers.edu> wrote:
>
> ?If I?m not mistaken, you can?t use Spectrum Scale Call Home at all except
on ESS, is that right? If so, that excludes anyone on Lenovo GSS/DSS-G,
though they do have IBM support.
>
> --
> #BlackLivesMatter
> ____
> || \\UTGERS,
|---------------------------*O*---------------------------
> ||_// the State     |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS
Campus
> ||  \\    of NJ     | Office of Advanced Research Computing - MSB C630,
Newark
>     `'
>
>> On Apr 9, 2021, at 4:57 PM, Mathias Dietz <MDIETZ at de.ibm.com> wrote:
>>
>> Hi Bryan,
>>
>> Spectrum Scale already has a feature to upload gpfs.snap to IBM support
and attach it to a ticket using a single command.
>>
>> To upload the gpfs.snap to IBM Service run the following command
(requires Spectrum Scale Call Home to be enabled) :
>> mmcallhome run SendFile --file <snap file> --pmr  <Support ticket number
TSxxxxxxxxx>
>>
>> When using the Spectrum Scale GUI you can collect a gpfs.snap and upload
it to IBM support through a single GUI panel too:
>>
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=issues-collecting-diagnostic-data-through-gui

>>
>>
>> Mit freundlichen Gr??en / Kind regards
>>
>> Mathias Dietz
>>
>> Spectrum Scale RAS Architect
>> Senior Technical Staff Member (STSM)
>> Dept. M925 - IBM Spectrum Scale Software Development
>>
>> Phone:
>> +49-15152801035
>> Am Weiher 24
>>
>> Email:    mdietz at de.ibm.com            65451 Kelsterbach
>>
>> IBM Data Privacy Statement
>> IBM Deutschland Research & Development GmbH
>> Vorsitzender des Aufsichtsrats: Gregor Pillen
>> Gesch?ftsf?hrung: Dirk Wittkopp
>> Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht
Stuttgart, HRB 243294
>>
>>
>> ----- Original message -----
>> From: Bryan Banister <bbanister at jumptrading.com>
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> Cc:
>> Subject: [EXTERNAL] [gpfsug-discuss] New RFE! Add option to gpfs.snap
that takes case number and uploads snap data to IBM automatically
>> Date: Thu, Apr 8, 2021 20:01
>>
>> Hey All,
>>
>> Hope you?ll up vote this RFE I just made:
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=149787
>>
>> Just think of the collective time saved from having to manually upload
data once the gpfs.snap is collected for support.
>>
>> Hope all your clusters are healthy,
>> -Bryan
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210410/23ecc2d1/attachment-0001.htm>

From carlz at us.ibm.com  Tue Apr 13 17:05:22 2021
From: carlz at us.ibm.com (Carl Zetie)
Date: Tue, 13 Apr 2021 16:05:22 +0000
Subject: [gpfsug-discuss] Licensing fun: opportunity for Sockets customers
Message-ID: <0D85EF34-2A25-45C9-A33E-75E7523EDB4A@us.ibm.com>

Everybody loves licensing conversations?

Is there anybody who is

  1.  On Socket licensing, and
  2.  Potentially interested in moving from Perpetual to Subscription licensing model?

Please note that this is purely an investigation at this point, trying to see if it is worth struggling with the inevitable IBM processes?

If interested, please contact me off-list.


Regards,



Carl Zetie
Program Director
Offering Management
Spectrum Scale
----
(919) 473 3318 ][ Research Triangle Park
carlz at us.ibm.com

[signature_1627441165]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210413/4970a477/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 69558 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210413/4970a477/attachment-0001.png>

From alvise.dorigo at psi.ch  Tue Apr 20 11:12:20 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 10:12:20 +0000
Subject: [gpfsug-discuss] NFSIO metrics absent in pmcollector
Message-ID: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>

Dear Community,

I've activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
        name = "NFSIO"
        period = 5
},
{
        name = "SMBGlobalStats"
        period = 5
},
{
        name = "SMBStats"
        period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[...]

Any clue ? Is the activation of NFSIO enough in the mmperfmon's config file (this is the first time I try to configure CES monitoring) ?

Thank you,

   Alvise
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/0fa14786/attachment-0001.htm>

From janfrode at tanso.net  Tue Apr 20 11:31:01 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Tue, 20 Apr 2021 12:31:01 +0200
Subject: [gpfsug-discuss] NFSIO metrics absent in pmcollector
In-Reply-To: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
Message-ID: <CAHwPatiiPro-62OEbmL2ta_Ycp=ZyJCoUCjJ0YBwomihDvh9yA@mail.gmail.com>

Have you installed the gpfs.pm-ganesha package, and do you have any active
NFS exports/clients ?


  -jf

On Tue, Apr 20, 2021 at 12:19 PM Dorigo Alvise (PSI) <alvise.dorigo at psi.ch>
wrote:

> Dear Community,
>
>
>
> I?ve activated CES-related metrics by simply doing:
>
> [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
>
>         name = "NFSIO"
>
>         period = 5
>
> },
>
> {
>
>         name = "SMBGlobalStats"
>
>         period = 5
>
> },
>
> {
>
>         name = "SMBStats"
>
>         period = 5
>
> }
>
>
>
> Despite that, I get no result for any nfs* metrics:
>
>
>
> --------------------------------------------
>
> [root at xbl-ces-91 ~]# telnet localhost 9084
>
> Trying 127.0.0.1...
>
> Connected to localhost.
>
> Escape character is '^]'.
>
> get -j metrics nfs_read bucket_size 5 last 1
>
> Error: no data available for query
>
> .
>
> Connection closed by foreign host.
>
>
>
> [root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
>
> Error: no data available for query
>
> .
>
>
>
> mmperfmon: Command failed. Examine previous error messages to determine
> cause.
>
> --------------------------------------------
>
>
>
>
>
> I am pretty sure my collector is working correctly as it returns data for
> a different metrics:
>
>
>
> [root at xbl-ces-91 ~]# telnet localhost 9084
>
> Trying 127.0.0.1...
>
> Connected to localhost.
>
> Escape character is '^]'.
>
> get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
>
> {"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" :
> 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" :
> 2,"aggrType" : "value difference between start and end of the
> bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|
> lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
> "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
> start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch
> |GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption"
> : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
> start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch
> |GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
> "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
> start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch
> |GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
>
> [?]
>
>
>
> Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config
> file (this is the first time I try to configure CES monitoring) ?
>
>
>
> Thank you,
>
>
>
>    Alvise
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/076cc591/attachment-0001.htm>

From rohwedder at de.ibm.com  Tue Apr 20 11:34:59 2021
From: rohwedder at de.ibm.com (Markus Rohwedder)
Date: Tue, 20 Apr 2021 12:34:59 +0200
Subject: [gpfsug-discuss] NFSIO metrics absent in pmcollector
In-Reply-To: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
Message-ID: <OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>

Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors
rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
        name = "NFSIO"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics


 /var/log/zimon/ZIMonSensors.log may provides some additional clues as
well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center
                                                                                             
                                                                                             
                                                                                             
                                                                                             
                                                                                             
 Phone:            +49 162 4159920       IBM Deutschland GmbH                                
                                                                                             
 E-Mail:           rohwedder at de.ibm.com  Am Weiher 24                                        
                                                                                             
                                         65451 Kelsterbach                                   
                                                                                             
                                         Germany                                             
                                                                                             
                                                                                             
                                                                                             
 IBM Deutschland                                                                             
 GmbH /                                                                                      
 Vorsitzender des                                                                            
 Aufsichtsrats:                                                                              
 Sebastian                                                                                   
 Krause /                                                                                    
 Gesch?ftsf?hrung:                                                                           
 Gregor Pillen                                                                               
 (Vorsitzender),                                                                             
 Agnes Heftberger,                                                                           
 Norbert Janzen,                                                                             
 Markus Koerner,                                                                             
 Christian Noll,                                                                             
 Nicole Reimer /                                                                             
 Sitz der                                                                                    
 Gesellschaft:                                                                               
 71139 Ehningen,                                                                             
 IBM-Allee 1 /                                                                               
 Registergericht:                                                                            
 Amtsgericht                                                                                 
 Stuttgart,                                                                                  
 HRB14562                                                                                    
                                                                                             
                                                                                             
                                                                                             





From:	"Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch>
To:	"'gpfsug-discuss at spectrumscale.org'"
            <gpfsug-discuss at spectrumscale.org>
Cc:	"'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch>
Date:	20.04.2021 12:19
Subject:	[EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Dear Community, I?ve activated CES-related metrics by simply doing:
[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name =
"NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
        name = "NFSIO"
        period = 5
},
{
        name = "SMBGlobalStats"
        period = 5
},
{
        name = "SMBStats"
        period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine
cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a
different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" :
1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" :
2,"aggrType" : "value difference between start and end of the
bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|
lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
"gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|
GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" :
"gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|
GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
"gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|
GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file
(this is the first time I try to configure CES monitoring) ?

Thank you,

   Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment-0003.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1A548529.gif
Type: image/gif
Size: 4659 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment-0004.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment-0005.gif>

From u.sibiller at science-computing.de  Tue Apr 20 12:18:15 2021
From: u.sibiller at science-computing.de (Ulrich Sibiller)
Date: Tue, 20 Apr 2021 13:18:15 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
Message-ID: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>


Hello *,

I have to delete a subtree of about ~50 million files in thousands of subdirs, ~14TB of data. 
Running a recursive rm is very slow so I setup a simple policy file:

RULE 'delstuff' DELETE
     DIRECTORIES_PLUS
     WHERE PATH_NAME LIKE '/mypath/%'

This kinda works but is not really fast, either. It even requires a second run because files and 
directories within the tree will be processed in arbitrary order so it will happen quite frequently 
that a directory is going to be deleted before its content has been removed completely. For those 
dirs I see an error message and have to delete afterwards.

I am wondering if there's a quicker way. Given the fact that this is a whole tree I think there's 
should be a quick way to unlink the complete inode hierachy.

Unfortunately we are not using a fileset for that tree...

So are there any ideas how to solve that more efficiently?

Uli
-- 
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196

From jonathan.buzzard at strath.ac.uk  Tue Apr 20 12:47:44 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Tue, 20 Apr 2021 12:47:44 +0100
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
Message-ID: <70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>

On 20/04/2021 12:18, Ulrich Sibiller wrote:
> 
> Hello *,
> 
> I have to delete a subtree of about ~50 million files in thousands of
>  subdirs, ~14TB of data. Running a recursive rm is very slow so I
> setup a simple policy file:
> 
> RULE 'delstuff' DELETE
>    DIRECTORIES_PLUS >    WHERE PATH_NAME LIKE '/mypath/%'
> 
> This kinda works but is not really fast, either. It even requires a 
> second run because files and directories within the tree will be
> processed in arbitrary order so it will happen quite frequently that
> a directory is going to be deleted before its content has been 
> removed completely. For those dirs I see an error message and have to
> delete afterwards.
> 

You are going to have to remove all the inodes and hence the speed is 
going to be dependant on your metadata performance no matter how you 
approach the problem. I doubt that you are going to get much better than 
a recursive rm.

You could try running a series of parallel rm's attacking subsection of 
the directory tree on different nodes, but I suspect that it will make 
very little difference. You could my sync/restore script to split the 
problem up between different nodes

https://github.com/digitalcabbage/syncrestore

> I am wondering if there's a quicker way. Given the fact that this is
> a whole tree I think there's should be a quick way to unlink the
> complete inode hierachy.
> 
> Unfortunately we are not using a fileset for that tree...
> 
> So are there any ideas how to solve that more efficiently?
> 

Does it matter that it takes a long time? Use screen or tmux to run the 
command in the background safe from accidental detaches and forget 
about. Check on it now and then to see how it's going or just forget 
about it, it will finish. Something like

screen -S filedel -L -Logfile /tmp/filedel.log -d -m /bin/rm -rf <tree>

Consider using mv to move it out the way or hide it while the delete is 
in progress. If you do that think carefully about backups, you don't 
want to back it all up again while it is being deleted :-)


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From alvise.dorigo at psi.ch  Tue Apr 20 12:50:08 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 11:50:08 +0000
Subject: [gpfsug-discuss] R: [sls-htc-admins] NFSIO metrics absent in
	pmcollector
In-Reply-To: <CAHwPatiiPro-62OEbmL2ta_Ycp=ZyJCoUCjJ0YBwomihDvh9yA@mail.gmail.com>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<CAHwPatiiPro-62OEbmL2ta_Ycp=ZyJCoUCjJ0YBwomihDvh9yA@mail.gmail.com>
Message-ID: <0f96be99814648bd8e405389cafe5ba9@psi.ch>

Hello, Jan

The answer is definitely yes to both questions ?

  A

Da: sls-htc-admins-request at lists.psi.ch <sls-htc-admins-request at lists.psi.ch> Per conto di Jan-Frode Myklebust
Inviato: marted? 20 aprile 2021 12:31
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: sls-htc-admins at lists.psi.ch
Oggetto: Re: [sls-htc-admins] [gpfsug-discuss] NFSIO metrics absent in pmcollector

Have you installed the gpfs.pm-ganesha package, and do you have any active NFS exports/clients ?


  -jf

On Tue, Apr 20, 2021 at 12:19 PM Dorigo Alvise (PSI) <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>> wrote:
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
        name = "NFSIO"
        period = 5
},
{
        name = "SMBGlobalStats"
        period = 5
},
{
        name = "SMBStats"
        period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch<http://xbl-ces-91.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch<http://xbl-ces-91.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch<http://xbl-ces-92.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch<http://xbl-ces-92.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

   Alvise
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e9f2934a/attachment-0001.htm>

From alvise.dorigo at psi.ch  Tue Apr 20 12:51:19 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 11:51:19 +0000
Subject: [gpfsug-discuss] R:  NFSIO metrics absent in pmcollector
In-Reply-To: <OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
Message-ID: <7dfae5dee31a46068457c8398ba154ae@psi.ch>

Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss-bounces at spectrumscale.org; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________



Phone:

+49 162 4159920

IBM Deutschland GmbH

[cid:image002.png at 01D735EC.3C210570]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________

IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562




[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 4659 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.gif
Type: image/gif
Size: 105 bytes
Desc: image003.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 167 bytes
Desc: image004.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment-0003.png>

From janfrode at tanso.net  Tue Apr 20 12:52:50 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Tue, 20 Apr 2021 13:52:50 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
Message-ID: <CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>

A couple of ideas.

The KC recommends adding WEIGHT(DIRECTORY_HASH) to group deletions within a
directory. Then maybe also do it as a 2-step process, in the same policy
run. Where you delete all non-directories first, and then deletes the
directories in a depth-first order using WEIGTH(Length(PATH_NAME)):


RULE 'delnondir' DELETE
     WEIGHT(DIRECTORY_HASH)
     DIRECTORIES_PLUS
     WHERE PATH_NAME LIKE '/mypath/%' AND NOT MISC_ATTRIBUTES LIKE '%D%'

RULE 'deldir' DELETE
     DIRECTORIES_PLUS
    WEIGHT(Length(PATH_NAME))
     WHERE PATH_NAME LIKE '/mypath/%'  AND MISC_ATTRIBUTES LIKE '%D%'

HTH


On Tue, Apr 20, 2021 at 1:18 PM Ulrich Sibiller <
u.sibiller at science-computing.de> wrote:

>
> Hello *,
>
> I have to delete a subtree of about ~50 million files in thousands of
> subdirs, ~14TB of data.
> Running a recursive rm is very slow so I setup a simple policy file:
>
> RULE 'delstuff' DELETE
>      DIRECTORIES_PLUS
>      WHERE PATH_NAME LIKE '/mypath/%'
>
> This kinda works but is not really fast, either. It even requires a second
> run because files and
> directories within the tree will be processed in arbitrary order so it
> will happen quite frequently
> that a directory is going to be deleted before its content has been
> removed completely. For those
> dirs I see an error message and have to delete afterwards.
>
> I am wondering if there's a quicker way. Given the fact that this is a
> whole tree I think there's
> should be a quick way to unlink the complete inode hierachy.
>
> Unfortunately we are not using a fileset for that tree...
>
> So are there any ideas how to solve that more efficiently?
>
> Uli
> --
> Science + Computing AG
> Vorstandsvorsitzender/Chairman of the board of management:
> Dr. Martin Matzke
> Vorstand/Board of Management:
> Matthias Schempp, Sabine Hohenstein
> Vorsitzender des Aufsichtsrats/
> Chairman of the Supervisory Board:
> Philippe Miltin
> Aufsichtsrat/Supervisory Board:
> Martin Wibbe, Ursula Morgenstern
> Sitz/Registered Office: Tuebingen
> Registergericht/Registration Court: Stuttgart
> Registernummer/Commercial Register No.: HRB 382196
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/af8e2b54/attachment-0001.htm>

From u.sibiller at science-computing.de  Tue Apr 20 13:09:42 2021
From: u.sibiller at science-computing.de (Ulrich Sibiller)
Date: Tue, 20 Apr 2021 14:09:42 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
Message-ID: <d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>

On 4/20/21 1:47 PM, Jonathan Buzzard wrote:
> You are going to have to remove all the inodes and hence the speed is
> going to be dependant on your metadata performance no matter how you
> approach the problem. I doubt that you are going to get much better than
> a recursive rm.
> 
> You could try running a series of parallel rm's attacking subsection of
> the directory tree on different nodes, but I suspect that it will make
> very little difference. You could my sync/restore script to split the
> problem up between different nodes
> 
> https://github.com/digitalcabbage/syncrestore

might be worth a try, thanks.

> Does it matter that it takes a long time? Use screen or tmux to run the

Unfortunately it makes sense since I need the sapce for something else...

> command in the background safe from accidental detaches and forget
> about. Check on it now and then to see how it's going or just forget
> about it, it will finish. Something like
> 
> screen -S filedel -L -Logfile /tmp/filedel.log -d -m /bin/rm -rf <tree>
> 
> Consider using mv to move it out the way or hide it while the delete is
> in progress. If you do that think carefully about backups, you don't
> want to back it all up again while it is being deleted :-)

;-) Yeah, that's why I did not the do the mv in the first place ;-)

Thanks,

Uli


-- 
Dipl.-Inf. Ulrich Sibiller           science + computing ag
System Administration                    Hagellocher Weg 73
Hotline +49 7071 9457 681          72070 Tuebingen, Germany
                           https://atos.net/de/deutschland/sc
-- 
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196

From u.sibiller at science-computing.de  Tue Apr 20 13:09:51 2021
From: u.sibiller at science-computing.de (Ulrich Sibiller)
Date: Tue, 20 Apr 2021 14:09:51 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>
Message-ID: <084d86ed-38b9-8b7f-a840-863dc6686bb5@science-computing.de>

On 4/20/21 1:52 PM, Jan-Frode Myklebust wrote:
> A couple of ideas.
> 
> The KC recommends adding WEIGHT(DIRECTORY_HASH) to group?deletions within a directory. Then maybe > also do it as a 2-step process, in the same policy run. Where you delete all non-directories first,
> and then deletes the directories in a depth-first order using WEIGTH(Length(PATH_NAME)):
> 
> 
> RULE 'delnondir' DELETE
> WEIGHT(DIRECTORY_HASH)
>  ? ? ?DIRECTORIES_PLUS
>  ? ? ?WHERE PATH_NAME LIKE '/mypath/%' AND NOT MISC_ATTRIBUTES LIKE '%D%'
> 
> RULE 'deldir' DELETE
>  ? ? ?DIRECTORIES_PLUS
>  ? ? WEIGHT(Length(PATH_NAME))
>  ? ? ?WHERE PATH_NAME LIKE '/mypath/%' ?AND MISC_ATTRIBUTES LIKE '%D%'


Thanks, I am aware of that but it will not really help with my speed concerns.

Uli


-- 
Dipl.-Inf. Ulrich Sibiller           science + computing ag
System Administration                    Hagellocher Weg 73
Hotline +49 7071 9457 681          72070 Tuebingen, Germany
                           https://atos.net/de/deutschland/sc
-- 
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196

From alvise.dorigo at psi.ch  Tue Apr 20 13:37:57 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 12:37:57 +0000
Subject: [gpfsug-discuss] R: [EXT] R: NFSIO metrics absent in pmcollector
In-Reply-To: <SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
	<7dfae5dee31a46068457c8398ba154ae@psi.ch>
	<SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>
Message-ID: <8f597b3c85cd4f6ea8448f3fc49ce5da@psi.ch>

Hello Larry,
I?ve implemented your suggestion to my perfmon configuration (proxy, restriction to relevat node, type).
But the problem is still there.
I guess I need to open a PMR to IBM, if not other clue?

    Alvise

Da: sls-htc-admins-request at lists.psi.ch <sls-htc-admins-request at lists.psi.ch> Per conto di Henson Jr.,Larry J
Inviato: marted? 20 aprile 2021 14:22
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss-bounces at spectrumscale.org; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>
Oggetto: RE: [sls-htc-admins] [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

Hello Dorigo,

This is what I have on the end of our ZIMonSensors.cfg file.  Without it the GUI shows no NFS/SMB metrics.

{
        name = "NFSIO"
        period = 10
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBGlobalStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Regards,
Larry Henson
IT Operations Storage Team
Office (832) 750-1403
Cell (713) 702-4896
[cid:image001.png at 01D735F2.BFA83A70]
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of Dorigo Alvise (PSI)
Sent: Tuesday, April 20, 2021 6:51 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Subject: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

WARNING: This email originated from outside of MD Anderson. Please validate the sender's email address before clicking on links or attachments as they may not be safe.


Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYLEsnxBNw$>

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________



Phone:

+49 162 4159920

IBM Deutschland GmbH

[cid:image003.png at 01D735F2.BFA83A70]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________

IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562




[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYI-Ipx8NA$>
The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 19425 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 166 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 4659 bytes
Desc: image003.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.gif
Type: image/gif
Size: 105 bytes
Desc: image004.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0001.gif>

From stockf at us.ibm.com  Tue Apr 20 14:00:50 2021
From: stockf at us.ibm.com (Frederick Stock)
Date: Tue, 20 Apr 2021 13:00:50 +0000
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <084d86ed-38b9-8b7f-a840-863dc6686bb5@science-computing.de>
References: <084d86ed-38b9-8b7f-a840-863dc6686bb5@science-computing.de>,
	<f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de><CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>
Message-ID: <OFB96766C9.C77426E0-ON002586BD.0046E2E0-002586BD.00477CF1@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/74ce6742/attachment-0001.htm>

From LJHenson at mdanderson.org  Tue Apr 20 13:21:34 2021
From: LJHenson at mdanderson.org (Henson Jr.,Larry J)
Date: Tue, 20 Apr 2021 12:21:34 +0000
Subject: [gpfsug-discuss] [EXT] R: NFSIO metrics absent in pmcollector
In-Reply-To: <7dfae5dee31a46068457c8398ba154ae@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
	<7dfae5dee31a46068457c8398ba154ae@psi.ch>
Message-ID: <SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>

Hello Dorigo,

This is what I have on the end of our ZIMonSensors.cfg file.  Without it the GUI shows no NFS/SMB metrics.

{
        name = "NFSIO"
        period = 10
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBGlobalStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Regards,
Larry Henson
IT Operations Storage Team
Office (832) 750-1403
Cell (713) 702-4896
[cid:image001.png at 01D735B5.C8961270]
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Dorigo Alvise (PSI)
Sent: Tuesday, April 20, 2021 6:51 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss-bounces at spectrumscale.org; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>
Subject: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

WARNING: This email originated from outside of MD Anderson. Please validate the sender's email address before clicking on links or attachments as they may not be safe.

Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYLEsnxBNw$>

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________

Phone:
+49 162 4159920
IBM Deutschland GmbH
[cid:image006.png at 01D735B5.C8961270]
E-Mail:
rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>
Am Weiher 24


65451 Kelsterbach


Germany
________________________________
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562


[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYI-Ipx8NA$>


The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 19425 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 4659 bytes
Desc: image006.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.gif
Type: image/gif
Size: 105 bytes
Desc: image007.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 166 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0005.png>

From alvise.dorigo at psi.ch  Tue Apr 20 14:39:03 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 13:39:03 +0000
Subject: [gpfsug-discuss] R: [EXT] R: NFSIO metrics absent in pmcollector
In-Reply-To: <8f597b3c85cd4f6ea8448f3fc49ce5da@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
	<7dfae5dee31a46068457c8398ba154ae@psi.ch>
	<SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>
	<8f597b3c85cd4f6ea8448f3fc49ce5da@psi.ch>
Message-ID: <c0853ab6defc496692d8c26246562003@psi.ch>

Hello again,
found the reason: no client had already generated any traffic (because this is just a test cluster).
Now, some read/write has been done, and the related table was created in the collector?s DB.

Thanks and sorry, I probably missed this detail.

   Alvise

Da: sls-htc-admins-request at lists.psi.ch <sls-htc-admins-request at lists.psi.ch> Per conto di Dorigo Alvise (PSI)
Inviato: marted? 20 aprile 2021 14:38
A: 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>; 'Henson Jr.,Larry J' <LJHenson at mdanderson.org>; 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org>
Cc: 'gpfsug-discuss-bounces at spectrumscale.org' <gpfsug-discuss-bounces at spectrumscale.org>
Oggetto: [sls-htc-admins] R: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

Hello Larry,
I?ve implemented your suggestion to my perfmon configuration (proxy, restriction to relevat node, type).
But the problem is still there.
I guess I need to open a PMR to IBM, if not other clue?

    Alvise

Da: sls-htc-admins-request at lists.psi.ch<mailto:sls-htc-admins-request at lists.psi.ch> <sls-htc-admins-request at lists.psi.ch<mailto:sls-htc-admins-request at lists.psi.ch>> Per conto di Henson Jr.,Larry J
Inviato: marted? 20 aprile 2021 14:22
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: RE: [sls-htc-admins] [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

Hello Dorigo,

This is what I have on the end of our ZIMonSensors.cfg file.  Without it the GUI shows no NFS/SMB metrics.

{
        name = "NFSIO"
        period = 10
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBGlobalStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Regards,
Larry Henson
IT Operations Storage Team
Office (832) 750-1403
Cell (713) 702-4896
[cid:image001.png at 01D735FB.48EDDAD0]
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of Dorigo Alvise (PSI)
Sent: Tuesday, April 20, 2021 6:51 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Subject: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

WARNING: This email originated from outside of MD Anderson. Please validate the sender's email address before clicking on links or attachments as they may not be safe.


Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYLEsnxBNw$>

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________



Phone:

+49 162 4159920

IBM Deutschland GmbH

[cid:image003.png at 01D735FB.48EDDAD0]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________

IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562




[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYI-Ipx8NA$>
The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 19425 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 166 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 4659 bytes
Desc: image003.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.gif
Type: image/gif
Size: 105 bytes
Desc: image004.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0001.gif>

From jonathan.buzzard at strath.ac.uk  Tue Apr 20 14:51:35 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Tue, 20 Apr 2021 14:51:35 +0100
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
	<d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>
Message-ID: <c19dc2fa-448c-d6a5-2a17-56f7fd307a41@strath.ac.uk>

On 20/04/2021 13:09, Ulrich Sibiller wrote:

>>
>> Consider using mv to move it out the way or hide it while the delete is
>> in progress. If you do that think carefully about backups, you don't
>> want to back it all up again while it is being deleted :-)
> 
> ;-) Yeah, that's why I did not the do the mv in the first place ;-)
> 

I would estimate (based on my experience) is that you should be able to 
delete that amount of data/files in under 24 hours anyway with a simple 
rm -rf. Which is why I question trying to find faster methods. You have 
already wasted a significant amount of that time :-)

If your using TSM for the backup then just exclude it from the backup in 
your dsm.opts file

exclude.dir  <path to directory>

We have a NOBACK option that allows users to select if they don't want 
something backing up. Helpful if your job generates lots of temporary 
files or data that are junked as soon as the job finishes. Anything 
under a directory called NOBACK does not get backed up.

exclude.dir  /.../NOBACK/


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From anacreo at gmail.com  Tue Apr 20 17:44:53 2021
From: anacreo at gmail.com (Alec)
Date: Tue, 20 Apr 2021 09:44:53 -0700
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <c19dc2fa-448c-d6a5-2a17-56f7fd307a41@strath.ac.uk>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
	<d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>
	<c19dc2fa-448c-d6a5-2a17-56f7fd307a41@strath.ac.uk>
Message-ID: <CAGhSTwhiLq_uk1f1NKMyfL_RHjGeHfjhM5eLz2HeXU9fkOmujA@mail.gmail.com>

I would start with the mv to hide it, and then allow the delete to progress
in the background.  I would seperate out the delete of files from
directories... And i would try using mmxargs with a rm command to get
parallelism plus reduce the number of execs in one policy, followed by a
simple rm -r of the directory tree.

Maybe it's cheaper to make a new filesystem and just retain the data you
want though...

Alec

On Tue, Apr 20, 2021, 6:51 AM Jonathan Buzzard <
jonathan.buzzard at strath.ac.uk> wrote:

> On 20/04/2021 13:09, Ulrich Sibiller wrote:
>
> >>
> >> Consider using mv to move it out the way or hide it while the delete is
> >> in progress. If you do that think carefully about backups, you don't
> >> want to back it all up again while it is being deleted :-)
> >
> > ;-) Yeah, that's why I did not the do the mv in the first place ;-)
> >
>
> I would estimate (based on my experience) is that you should be able to
> delete that amount of data/files in under 24 hours anyway with a simple
> rm -rf. Which is why I question trying to find faster methods. You have
> already wasted a significant amount of that time :-)
>
> If your using TSM for the backup then just exclude it from the backup in
> your dsm.opts file
>
> exclude.dir  <path to directory>
>
> We have a NOBACK option that allows users to select if they don't want
> something backing up. Helpful if your job generates lots of temporary
> files or data that are junked as soon as the job finishes. Anything
> under a directory called NOBACK does not get backed up.
>
> exclude.dir  /.../NOBACK/
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                         Tel: +44141-5483420
> HPC System Administrator, ARCHIE-WeSt.
> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/86cec6ae/attachment-0001.htm>

From tomasz.rachobinski at gmail.com  Mon Apr 26 11:10:04 2021
From: tomasz.rachobinski at gmail.com (=?UTF-8?Q?Tomasz_Rachobi=C5=84ski?=)
Date: Mon, 26 Apr 2021 12:10:04 +0200
Subject: [gpfsug-discuss] Hello from Poland
Message-ID: <CAOnndBP6Db087Z8Gd=h8Kud743dBMZuM=7Lgh31uoAdPQhOjYQ@mail.gmail.com>

Hello All,
My name is Tom, I'm from Warsaw, Poland. I work for polish Bank PKO,
usually I'm a mainframe sysprog but for 1 year I'm learning a lot of gpfs -
in my case it's Spectrum Scale Data Access due to mixed Linux/Windows env.
I have many questions in my head, but I'll try not to bore You too much :)

Greetings
Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210426/2eee42e4/attachment-0001.htm>

From david_johnson at brown.edu  Wed Apr  7 18:10:56 2021
From: david_johnson at brown.edu (David Johnson)
Date: Wed, 7 Apr 2021 13:10:56 -0400
Subject: [gpfsug-discuss] Strategies for keeping GPFS copy for Disaster
	Recovery
Message-ID: <6612F282-02AD-47B0-ABC9-5E26427B7446@brown.edu>

We plan to use rsync to keep a DR copy of our filesystem. The production filesystem contains
hundreds of dependent fillets and a much smaller number of independent filesets.

A question for any of you folks out there with a similar situation:
do you synchronize filesets in parallel on the DR copy?  If so, how do you handle day-to-day fileset create/delete?
I looked to see if there was a user exit script that could be called on fileset create, but did not see any.

Thanks,
 -- ddj
Dave Johnson

From bbanister at jumptrading.com  Thu Apr  8 18:43:03 2021
From: bbanister at jumptrading.com (Bryan Banister)
Date: Thu, 8 Apr 2021 17:43:03 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes case
 number and uploads snap data to IBM automatically
Message-ID: <BN7PR14MB233825B11340FAC935A7F7F2A2749@BN7PR14MB2338.namprd14.prod.outlook.com>

Hey All,

Hope you'll up vote this RFE I just made: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=149787

Just think of the collective time saved from having to manually upload data once the gpfs.snap is collected for support.

Hope all your clusters are healthy,
-Bryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210408/7600f989/attachment-0002.htm>

From MDIETZ at de.ibm.com  Fri Apr  9 21:57:30 2021
From: MDIETZ at de.ibm.com (Mathias Dietz)
Date: Fri, 9 Apr 2021 20:57:30 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes
 case number and uploads snap data to IBM automatically
In-Reply-To: <BN7PR14MB233825B11340FAC935A7F7F2A2749@BN7PR14MB2338.namprd14.prod.outlook.com>
Message-ID: <OF09C93E4E.9826481B-ON002586B2.0072328B-002586B2.007320DB@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210409/0edc4802/attachment-0002.htm>

From novosirj at rutgers.edu  Fri Apr  9 22:02:01 2021
From: novosirj at rutgers.edu (Ryan Novosielski)
Date: Fri, 9 Apr 2021 21:02:01 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes
 case number and uploads snap data to IBM automatically
In-Reply-To: <OF09C93E4E.9826481B-ON002586B2.0072328B-002586B2.007320DB@notes.na.collabserv.com>
References: <OF09C93E4E.9826481B-ON002586B2.0072328B-002586B2.007320DB@notes.na.collabserv.com>
Message-ID: <42F8BF21-5727-4860-9B22-4E907F761C35@rutgers.edu>

If I?m not mistaken, you can?t use Spectrum Scale Call Home at all except on ESS, is that right? If so, that excludes anyone on Lenovo GSS/DSS-G, though they do have IBM support.

--
#BlackLivesMatter
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Apr 9, 2021, at 4:57 PM, Mathias Dietz <MDIETZ at de.ibm.com> wrote:
> 
> Hi Bryan,
>  
> Spectrum Scale already has a feature to upload gpfs.snap to IBM support and attach it to a ticket using a single command.
>  
> To upload the gpfs.snap to IBM Service run the following command (requires Spectrum Scale Call Home to be enabled) :
> mmcallhome run SendFile --file <snap file> --pmr  <Support ticket number TSxxxxxxxxx>
>  
> When using the Spectrum Scale GUI you can collect a gpfs.snap and upload it to IBM support through a single GUI panel too:
> https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=issues-collecting-diagnostic-data-through-gui
>  
>  
> Mit freundlichen Gr??en / Kind regards
>  
> Mathias Dietz
>  
> Spectrum Scale RAS Architect
> Senior Technical Staff Member (STSM)
> Dept. M925 - IBM Spectrum Scale Software Development
>  
> Phone:
> +49-15152801035	
> Am Weiher 24  
> 
> Email:	mdietz at de.ibm.com        	65451 Kelsterbach
>  	 	 
> IBM Data Privacy Statement
> IBM Deutschland Research & Development GmbH
> Vorsitzender des Aufsichtsrats: Gregor Pillen
> Gesch?ftsf?hrung: Dirk Wittkopp
> Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
>  
>  
> ----- Original message -----
> From: Bryan Banister <bbanister at jumptrading.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes case number and uploads snap data to IBM automatically
> Date: Thu, Apr 8, 2021 20:01
>  
> Hey All,
>  
> Hope you?ll up vote this RFE I just made: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=149787
>  
> Just think of the collective time saved from having to manually upload data once the gpfs.snap is collected for support.
>  
> Hope all your clusters are healthy,
> -Bryan
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
>  
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


From abeattie at au1.ibm.com  Sat Apr 10 02:02:24 2021
From: abeattie at au1.ibm.com (Andrew Beattie)
Date: Sat, 10 Apr 2021 01:02:24 +0000
Subject: [gpfsug-discuss] New RFE! Add option to gpfs.snap that takes
 case number and uploads snap data to IBM automatically
In-Reply-To: <42F8BF21-5727-4860-9B22-4E907F761C35@rutgers.edu>
Message-ID: <OF1143CB1F.687C0BB8-ON002586B3.0005B62E-1618016543986@notes.na.collabserv.com>


Ryan,

There are 2 parts to call home,
The spectrum scale Software call home and the IBM ESS hardware call home l.

The software call home should work regardless of hardware platform.
However it will not pull the detail of the hardware, you will need to
upload hardware logs separately.

Sent from my iPhone

> On 10 Apr 2021, at 07:02, Ryan Novosielski <novosirj at rutgers.edu> wrote:
>
> ?If I?m not mistaken, you can?t use Spectrum Scale Call Home at all except
on ESS, is that right? If so, that excludes anyone on Lenovo GSS/DSS-G,
though they do have IBM support.
>
> --
> #BlackLivesMatter
> ____
> || \\UTGERS,
|---------------------------*O*---------------------------
> ||_// the State     |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS
Campus
> ||  \\    of NJ     | Office of Advanced Research Computing - MSB C630,
Newark
>     `'
>
>> On Apr 9, 2021, at 4:57 PM, Mathias Dietz <MDIETZ at de.ibm.com> wrote:
>>
>> Hi Bryan,
>>
>> Spectrum Scale already has a feature to upload gpfs.snap to IBM support
and attach it to a ticket using a single command.
>>
>> To upload the gpfs.snap to IBM Service run the following command
(requires Spectrum Scale Call Home to be enabled) :
>> mmcallhome run SendFile --file <snap file> --pmr  <Support ticket number
TSxxxxxxxxx>
>>
>> When using the Spectrum Scale GUI you can collect a gpfs.snap and upload
it to IBM support through a single GUI panel too:
>>
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=issues-collecting-diagnostic-data-through-gui

>>
>>
>> Mit freundlichen Gr??en / Kind regards
>>
>> Mathias Dietz
>>
>> Spectrum Scale RAS Architect
>> Senior Technical Staff Member (STSM)
>> Dept. M925 - IBM Spectrum Scale Software Development
>>
>> Phone:
>> +49-15152801035
>> Am Weiher 24
>>
>> Email:    mdietz at de.ibm.com            65451 Kelsterbach
>>
>> IBM Data Privacy Statement
>> IBM Deutschland Research & Development GmbH
>> Vorsitzender des Aufsichtsrats: Gregor Pillen
>> Gesch?ftsf?hrung: Dirk Wittkopp
>> Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht
Stuttgart, HRB 243294
>>
>>
>> ----- Original message -----
>> From: Bryan Banister <bbanister at jumptrading.com>
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> Cc:
>> Subject: [EXTERNAL] [gpfsug-discuss] New RFE! Add option to gpfs.snap
that takes case number and uploads snap data to IBM automatically
>> Date: Thu, Apr 8, 2021 20:01
>>
>> Hey All,
>>
>> Hope you?ll up vote this RFE I just made:
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=149787
>>
>> Just think of the collective time saved from having to manually upload
data once the gpfs.snap is collected for support.
>>
>> Hope all your clusters are healthy,
>> -Bryan
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210410/23ecc2d1/attachment-0002.htm>

From carlz at us.ibm.com  Tue Apr 13 17:05:22 2021
From: carlz at us.ibm.com (Carl Zetie)
Date: Tue, 13 Apr 2021 16:05:22 +0000
Subject: [gpfsug-discuss] Licensing fun: opportunity for Sockets customers
Message-ID: <0D85EF34-2A25-45C9-A33E-75E7523EDB4A@us.ibm.com>

Everybody loves licensing conversations?

Is there anybody who is

  1.  On Socket licensing, and
  2.  Potentially interested in moving from Perpetual to Subscription licensing model?

Please note that this is purely an investigation at this point, trying to see if it is worth struggling with the inevitable IBM processes?

If interested, please contact me off-list.


Regards,



Carl Zetie
Program Director
Offering Management
Spectrum Scale
----
(919) 473 3318 ][ Research Triangle Park
carlz at us.ibm.com

[signature_1627441165]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210413/4970a477/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 69558 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210413/4970a477/attachment-0002.png>

From alvise.dorigo at psi.ch  Tue Apr 20 11:12:20 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 10:12:20 +0000
Subject: [gpfsug-discuss] NFSIO metrics absent in pmcollector
Message-ID: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>

Dear Community,

I've activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
        name = "NFSIO"
        period = 5
},
{
        name = "SMBGlobalStats"
        period = 5
},
{
        name = "SMBStats"
        period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[...]

Any clue ? Is the activation of NFSIO enough in the mmperfmon's config file (this is the first time I try to configure CES monitoring) ?

Thank you,

   Alvise
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/0fa14786/attachment-0002.htm>

From janfrode at tanso.net  Tue Apr 20 11:31:01 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Tue, 20 Apr 2021 12:31:01 +0200
Subject: [gpfsug-discuss] NFSIO metrics absent in pmcollector
In-Reply-To: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
Message-ID: <CAHwPatiiPro-62OEbmL2ta_Ycp=ZyJCoUCjJ0YBwomihDvh9yA@mail.gmail.com>

Have you installed the gpfs.pm-ganesha package, and do you have any active
NFS exports/clients ?


  -jf

On Tue, Apr 20, 2021 at 12:19 PM Dorigo Alvise (PSI) <alvise.dorigo at psi.ch>
wrote:

> Dear Community,
>
>
>
> I?ve activated CES-related metrics by simply doing:
>
> [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
>
>         name = "NFSIO"
>
>         period = 5
>
> },
>
> {
>
>         name = "SMBGlobalStats"
>
>         period = 5
>
> },
>
> {
>
>         name = "SMBStats"
>
>         period = 5
>
> }
>
>
>
> Despite that, I get no result for any nfs* metrics:
>
>
>
> --------------------------------------------
>
> [root at xbl-ces-91 ~]# telnet localhost 9084
>
> Trying 127.0.0.1...
>
> Connected to localhost.
>
> Escape character is '^]'.
>
> get -j metrics nfs_read bucket_size 5 last 1
>
> Error: no data available for query
>
> .
>
> Connection closed by foreign host.
>
>
>
> [root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
>
> Error: no data available for query
>
> .
>
>
>
> mmperfmon: Command failed. Examine previous error messages to determine
> cause.
>
> --------------------------------------------
>
>
>
>
>
> I am pretty sure my collector is working correctly as it returns data for
> a different metrics:
>
>
>
> [root at xbl-ces-91 ~]# telnet localhost 9084
>
> Trying 127.0.0.1...
>
> Connected to localhost.
>
> Escape character is '^]'.
>
> get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
>
> {"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" :
> 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" :
> 2,"aggrType" : "value difference between start and end of the
> bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|
> lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
> "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
> start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch
> |GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption"
> : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
> start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch
> |GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
> "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
> start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch
> |GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
>
> [?]
>
>
>
> Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config
> file (this is the first time I try to configure CES monitoring) ?
>
>
>
> Thank you,
>
>
>
>    Alvise
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/076cc591/attachment-0002.htm>

From rohwedder at de.ibm.com  Tue Apr 20 11:34:59 2021
From: rohwedder at de.ibm.com (Markus Rohwedder)
Date: Tue, 20 Apr 2021 12:34:59 +0200
Subject: [gpfsug-discuss] NFSIO metrics absent in pmcollector
In-Reply-To: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
Message-ID: <OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>

Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors
rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
        name = "NFSIO"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics


 /var/log/zimon/ZIMonSensors.log may provides some additional clues as
well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center
                                                                                             
                                                                                             
                                                                                             
                                                                                             
                                                                                             
 Phone:            +49 162 4159920       IBM Deutschland GmbH                                
                                                                                             
 E-Mail:           rohwedder at de.ibm.com  Am Weiher 24                                        
                                                                                             
                                         65451 Kelsterbach                                   
                                                                                             
                                         Germany                                             
                                                                                             
                                                                                             
                                                                                             
 IBM Deutschland                                                                             
 GmbH /                                                                                      
 Vorsitzender des                                                                            
 Aufsichtsrats:                                                                              
 Sebastian                                                                                   
 Krause /                                                                                    
 Gesch?ftsf?hrung:                                                                           
 Gregor Pillen                                                                               
 (Vorsitzender),                                                                             
 Agnes Heftberger,                                                                           
 Norbert Janzen,                                                                             
 Markus Koerner,                                                                             
 Christian Noll,                                                                             
 Nicole Reimer /                                                                             
 Sitz der                                                                                    
 Gesellschaft:                                                                               
 71139 Ehningen,                                                                             
 IBM-Allee 1 /                                                                               
 Registergericht:                                                                            
 Amtsgericht                                                                                 
 Stuttgart,                                                                                  
 HRB14562                                                                                    
                                                                                             
                                                                                             
                                                                                             





From:	"Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch>
To:	"'gpfsug-discuss at spectrumscale.org'"
            <gpfsug-discuss at spectrumscale.org>
Cc:	"'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch>
Date:	20.04.2021 12:19
Subject:	[EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



Dear Community, I?ve activated CES-related metrics by simply doing:
[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name =
"NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
        name = "NFSIO"
        period = 5
},
{
        name = "SMBGlobalStats"
        period = 5
},
{
        name = "SMBStats"
        period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine
cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a
different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" :
1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" :
2,"aggrType" : "value difference between start and end of the
bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|
lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
"gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|
GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" :
"gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|
GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" :
"gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between
start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|
GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file
(this is the first time I try to configure CES monitoring) ?

Thank you,

   Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment-0006.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1A548529.gif
Type: image/gif
Size: 4659 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment-0007.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/960caaa5/attachment-0008.gif>

From u.sibiller at science-computing.de  Tue Apr 20 12:18:15 2021
From: u.sibiller at science-computing.de (Ulrich Sibiller)
Date: Tue, 20 Apr 2021 13:18:15 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
Message-ID: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>


Hello *,

I have to delete a subtree of about ~50 million files in thousands of subdirs, ~14TB of data. 
Running a recursive rm is very slow so I setup a simple policy file:

RULE 'delstuff' DELETE
     DIRECTORIES_PLUS
     WHERE PATH_NAME LIKE '/mypath/%'

This kinda works but is not really fast, either. It even requires a second run because files and 
directories within the tree will be processed in arbitrary order so it will happen quite frequently 
that a directory is going to be deleted before its content has been removed completely. For those 
dirs I see an error message and have to delete afterwards.

I am wondering if there's a quicker way. Given the fact that this is a whole tree I think there's 
should be a quick way to unlink the complete inode hierachy.

Unfortunately we are not using a fileset for that tree...

So are there any ideas how to solve that more efficiently?

Uli
-- 
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196

From jonathan.buzzard at strath.ac.uk  Tue Apr 20 12:47:44 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Tue, 20 Apr 2021 12:47:44 +0100
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
Message-ID: <70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>

On 20/04/2021 12:18, Ulrich Sibiller wrote:
> 
> Hello *,
> 
> I have to delete a subtree of about ~50 million files in thousands of
>  subdirs, ~14TB of data. Running a recursive rm is very slow so I
> setup a simple policy file:
> 
> RULE 'delstuff' DELETE
>    DIRECTORIES_PLUS >    WHERE PATH_NAME LIKE '/mypath/%'
> 
> This kinda works but is not really fast, either. It even requires a 
> second run because files and directories within the tree will be
> processed in arbitrary order so it will happen quite frequently that
> a directory is going to be deleted before its content has been 
> removed completely. For those dirs I see an error message and have to
> delete afterwards.
> 

You are going to have to remove all the inodes and hence the speed is 
going to be dependant on your metadata performance no matter how you 
approach the problem. I doubt that you are going to get much better than 
a recursive rm.

You could try running a series of parallel rm's attacking subsection of 
the directory tree on different nodes, but I suspect that it will make 
very little difference. You could my sync/restore script to split the 
problem up between different nodes

https://github.com/digitalcabbage/syncrestore

> I am wondering if there's a quicker way. Given the fact that this is
> a whole tree I think there's should be a quick way to unlink the
> complete inode hierachy.
> 
> Unfortunately we are not using a fileset for that tree...
> 
> So are there any ideas how to solve that more efficiently?
> 

Does it matter that it takes a long time? Use screen or tmux to run the 
command in the background safe from accidental detaches and forget 
about. Check on it now and then to see how it's going or just forget 
about it, it will finish. Something like

screen -S filedel -L -Logfile /tmp/filedel.log -d -m /bin/rm -rf <tree>

Consider using mv to move it out the way or hide it while the delete is 
in progress. If you do that think carefully about backups, you don't 
want to back it all up again while it is being deleted :-)


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From alvise.dorigo at psi.ch  Tue Apr 20 12:50:08 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 11:50:08 +0000
Subject: [gpfsug-discuss] R: [sls-htc-admins] NFSIO metrics absent in
	pmcollector
In-Reply-To: <CAHwPatiiPro-62OEbmL2ta_Ycp=ZyJCoUCjJ0YBwomihDvh9yA@mail.gmail.com>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<CAHwPatiiPro-62OEbmL2ta_Ycp=ZyJCoUCjJ0YBwomihDvh9yA@mail.gmail.com>
Message-ID: <0f96be99814648bd8e405389cafe5ba9@psi.ch>

Hello, Jan

The answer is definitely yes to both questions ?

  A

Da: sls-htc-admins-request at lists.psi.ch <sls-htc-admins-request at lists.psi.ch> Per conto di Jan-Frode Myklebust
Inviato: marted? 20 aprile 2021 12:31
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: sls-htc-admins at lists.psi.ch
Oggetto: Re: [sls-htc-admins] [gpfsug-discuss] NFSIO metrics absent in pmcollector

Have you installed the gpfs.pm-ganesha package, and do you have any active NFS exports/clients ?


  -jf

On Tue, Apr 20, 2021 at 12:19 PM Dorigo Alvise (PSI) <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>> wrote:
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
        name = "NFSIO"
        period = 5
},
{
        name = "SMBGlobalStats"
        period = 5
},
{
        name = "SMBStats"
        period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch<http://xbl-ces-91.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch<http://xbl-ces-91.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch<http://xbl-ces-92.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch<http://xbl-ces-92.psi.ch>|GPFSFilesystem|lenovoxbl.psi.ch<http://lenovoxbl.psi.ch>|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

   Alvise
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e9f2934a/attachment-0002.htm>

From alvise.dorigo at psi.ch  Tue Apr 20 12:51:19 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 11:51:19 +0000
Subject: [gpfsug-discuss] R:  NFSIO metrics absent in pmcollector
In-Reply-To: <OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
Message-ID: <7dfae5dee31a46068457c8398ba154ae@psi.ch>

Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss-bounces at spectrumscale.org; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________



Phone:

+49 162 4159920

IBM Deutschland GmbH

[cid:image002.png at 01D735EC.3C210570]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________

IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562




[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 4659 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.gif
Type: image/gif
Size: 105 bytes
Desc: image003.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 167 bytes
Desc: image004.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/974389fb/attachment-0005.png>

From janfrode at tanso.net  Tue Apr 20 12:52:50 2021
From: janfrode at tanso.net (Jan-Frode Myklebust)
Date: Tue, 20 Apr 2021 13:52:50 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
Message-ID: <CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>

A couple of ideas.

The KC recommends adding WEIGHT(DIRECTORY_HASH) to group deletions within a
directory. Then maybe also do it as a 2-step process, in the same policy
run. Where you delete all non-directories first, and then deletes the
directories in a depth-first order using WEIGTH(Length(PATH_NAME)):


RULE 'delnondir' DELETE
     WEIGHT(DIRECTORY_HASH)
     DIRECTORIES_PLUS
     WHERE PATH_NAME LIKE '/mypath/%' AND NOT MISC_ATTRIBUTES LIKE '%D%'

RULE 'deldir' DELETE
     DIRECTORIES_PLUS
    WEIGHT(Length(PATH_NAME))
     WHERE PATH_NAME LIKE '/mypath/%'  AND MISC_ATTRIBUTES LIKE '%D%'

HTH


On Tue, Apr 20, 2021 at 1:18 PM Ulrich Sibiller <
u.sibiller at science-computing.de> wrote:

>
> Hello *,
>
> I have to delete a subtree of about ~50 million files in thousands of
> subdirs, ~14TB of data.
> Running a recursive rm is very slow so I setup a simple policy file:
>
> RULE 'delstuff' DELETE
>      DIRECTORIES_PLUS
>      WHERE PATH_NAME LIKE '/mypath/%'
>
> This kinda works but is not really fast, either. It even requires a second
> run because files and
> directories within the tree will be processed in arbitrary order so it
> will happen quite frequently
> that a directory is going to be deleted before its content has been
> removed completely. For those
> dirs I see an error message and have to delete afterwards.
>
> I am wondering if there's a quicker way. Given the fact that this is a
> whole tree I think there's
> should be a quick way to unlink the complete inode hierachy.
>
> Unfortunately we are not using a fileset for that tree...
>
> So are there any ideas how to solve that more efficiently?
>
> Uli
> --
> Science + Computing AG
> Vorstandsvorsitzender/Chairman of the board of management:
> Dr. Martin Matzke
> Vorstand/Board of Management:
> Matthias Schempp, Sabine Hohenstein
> Vorsitzender des Aufsichtsrats/
> Chairman of the Supervisory Board:
> Philippe Miltin
> Aufsichtsrat/Supervisory Board:
> Martin Wibbe, Ursula Morgenstern
> Sitz/Registered Office: Tuebingen
> Registergericht/Registration Court: Stuttgart
> Registernummer/Commercial Register No.: HRB 382196
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/af8e2b54/attachment-0002.htm>

From u.sibiller at science-computing.de  Tue Apr 20 13:09:42 2021
From: u.sibiller at science-computing.de (Ulrich Sibiller)
Date: Tue, 20 Apr 2021 14:09:42 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
Message-ID: <d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>

On 4/20/21 1:47 PM, Jonathan Buzzard wrote:
> You are going to have to remove all the inodes and hence the speed is
> going to be dependant on your metadata performance no matter how you
> approach the problem. I doubt that you are going to get much better than
> a recursive rm.
> 
> You could try running a series of parallel rm's attacking subsection of
> the directory tree on different nodes, but I suspect that it will make
> very little difference. You could my sync/restore script to split the
> problem up between different nodes
> 
> https://github.com/digitalcabbage/syncrestore

might be worth a try, thanks.

> Does it matter that it takes a long time? Use screen or tmux to run the

Unfortunately it makes sense since I need the sapce for something else...

> command in the background safe from accidental detaches and forget
> about. Check on it now and then to see how it's going or just forget
> about it, it will finish. Something like
> 
> screen -S filedel -L -Logfile /tmp/filedel.log -d -m /bin/rm -rf <tree>
> 
> Consider using mv to move it out the way or hide it while the delete is
> in progress. If you do that think carefully about backups, you don't
> want to back it all up again while it is being deleted :-)

;-) Yeah, that's why I did not the do the mv in the first place ;-)

Thanks,

Uli


-- 
Dipl.-Inf. Ulrich Sibiller           science + computing ag
System Administration                    Hagellocher Weg 73
Hotline +49 7071 9457 681          72070 Tuebingen, Germany
                           https://atos.net/de/deutschland/sc
-- 
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196

From u.sibiller at science-computing.de  Tue Apr 20 13:09:51 2021
From: u.sibiller at science-computing.de (Ulrich Sibiller)
Date: Tue, 20 Apr 2021 14:09:51 +0200
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>
Message-ID: <084d86ed-38b9-8b7f-a840-863dc6686bb5@science-computing.de>

On 4/20/21 1:52 PM, Jan-Frode Myklebust wrote:
> A couple of ideas.
> 
> The KC recommends adding WEIGHT(DIRECTORY_HASH) to group?deletions within a directory. Then maybe > also do it as a 2-step process, in the same policy run. Where you delete all non-directories first,
> and then deletes the directories in a depth-first order using WEIGTH(Length(PATH_NAME)):
> 
> 
> RULE 'delnondir' DELETE
> WEIGHT(DIRECTORY_HASH)
>  ? ? ?DIRECTORIES_PLUS
>  ? ? ?WHERE PATH_NAME LIKE '/mypath/%' AND NOT MISC_ATTRIBUTES LIKE '%D%'
> 
> RULE 'deldir' DELETE
>  ? ? ?DIRECTORIES_PLUS
>  ? ? WEIGHT(Length(PATH_NAME))
>  ? ? ?WHERE PATH_NAME LIKE '/mypath/%' ?AND MISC_ATTRIBUTES LIKE '%D%'


Thanks, I am aware of that but it will not really help with my speed concerns.

Uli


-- 
Dipl.-Inf. Ulrich Sibiller           science + computing ag
System Administration                    Hagellocher Weg 73
Hotline +49 7071 9457 681          72070 Tuebingen, Germany
                           https://atos.net/de/deutschland/sc
-- 
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196

From alvise.dorigo at psi.ch  Tue Apr 20 13:37:57 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 12:37:57 +0000
Subject: [gpfsug-discuss] R: [EXT] R: NFSIO metrics absent in pmcollector
In-Reply-To: <SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
	<7dfae5dee31a46068457c8398ba154ae@psi.ch>
	<SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>
Message-ID: <8f597b3c85cd4f6ea8448f3fc49ce5da@psi.ch>

Hello Larry,
I?ve implemented your suggestion to my perfmon configuration (proxy, restriction to relevat node, type).
But the problem is still there.
I guess I need to open a PMR to IBM, if not other clue?

    Alvise

Da: sls-htc-admins-request at lists.psi.ch <sls-htc-admins-request at lists.psi.ch> Per conto di Henson Jr.,Larry J
Inviato: marted? 20 aprile 2021 14:22
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss-bounces at spectrumscale.org; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>
Oggetto: RE: [sls-htc-admins] [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

Hello Dorigo,

This is what I have on the end of our ZIMonSensors.cfg file.  Without it the GUI shows no NFS/SMB metrics.

{
        name = "NFSIO"
        period = 10
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBGlobalStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Regards,
Larry Henson
IT Operations Storage Team
Office (832) 750-1403
Cell (713) 702-4896
[cid:image001.png at 01D735F2.BFA83A70]
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of Dorigo Alvise (PSI)
Sent: Tuesday, April 20, 2021 6:51 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Subject: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

WARNING: This email originated from outside of MD Anderson. Please validate the sender's email address before clicking on links or attachments as they may not be safe.


Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYLEsnxBNw$>

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________



Phone:

+49 162 4159920

IBM Deutschland GmbH

[cid:image003.png at 01D735F2.BFA83A70]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________

IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562




[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYI-Ipx8NA$>
The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 19425 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 166 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 4659 bytes
Desc: image003.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.gif
Type: image/gif
Size: 105 bytes
Desc: image004.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/64c7213f/attachment-0002.gif>

From stockf at us.ibm.com  Tue Apr 20 14:00:50 2021
From: stockf at us.ibm.com (Frederick Stock)
Date: Tue, 20 Apr 2021 13:00:50 +0000
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <084d86ed-38b9-8b7f-a840-863dc6686bb5@science-computing.de>
References: <084d86ed-38b9-8b7f-a840-863dc6686bb5@science-computing.de>,
	<f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de><CAHwPatgq2U85=e4ZV4gNXwEMj6r0siG0tLoduH_qpz=oqNqXvA@mail.gmail.com>
Message-ID: <OFB96766C9.C77426E0-ON002586BD.0046E2E0-002586BD.00477CF1@notes.na.collabserv.com>

An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/74ce6742/attachment-0002.htm>

From LJHenson at mdanderson.org  Tue Apr 20 13:21:34 2021
From: LJHenson at mdanderson.org (Henson Jr.,Larry J)
Date: Tue, 20 Apr 2021 12:21:34 +0000
Subject: [gpfsug-discuss] [EXT] R: NFSIO metrics absent in pmcollector
In-Reply-To: <7dfae5dee31a46068457c8398ba154ae@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
	<7dfae5dee31a46068457c8398ba154ae@psi.ch>
Message-ID: <SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>

Hello Dorigo,

This is what I have on the end of our ZIMonSensors.cfg file.  Without it the GUI shows no NFS/SMB metrics.

{
        name = "NFSIO"
        period = 10
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBGlobalStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Regards,
Larry Henson
IT Operations Storage Team
Office (832) 750-1403
Cell (713) 702-4896
[cid:image001.png at 01D735B5.C8961270]
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Dorigo Alvise (PSI)
Sent: Tuesday, April 20, 2021 6:51 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss-bounces at spectrumscale.org; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>
Subject: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

WARNING: This email originated from outside of MD Anderson. Please validate the sender's email address before clicking on links or attachments as they may not be safe.

Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYLEsnxBNw$>

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________

Phone:
+49 162 4159920
IBM Deutschland GmbH
[cid:image006.png at 01D735B5.C8961270]
E-Mail:
rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>
Am Weiher 24


65451 Kelsterbach


Germany
________________________________
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562


[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYI-Ipx8NA$>


The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 19425 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 4659 bytes
Desc: image006.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.gif
Type: image/gif
Size: 105 bytes
Desc: image007.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 166 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/3d7d9898/attachment-0008.png>

From alvise.dorigo at psi.ch  Tue Apr 20 14:39:03 2021
From: alvise.dorigo at psi.ch (Dorigo Alvise (PSI))
Date: Tue, 20 Apr 2021 13:39:03 +0000
Subject: [gpfsug-discuss] R: [EXT] R: NFSIO metrics absent in pmcollector
In-Reply-To: <8f597b3c85cd4f6ea8448f3fc49ce5da@psi.ch>
References: <f1cd3eb6439040ff98a4c9f496105af9@psi.ch>
	<OF31DF44B6.CE4FF8E6-ON002586BD.003985E9-C12586BD.003A2298@notes.na.collabserv.com>
	<7dfae5dee31a46068457c8398ba154ae@psi.ch>
	<SN6PR14MB246290EDF605CBA85EC923B2BE489@SN6PR14MB2462.namprd14.prod.outlook.com>
	<8f597b3c85cd4f6ea8448f3fc49ce5da@psi.ch>
Message-ID: <c0853ab6defc496692d8c26246562003@psi.ch>

Hello again,
found the reason: no client had already generated any traffic (because this is just a test cluster).
Now, some read/write has been done, and the related table was created in the collector?s DB.

Thanks and sorry, I probably missed this detail.

   Alvise

Da: sls-htc-admins-request at lists.psi.ch <sls-htc-admins-request at lists.psi.ch> Per conto di Dorigo Alvise (PSI)
Inviato: marted? 20 aprile 2021 14:38
A: 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch>; 'Henson Jr.,Larry J' <LJHenson at mdanderson.org>; 'gpfsug main discussion list' <gpfsug-discuss at spectrumscale.org>
Cc: 'gpfsug-discuss-bounces at spectrumscale.org' <gpfsug-discuss-bounces at spectrumscale.org>
Oggetto: [sls-htc-admins] R: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

Hello Larry,
I?ve implemented your suggestion to my perfmon configuration (proxy, restriction to relevat node, type).
But the problem is still there.
I guess I need to open a PMR to IBM, if not other clue?

    Alvise

Da: sls-htc-admins-request at lists.psi.ch<mailto:sls-htc-admins-request at lists.psi.ch> <sls-htc-admins-request at lists.psi.ch<mailto:sls-htc-admins-request at lists.psi.ch>> Per conto di Henson Jr.,Larry J
Inviato: marted? 20 aprile 2021 14:22
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: RE: [sls-htc-admins] [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

Hello Dorigo,

This is what I have on the end of our ZIMonSensors.cfg file.  Without it the GUI shows no NFS/SMB metrics.

{
        name = "NFSIO"
        period = 10
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
},
{
        name = "SMBGlobalStats"
        period = 10
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Regards,
Larry Henson
IT Operations Storage Team
Office (832) 750-1403
Cell (713) 702-4896
[cid:image001.png at 01D735FB.48EDDAD0]
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of Dorigo Alvise (PSI)
Sent: Tuesday, April 20, 2021 6:51 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Subject: [EXT] [gpfsug-discuss] R: NFSIO metrics absent in pmcollector

WARNING: This email originated from outside of MD Anderson. Please validate the sender's email address before clicking on links or attachments as they may not be safe.


Hello Markus,
ganesha is installed, but I will try what you suggest about configuration.

   A

Da: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> Per conto di Markus Rohwedder
Inviato: marted? 20 aprile 2021 12:35
A: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>; 'sls-htc-admins at lists.psi.ch' <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Oggetto: Re: [gpfsug-discuss] NFSIO metrics absent in pmcollector


Hello Dorigo,

in contrast to the basic Spectrum Scale metrics, the NFS and SMB sensors rely on a proxy mechanism that provides the metrics out of the CES stack.
In case you have not used the installer, you may miss some steps.

- Please check if you have the requires packages installed.
(For example gpfs.pm-ganesha-10.0.0-2.el8.x86_64 or similar for NFS)

- The config section should have the "Generic" keyword, like this

{
name = "NFSIO"
period = 10
restrict = "cesNodes"
type = "Generic"
},

See here in the KC:
https://www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/spectrum-scale/5.0.5?topic=tool-enabling-protocol-metrics__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYLEsnxBNw$>

/var/log/zimon/ZIMonSensors.log may provides some additional clues as well.

Mit freundlichen Gr??en / Kind regards

Dr. Markus Rohwedder

IBM Systems / Lab Services Europe / EMEA Storage Competence Center

________________________________



Phone:

+49 162 4159920

IBM Deutschland GmbH

[cid:image003.png at 01D735FB.48EDDAD0]

E-Mail:

rohwedder at de.ibm.com<mailto:rohwedder at de.ibm.com>

Am Weiher 24





65451 Kelsterbach





Germany

________________________________

IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Sebastian Krause / Gesch?ftsf?hrung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer / Sitz der Gesellschaft: 71139 Ehningen, IBM-Allee 1 / Registergericht: Amtsgericht Stuttgart, HRB14562




[Inactive hide details for "Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by]"Dorigo Alvise (PSI)" ---20.04.2021 12:19:44---Dear Community, I've activated CES-related metrics by simply doing:

From: "Dorigo Alvise (PSI)" <alvise.dorigo at psi.ch<mailto:alvise.dorigo at psi.ch>>
To: "'gpfsug-discuss at spectrumscale.org'" <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Cc: "'sls-htc-admins at lists.psi.ch'" <sls-htc-admins at lists.psi.ch<mailto:sls-htc-admins at lists.psi.ch>>
Date: 20.04.2021 12:19
Subject: [EXTERNAL] [gpfsug-discuss] NFSIO metrics absent in pmcollector
Sent by: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>

________________________________



Dear Community, I?ve activated CES-related metrics by simply doing: [root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB' name = "NFSIO" period = 5 ??????????????????????????????????????????????????????????????????ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Dear Community,

I?ve activated CES-related metrics by simply doing:

[root at xbl-ces-91 ~]# mmperfmon config show |egrep -A4 'NFS|SMB'
name = "NFSIO"
period = 5
},
{
name = "SMBGlobalStats"
period = 5
},
{
name = "SMBStats"
period = 5
}

Despite that, I get no result for any nfs* metrics:

--------------------------------------------
[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics nfs_read bucket_size 5 last 1
Error: no data available for query
.
Connection closed by foreign host.

[root at xbl-ces-91 ~]# mmperfmon query "nfs_read"
Error: no data available for query
.

mmperfmon: Command failed. Examine previous error messages to determine cause.
--------------------------------------------


I am pretty sure my collector is working correctly as it returns data for a different metrics:

[root at xbl-ces-91 ~]# telnet localhost 9084
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get -j metrics gpfs_fs_bytes_read last 20 bucket_size 1
{"header" : {"bcount" : 20,"bsize" : 1,"t_start" : 1618912752,"t_end" : 1618912772},"legend" : [{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-91.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|perf|gpfs_fs_bytes_read"]},{"caption" : "gpfs_fs_bytes_read","semType" : 2,"aggrType" : "value difference between start and end of the bucket","type" : 4,"keys" : ["xbl-ces-92.psi.ch|GPFSFilesystem|lenovoxbl.psi.ch|tiered|gpfs_fs_bytes_read"]}],"rows" :
[?]

Any clue ? Is the activation of NFSIO enough in the mmperfmon?s config file (this is the first time I try to configure CES monitoring) ?

Thank you,

Alvise_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!PfbeBCCAmug!2C8HSpiMkc87P7g8ryyOodOb1MgL5CwNKOr12XJxMIW3BaEKcvf8kyDdMYI-Ipx8NA$>
The information contained in this e-mail message may be privileged, confidential, and/or protected from disclosure. This e-mail message may contain protected health information (PHI); dissemination of PHI should comply with applicable federal and state laws. If you are not the intended recipient, or an authorized representative of the intended recipient, any further review, disclosure, use, dissemination, distribution, or copying of this message or any attachment (or the information contained therein) is strictly prohibited. If you think that you have received this e-mail message in error, please notify the sender by return e-mail and delete all references to it and its contents from your systems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 19425 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 166 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 4659 bytes
Desc: image003.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.gif
Type: image/gif
Size: 105 bytes
Desc: image004.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/e5a32952/attachment-0002.gif>

From jonathan.buzzard at strath.ac.uk  Tue Apr 20 14:51:35 2021
From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard)
Date: Tue, 20 Apr 2021 14:51:35 +0100
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
	<d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>
Message-ID: <c19dc2fa-448c-d6a5-2a17-56f7fd307a41@strath.ac.uk>

On 20/04/2021 13:09, Ulrich Sibiller wrote:

>>
>> Consider using mv to move it out the way or hide it while the delete is
>> in progress. If you do that think carefully about backups, you don't
>> want to back it all up again while it is being deleted :-)
> 
> ;-) Yeah, that's why I did not the do the mv in the first place ;-)
> 

I would estimate (based on my experience) is that you should be able to 
delete that amount of data/files in under 24 hours anyway with a simple 
rm -rf. Which is why I question trying to find faster methods. You have 
already wasted a significant amount of that time :-)

If your using TSM for the backup then just exclude it from the backup in 
your dsm.opts file

exclude.dir  <path to directory>

We have a NOBACK option that allows users to select if they don't want 
something backing up. Helpful if your job generates lots of temporary 
files or data that are junked as soon as the job finishes. Anything 
under a directory called NOBACK does not get backed up.

exclude.dir  /.../NOBACK/


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


From anacreo at gmail.com  Tue Apr 20 17:44:53 2021
From: anacreo at gmail.com (Alec)
Date: Tue, 20 Apr 2021 09:44:53 -0700
Subject: [gpfsug-discuss] Quick delete of huge tree
In-Reply-To: <c19dc2fa-448c-d6a5-2a17-56f7fd307a41@strath.ac.uk>
References: <f829dfbc-da3b-b922-ad97-9eaf2b7d394e@science-computing.de>
	<70be359d-8c64-3505-e725-7428298b89df@strath.ac.uk>
	<d7ece217-c9ef-476b-1cb1-822d6732ea58@science-computing.de>
	<c19dc2fa-448c-d6a5-2a17-56f7fd307a41@strath.ac.uk>
Message-ID: <CAGhSTwhiLq_uk1f1NKMyfL_RHjGeHfjhM5eLz2HeXU9fkOmujA@mail.gmail.com>

I would start with the mv to hide it, and then allow the delete to progress
in the background.  I would seperate out the delete of files from
directories... And i would try using mmxargs with a rm command to get
parallelism plus reduce the number of execs in one policy, followed by a
simple rm -r of the directory tree.

Maybe it's cheaper to make a new filesystem and just retain the data you
want though...

Alec

On Tue, Apr 20, 2021, 6:51 AM Jonathan Buzzard <
jonathan.buzzard at strath.ac.uk> wrote:

> On 20/04/2021 13:09, Ulrich Sibiller wrote:
>
> >>
> >> Consider using mv to move it out the way or hide it while the delete is
> >> in progress. If you do that think carefully about backups, you don't
> >> want to back it all up again while it is being deleted :-)
> >
> > ;-) Yeah, that's why I did not the do the mv in the first place ;-)
> >
>
> I would estimate (based on my experience) is that you should be able to
> delete that amount of data/files in under 24 hours anyway with a simple
> rm -rf. Which is why I question trying to find faster methods. You have
> already wasted a significant amount of that time :-)
>
> If your using TSM for the backup then just exclude it from the backup in
> your dsm.opts file
>
> exclude.dir  <path to directory>
>
> We have a NOBACK option that allows users to select if they don't want
> something backing up. Helpful if your job generates lots of temporary
> files or data that are junked as soon as the job finishes. Anything
> under a directory called NOBACK does not get backed up.
>
> exclude.dir  /.../NOBACK/
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                         Tel: +44141-5483420
> HPC System Administrator, ARCHIE-WeSt.
> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210420/86cec6ae/attachment-0002.htm>

From tomasz.rachobinski at gmail.com  Mon Apr 26 11:10:04 2021
From: tomasz.rachobinski at gmail.com (=?UTF-8?Q?Tomasz_Rachobi=C5=84ski?=)
Date: Mon, 26 Apr 2021 12:10:04 +0200
Subject: [gpfsug-discuss] Hello from Poland
Message-ID: <CAOnndBP6Db087Z8Gd=h8Kud743dBMZuM=7Lgh31uoAdPQhOjYQ@mail.gmail.com>

Hello All,
My name is Tom, I'm from Warsaw, Poland. I work for polish Bank PKO,
usually I'm a mainframe sysprog but for 1 year I'm learning a lot of gpfs -
in my case it's Spectrum Scale Data Access due to mixed Linux/Windows env.
I have many questions in my head, but I'll try not to bore You too much :)

Greetings
Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210426/2eee42e4/attachment-0002.htm>