[gpfsug-discuss] Backend corruption

Stef Coene stef.coene at docum.org
Tue Aug 4 07:41:26 BST 2020


Hi,

I tried to use a policy to find out what files are located on the broken 
disks.
But this is not finding any files or directories (I cleaned some of the 
output):

[I] GPFS Current Data Pool Utilization in KB and %
Pool_Name                   KB_Occupied        KB_Total  Percent_Occupied
V500003                       173121536     69877104640      0.247751444%

[I] 29609813 of 198522880 inodes used: 14.915063%.

[I] Loaded policy rules from test.rule.
rule 'ListRule'
    list 'ListName'
       from pool 'V500003'

[I] Directories scan: 28649029 files, 960844 directories, 0 other 
objects, 0 'skipped' files and/or errors.

[I] Inodes scan: 28649029 files, 960844 directories, 0 other objects, 0 
'skipped' files and/or errors.

[I] Summary of Rule Applicability and File Choices:
  Rule#      Hit_Cnt          KB_Hit          Chosen       KB_Chosen 
      KB_Ill     Rule
      0            0               0               0               0 
           0     RULE 'ListRule' LIST 'ListName' FROM POOL 'V500003'

[I] Filesystem objects with no applicable rules: 29609873.

[I] A total of 0 files have been migrated, deleted or processed by an 
EXTERNAL EXEC/script;
         0 'skipped' files and/or errors.


So the policy is not finding any files but there is still some data on 
the V50003 pool?


Stef

On 2020-08-03 17:21, Uwe Falke wrote:
> Hi, Stef,
> 
> if just that V5000 has provided the storage for one of your pools
> entirely, and if your metadata are still incorrupted, a inode scan with a
> suited policy should yield the list of files on that pool.
> If I am not mistaken, the list policy could look like
> 
> RULE 'list_v5000'  LIST 'v5000_filelist'  FROM POOL <your_v5000_pool>
> 
> paut it into a (policy) file,  run that by mmapplypolicy against the file
> system in question, it should produce a file listing in
> /tmp/v5000_filelist. If it doesn#T work exactly like that (I might have
> made one or mor mistakes), check out the information lifycacle section in
> the scal admin guide.
> 
> If the prereqs for the above are not met, you need to run more expensive
> investigations (using tsdbfs for all block addresses on v5000-provided
> NSDs).
> 
> Mit freundlichen Grüßen / Kind regards
> 
> Dr. Uwe Falke
> IT Specialist
> Global Technology Services / Project Services Delivery / High Performance
> Computing
> +49 175 575 2877 Mobile
> Rathausstr. 7, 09111 Chemnitz, Germany
> uwefalke at de.ibm.com
> 
> IBM Services
> 
> IBM Data Privacy Statement
> 
> IBM Deutschland Business & Technology Services GmbH
> Geschäftsführung: Dr. Thomas Wolter, Sven Schooss
> Sitz der Gesellschaft: Ehningen
> Registergericht: Amtsgericht Stuttgart, HRB 17122
> 
> 
> 
> From:   Stef Coene <stef.coene at docum.org>
> To:     gpfsug-discuss at spectrumscale.org
> Date:   03/08/2020 16:07
> Subject:        [EXTERNAL] [gpfsug-discuss] Backend corruption
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> 
> 
> 
> Hi,
> 
> We have a GPFS file system which uses, among other storage, a V5000 as
> backend.
> There was an error in the fire detection alarm in the datacenter and a
> fire alarm was triggered.
> The result was that the V5000 had a lot of broken disks. Most of the
> disks recovered fine after a reseat, but some data is corrupted on the
> V5000.
> 
> This means that for 22MB of data, the V5000 returns a read error to the
> GPFS.
> 
> We migrated most of the data to an disks but there is still 165 GB left
> on the V5000 pool.
> 
> When we try to remove the disks with mmdeldisk, it fails after a while
> and places some of the disks as down.
> It generated a file with inodes, this an example of a 2 lines:
>    9168519      0:0        0           1                 1
>     exposed illreplicated illplaced REGULAR_FILE Error: 218 Input/output
> error
>    9251611      0:0        0           1                 1
>     exposed illreplicated REGULAR_FILE Error: 218 Input/output error
> 
> 
> How can I get a list of files that uses data of the V5000 pool?
> The data is written by CommVault. When I have a list of files, I can
> determine the impact on the application.
> 
> 
> Stef
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=fTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8&m=HhbxQEWLNTXFDCFT5LDpMD4YvYTUEdl6Nt6IgjdVlNo&s=fxsoDddp4OUnP7gORNUOnAmrnHPIU57OQMnraXEEO0k&e=
> 
> 
> 
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 



More information about the gpfsug-discuss mailing list