[gpfsug-discuss] mass recall from on-prem COS using a policy
Paul Ward
p.ward at nhm.ac.uk
Fri Jul 29 11:51:25 BST 2022
Dear Anuj and Amey,
Thank you so much for your testing…
I repeated my tests yesterday, and actually ran the policy.
All the files recalled ! yay!
A few minutes later, I checked and half of the files were migrated again, and a little while later they were all migrated…. !?
This is when I remembered I have a policy running every 30 mins with a rule for the parent folder to migrate any files not accessed for 90 days:
/* Workspace Area rules ================================================================= */
/* Workspace Area DEFAULT rule ================================================================= */
RULE 'migrate_bulk_workspace_90' MIGRATE FROM POOL 'data' TO POOL 'migrate' WHERE (DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME) > 90) AND (PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/%')
This is what is migrating the files straight back after I ran the specific policy to recall them!
I had the recall rule in the same policy, I am guessing this was conflicting with the default migrate rule:
/* Workspace Area SPECIFIC rules ================================================================= */
RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE (PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%')
So I have added this exclude:
/* Exclude rules ================================================================= */
RULE 'exclude_bulk_workspace_axioimager_2021' EXCLUDE WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%'
And now the recalled files stay recalled 😊
========================
I will try and test today with putting recall rules higher in the policy file, before default parent migrate rules.
The rest of this email I wrote before realising the rule conflict….
OK so you are getting the same results and they are working as designed…
Do the quantities on disk match the amounts the policy found?
14:56:13 [root at scale-sk-pn-1 2021]# du -hs .
1.3M .
(need to half this value as we have two mirrored ESS, storing 2 copies)
14:56:22 [root at scale-sk-pn-1 2021]# du -bhs .
747G .
Compared to the policy result:
[I] Summary of Rule Applicability and File Choices:
Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule
0 996 736 996 1566374532 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.)
KB_chosen:
1,566,374,532 = 1,566 GB more than double the amount du reports?
----------------------------------------------
I actually ran the policy:
Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule
0 996 736 996 1566374532 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.)
[I] Filesystem objects with no applicable rules: 53.
[I] GPFS Policy Decisions and File Choice Totals:
Chose to migrate 1566374532KB: 996 of 996 candidates;
1566373796KB of chosen data is illplaced or illreplicated;
Predicted Data Pool Utilization in KB and %:
Pool_Name KB_Occupied KB_Total Percent_Occupied
data 173720590244 245111980032 70.873969612%
system 0 0 0.000000000% (no user data)
[I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading!
<1> mmcloudgateway: Command completed.cution. 996 files dispatched. ....|...
<1> mmcloudgateway: Command completed.cution. 996 files dispatched. ......-.
<1> mmcloudgateway: Command completed.cution. 996 files dispatched. /.......
<1> mmcloudgateway: Command completed.cution. 996 files dispatched. ....>...
<1> mmcloudgateway: Command completed.cution. 996 files dispatched. ..#.....
<1> mmcloudgateway: Command completed.cution. 996 files dispatched. ...@<mailto:...@>....
<1> mmcloudgateway: Command completed.cution. 996 files dispatched. .....&..
<1> mmcloudgateway: Command completed.
<1> mmcloudgateway: Command completed.cution. 996 files dispatched. ../.....
<1> mmcloudgateway: Command completed.cution. 996 files dispatched. ....<...
[I] 2022-07-28 at 14:32:48.892<mailto:2022-07-28 at 14:32:48.892> Policy execution. 996 files dispatched.
[I] A total of 996 files have been migrated, deleted or processed by an EXTERNAL EXEC/script;
0 'skipped' files and/or errors.
After recall:
15:32:48 [root at scale-sk-pn-1 2021]# du -hs .
1003G .
15:33:36 [root at scale-sk-pn-1 2021]# du -bhs .
747G .
So something has come back…
When I check what files are migrated and what are resident:
Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule
0 405 0 405 0 0 RULE 'migrated' LIST 'migrated_list' WHERE(.)
1 592 1050944608 592 1050944608 0 RULE 'resident' LIST 'resident_list' WHERE(.)
not all of the files have been recalled?
This is when I realised about the migrate rule running every 30 mins.
I will update after re-ordering the rules.
Paul
From: Anuj Agnihotri <Anuj.Agnihotri at ibm.com>
Sent: 28 July 2022 13:48
To: Paul Ward <p.ward at nhm.ac.uk>
Cc: Amey P Gokhale <amey.gokhale at in.ibm.com>; gpfsug-discuss at gpfsug.org
Subject: RE: [gpfsug-discuss] mass recall from on-prem COS using a policy
Hi Paul,
We were able to recreate the following scenario on our end and it is working as designed. I have created 5 files of 20MB each and migrated it to the cloud and then recall them via policy. When you recall the files using policy it decodes the ‘%2F’ to ‘/’ character as generated in the policy-list file and when you try to recall manually using the path generated by policy it will fail as it is not the actual path. Below is the output attached:
1. [root at c350f3u1 ~]# mmapplypolicy /gpfs0 -f /gpfs0/s3default/anujnhm/policy-list -I defer -L 3 -P recallFromCloud.policy.template
[I] Summary of Rule Applicability and File Choices:
Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule
0 5 20480 5 102400 102400 RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE(.)
[I] Filesystem objects with no applicable rules: 349.
[I] GPFS Policy Decisions and File Choice Totals:
Chose to migrate 102400KB: 5 of 5 candidates;
102400KB of chosen data is illplaced or illreplicated;
Predicted Data Pool Utilization in KB and %:
Pool_Name KB_Occupied KB_Total Percent_Occupied
system 28884992 31457280 91.822916667%
[I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading!
[I] 2022-07-28 at 05:00:07.154<mailto:2022-07-28 at 05:00:07.154> Policy execution. 0 files dispatched.
[I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script;
0 'skipped' files and/or errors.
2. [root at c350f3u1 anujnhm]# head policy-list.recall.mcstore
156416 97874145 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_1
156417 1726931404 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2
156418 1426209538 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_3
156419 1880128499 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_4
156420 1841735731 0 -- %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_5
3. [root at c350f3u1 anujnhm]# mmcloudgateway files recall %2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2
mmcloudgateway: Internal cloud services returned an error:
file system object /gpfs0/s3default/anujnhm/%2Fgpfs0%2Fs3default%2Fanujnhm%2F20MB_file_2 does not exist
mmcloudgateway: Command failed. Examine previous error messages to determine cause.
When TCT recalls files directly via policy, we do %2F decoding that the policy adds, but when we do explicit recalls using the standalone command, we do not do any decoding, because we don't expect files to be passed with any decoding to the explicit command.
Thanks,
Anuj Agnihotri
From: Amey P Gokhale <amey.gokhale at in.ibm.com<mailto:amey.gokhale at in.ibm.com>>
Date: Friday, 22 July 2022 at 2:42 PM
To: Anuj Agnihotri <Anuj.Agnihotri at ibm.com<mailto:Anuj.Agnihotri at ibm.com>>
Subject: FW: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy
FYI -
AMEY GOKHALE
Senior Software Engineer – Spectrum Scale
Phone: 91-988 100 8675
E-mail: amey.gokhale at in.ibm.com<mailto:amey.gokhale at in.ibm.com>
From: Huzefa H Pancha <hupancha at in.ibm.com<mailto:hupancha at in.ibm.com>> On Behalf Of scale at us.ibm.com<mailto:scale at us.ibm.com>
Sent: 22 July 2022 00:49
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Cc: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>>; Amey P Gokhale <amey.gokhale at in.ibm.com<mailto:amey.gokhale at in.ibm.com>>
Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy
Adding back Amey in the chain.
Paul please explicitly add Amey in your replies.
Regards, The Spectrum Scale (GPFS) team
------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdeveloperworks%2Fcommunity%2Fforums%2Fhtml%2Fforum%3Fid%3D11111111-0000-0000-0000-000000000479&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cc47cc96d12ae4c07575f08da7097d3f5%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637946094988077487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ckqDPlU9imJlW2byM9iRDcN0H80nim03%2BL26Qq3Glhc%3D&reserved=0>.
If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries.
The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.
[Inactive hide details for "Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: #]"Paul Ward" ---20-07-2022 08.20.55 PM---I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migr
From: "Paul Ward" <p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>>
To: "gpfsug main discussion list" <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: 20-07-2022 08.20 PM
Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy
Sent by: "gpfsug-discuss" <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>>
________________________________
I used the template policy, substituting where applicable: # more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol /*******************************************************************************
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I used the template policy, substituting where applicable:
# more /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol
/*******************************************************************************
* Licensed Materials - Property of IBM
*
* OCO Source Materials
*
* (C) Copyright IBM Corp. 2016-2017 All Rights Reserved
*
* The source code for this program is not published or other-
* wise divested of its trade secrets, irrespective of what has
* been deposited with the U.S. Copyright Office.
*******************************************************************************/
define(
exclude_list,
(
FALSE
OR PATH_NAME LIKE '%/.mcstore/%'
)
)
RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -'
/*
RULE EXTERNAL POOL 'mcstore' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -'
RULE 'dmrecall1' MIGRATE FROM POOL 'mcstore' TO POOL 'system' WHERE
(
(
PATH_NAME LIKE '/gpfs/test_dir/%'
)
AND NOT (exclude_list)
)
*/
RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE
(
(
PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%'
)
AND NOT (exclude_list)
)
-----------------------------------
I changed to the folder listed in the path and ran this command, with the defer option:
# pwd
/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021
]# mmapplypolicy $(pwd) -f $(pwd)/policy-lists -I defer -L 3 -P /gpfs/nhmfsa/custom/valentine-migrations/policy/iac-workspace_axio-imager_2021.pol
The result of the policy was:
[I] Summary of Rule Applicability and File Choices:
Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule
0 994 0 994 1566373796 1566373796 RULE 'migrate_bulk_workspace_axioimager_2021' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.)
[I] Filesystem objects with no applicable rules: 53.
[I] GPFS Policy Decisions and File Choice Totals:
Chose to migrate 1566373796KB: 994 of 994 candidates;
1566373796KB of chosen data is illplaced or illreplicated;
Predicted Data Pool Utilization in KB and %:
Pool_Name KB_Occupied KB_Total Percent_Occupied
data 175459129252 245111980032 71.583253185%
system 0 0 0.000000000% (no user data)
[I] Because some data is illplaced or illreplicated, predicted pool utilization may be negative and/or misleading!
[I] 2022-07-20 at 14:22:23.943<mailto:2022-07-20 at 14:22:23.943> Policy execution. 0 files dispatched.
[I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script;
0 'skipped' files and/or errors.
---------------------------------------------------------------------
Its selected about the right number, as there are :
# find -type f | wc -l
996
--------------------------------------------------------
the head of the deferred output file list:
15:27:41 [root at scale-sk-pn-1 2021]# head policy-lists.recall.migrate
9085980 2146238895 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FRobin Armstrong 2021%2FThumbs.db
30157518 903677244 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_RL_s0c0x0-8849y0-8889.tif
30157519 321389117 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL.czi
30157520 1970925505 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8846y0-8892.tif
30157521 1773348368 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-113_TL_s0c0x0-8849y0-8889.tif
30157522 2126423502 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL.czi
30157523 1701851598 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_RL_s0c0x0-6359y0-6656.tif
30157524 1844809347 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL.czi
30157525 912638442 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-6656.tif
30157526 1280698844 0 -- %2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif
….
--------------------------
If I try to recall the file from these paths:
# mmcloudgateway files recall "%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif"
mmcloudgateway: Internal cloud services returned an error:
file system object /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/%2Fgpfs%2Fnhmfsa%2Fbulk%2Fshare%2Fworkspace%2Fiac-workspace%2Flight_microscopy%2Fzeiss_axioimager_m2%2F2021%2FMatt Loader 2021%2FML210215_OT-29_TL_s0c0x0-6359y0-7609.tif does not exist
mmcloudgateway: Command failed. Examine previous error messages to determine cause.
It seems the highlighted part is the current directory, as I tried from another location and it picked up the pwd I was in.
-----------------------
If I Give it the correctly formed path, it works:
# mmcloudgateway files recall "/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif"
mmcloudgateway: Command completed.
------------------------------------------
I also ran a policy just to identify migrated files:
define(ismigrated,
( MISC_ATTRIBUTES LIKE '%V%') )
Rule 'migrated' LIST 'migrated_list' WHERE ismigrated
Rule 'resident' LIST 'resident_list' WHERE not (ismigrated)
----------------------------------------------------------------
The head of that file correctly forms the posix path…
15:35:38 [root at scale-sk-pn-1 2021]# head list.migrated_list
9085980 2146238895 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Robin Armstrong 2021/Thumbs.db
30157518 903677244 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_RL_s0c0x0-8849y0-8889.tif
30157519 321389117 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL.czi
30157520 1970925505 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8846y0-8892.tif
30157521 1773348368 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-113_TL_s0c0x0-8849y0-8889.tif
30157522 2126423502 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL.czi
30157523 1701851598 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_RL_s0c0x0-6359y0-6656.tif
30157524 1844809347 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL.czi
30157525 912638442 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-6656.tif
30157526 1280698844 0 -- /gpfs/nhmfsa/bulk/share/workspace/iac-workspace/light_microscopy/zeiss_axioimager_m2/2021/Matt Loader 2021/ML210215_OT-29_TL_s0c0x0-6359y0-7609.tif
What’s going on?
What I have done in the past is take the output from my ‘ismigrated’ policy.
Put recall commands before each path.
Divide the file into 4 parts, and run each part of one of our 4 protocol nodes.
Manually doing what mmapplypolicy should do!
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Amey P Gokhale
Sent: 20 July 2022 07:55
To: scale at us.ibm.com<mailto:scale at us.ibm.com>; gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Cc: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>>
Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy
Paul – Recall policy looks to be correct. I see that it is in line with /opt/ibm/MCStore/samples/recallFromCloud.template.
1. At the end of policy run, what does the stats indicate, such as “X Files migrated, Y skipped/error” etc? I assume the path you have specified, has the files in ‘non-resident’ state, which you are trying to recall using the policy. Correct? Reason I ask this is because you mentioned around 650G data to be recalled, but policy seems to have chosen ~2.5G data across 4812 files. So if this path is actively used by users, it is likely that the files may have been transparently recalled on access, and are in co-resident state already. Hence no data movement visible.
2. In /var/MCStore/ras/logs/mcstore.log, do you see any recall specific errors, when you run the policy?
3. On our test setup, we are planning to run similar policy once, to see if error is reproducible. I will share what we find.
AMEY GOKHALE
Senior Software Engineer – Spectrum Scale
Phone: 91-988 100 8675
E-mail: amey.gokhale at in.ibm.com<mailto:amey.gokhale at in.ibm.com>
From: Huzefa H Pancha <hupancha at in.ibm.com<mailto:hupancha at in.ibm.com>> On Behalf Of scale at us.ibm.com<mailto:scale at us.ibm.com>
Sent: 19 July 2022 23:42
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>; Amey P Gokhale <amey.gokhale at in.ibm.com<mailto:amey.gokhale at in.ibm.com>>
Cc: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>>
Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy
Hi Amey,
Can you provide them guidance from TCT angle.
Regards, The Spectrum Scale (GPFS) team
------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdeveloperworks%2Fcommunity%2Fforums%2Fhtml%2Fforum%3Fid%3D11111111-0000-0000-0000-000000000479&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cc47cc96d12ae4c07575f08da7097d3f5%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637946094988077487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ckqDPlU9imJlW2byM9iRDcN0H80nim03%2BL26Qq3Glhc%3D&reserved=0>.
If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries.
The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.
[Inactive hide details for "Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?]"Paul Ward" ---19-07-2022 09.34.33 PM---Thank you. Ill-placed on ESS or COS?
From: "Paul Ward" <p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>>
To: "gpfsug main discussion list" <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: 19-07-2022 09.34 PM
Subject: [EXTERNAL] Re: [gpfsug-discuss] mass recall from on-prem COS using a policy
Sent by: "gpfsug-discuss" <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>>
________________________________
Thank you. Ill-placed on ESS or COS? I understood restriping was for NSDs, so that would be on our ESS not COS? The direction I want to move the files is from COS to ESS. We do not have AFM enabled, we are using TCT.
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Thank you.
Ill-placed on ESS or COS?
I understood restriping was for NSDs, so that would be on our ESS not COS?
The direction I want to move the files is from COS to ESS.
We do not have AFM enabled, we are using TCT.
From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of IBM Spectrum Scale
Sent: 18 July 2022 20:35
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>; Venkateswara R Puvvada <vpuvvada at in.ibm.com<mailto:vpuvvada at in.ibm.com>>
Subject: Re: [gpfsug-discuss] mass recall from on-prem COS using a policy
"KB_Ill" shows how much data are ill placed or ill replicated. They can be resolved by mmrestripefs or mmrestripefile.
Copying to AFM team regarding recall in AFM-COS environment.
Regards, The Spectrum Scale (GPFS) team
------------------------------------------------------------------------------------------------------------------
If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries.
[Inactive hide details for "Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfo]"Paul Ward" ---07/12/2022 11:40:52 AM---Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 fil
From: "Paul Ward" <p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>>
To: "gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>" <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Date: 07/12/2022 11:40 AM
Subject: [EXTERNAL] [gpfsug-discuss] mass recall from on-prem COS using a policy
Sent by: "gpfsug-discuss" <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>>
________________________________
Hi all, I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G. We have a policy that runs every 30 mins, and I added this line to it:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi all,
I need to recall from on-prem COS a folder with subfolders and files, approximately 4600 files making up 656G.
We have a policy that runs every 30 mins, and I added this line to it:
RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOl 'migrate' TO POOL 'data' WHERE PATH_NAME LIKE '/gpfs/nhmfsa/bulk/share/workspace/iac-workspace/Chem_Labs/%'
Migrate is an external pool
RULE EXTERNAL POOL 'migrate' EXEC '/usr/lpp/mmfs/bin/mmcloudgateway files' OPTS '-F' ESCAPE '% -'
And data is the default placement pool
RULE 'Placement' SET POOL 'data'
When it runs it identifies matching files:
Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule
13 4846 2491152 4846 2491152 7056 RULE 'migrate_iac-workspace_Chem-labs' MIGRATE FROM POOL 'migrate' TO POOL 'data' WHERE(.)
I can’t find what ‘KB_ Ill’ means but nothing is migrating.
Migrations in other policies work, but this is a ‘recall’ policy.
From this document https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=pools-migrate-recall-external<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdocs%2Fen%2Fspectrum-scale%2F5.1.3%3Ftopic%3Dpools-migrate-recall-external&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cc47cc96d12ae4c07575f08da7097d3f5%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637946094988077487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=DXp1NjofMz9q%2BLRffBZrkLfWiOWyAGrCVjmcrYtQS5U%3D&reserved=0>
It suggests it should be invoked as a recall.
Any ideas?
Kindest regards,
Paul
Paul Ward
TS Infrastructure Architect
Natural History Museum
T: 02079426450
E: p.ward at nhm.ac.uk<mailto:p.ward at nhm.ac.uk>
[A picture containing drawing Description automatically generated]
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cc47cc96d12ae4c07575f08da7097d3f5%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637946094988077487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=mKA8Ox8cBmKAyRiiFNlllA8L16eU2WRxY2KDpiEK9hs%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cc47cc96d12ae4c07575f08da7097d3f5%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637946094988077487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=mKA8Ox8cBmKAyRiiFNlllA8L16eU2WRxY2KDpiEK9hs%3D&reserved=0> _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss_gpfsug.org&data=05%7C01%7Cp.ward%40nhm.ac.uk%7Cc47cc96d12ae4c07575f08da7097d3f5%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637946094988077487%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=mKA8Ox8cBmKAyRiiFNlllA8L16eU2WRxY2KDpiEK9hs%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20220729/5675f3e6/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 105 bytes
Desc: image001.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20220729/5675f3e6/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 5356 bytes
Desc: image002.jpg
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20220729/5675f3e6/attachment-0002.jpg>
More information about the gpfsug-discuss
mailing list