[gpfsug-discuss] mmapplypolicy slow

[Peeples, Heath]

Well we want to run it against the snapshot so I think we need the -S option.  When you run it against the FS filtering by SNAP_ID, you miss any files that have been deleted since the snapshot.

Heath


From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Jan-Frode Myklebust
Sent: Tuesday, August 3, 2021 2:09 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] mmapplypolicy slow


Just guessing, but maybe some «where SNAP_ID('SnapshotName')» is better than -S 20210104?


?

tir. 3. aug. 2021 kl. 20:59 skrev Peeples, Heath <heathp at hpc.msstate.edu<mailto:heathp at hpc.msstate.edu>>:
Yes, that advertisement makes me smile 😊

I have tried both of those, and they do not seem to make any difference.  I have also played with the -A and -a parameters with no combination that I can find making it any better.

Thanks for the feedback.

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of Jan-Frode Myklebust
Sent: Tuesday, August 3, 2021 1:49 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] mmapplypolicy slow

So…. the advertisement ssys we should be able to do 1M files/s…
http://files.gpfsug.org/presentations/2018/USA/SpectrumScalePolicyBP.pdf<http://secure-web.cisco.com/1qQpvD9F4NtOuzGs8G_8PtFl-CwL-8JNJO1mx5gEbI7QwDLGI58rp5A1Zj3Q1nQoonzr0pdbS8-5R4oc5DLLmaBS6PQJ0HXdquIC5cLIlmasuY6rJfZ_dnC7zPJmBySjicZ0rFC-nrhKEnu-_J-arj4xmzx2VxDYJlbreMnsJfs8xW-c4B_Hnf7N9sPPXQ2b6C8jORSfUogYE0Lue9d2OEn-ITxLZdDzB34linSMOE1kYikuG5ewgP6p0nhcznywEWQr_aE1zvb_kgT9pW4ldt39RkTfGbH-O72Aa-sBxa9NilSUt8K-5wXh6KkXTvTln/http%3A%2F%2Ffiles.gpfsug.org%2Fpresentations%2F2018%2FUSA%2FSpectrumScalePolicyBP.pdf>

First I  would try is if maybe limiting which nodes are used for the processing helps. Maybe limit to the NSD-servers (-N nodenames) ?

Also,  --choice-algorithm fast might help..?



  -jf


tir. 3. aug. 2021 kl. 20:18 skrev Peeples, Heath <heathp at hpc.msstate.edu<mailto:heathp at hpc.msstate.edu>>:
I am trying to use mmapplypolicy to determine what files are modified in a particular snapshop.  Below I have the policy and mmapplypolicy parameters I am using.  It is taking 3-4 hours to list these files. There are roughly 176M files.  This is running on a DDN Gridscaler 12K system.   Seems to me after seeing others posts we should be able to run this much quicker.  Any help would be appreciated.


mmapplypolicy fs2 -S 20210104 -I defer -A 175 -a 72 -B 10000  -N all -g /fs1/tmp/heath -P /fs2/tmp/snaps/heath.pol -f /fs2/tmp/snaps/heath/output/20210104

RULE 'find_mods' list 'mods'
DIRECTORIES_PLUS
WEIGHT(0)
SHOW( VARCHAR(USER_ID) || ' ' || varchar(FILE_SIZE) || ' ' || varchar(MODIFICATION_SNAPID) )


Heath
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://secure-web.cisco.com/13y0L5a_-UdTp06HAY-SQnbTVm8HkHswo4iplosUAzEBhW3s6Y6LnTuCvikd3KQuk_YPuAuKhDLAzBjCcmelrVS146XsPHgLLLSbAb3bAKY9dZ10sV8_ErYKn0OqyE8gPgh4kjWnswfL7oomTnhYj4D1Q2mY83UI82dAJAX-2n8NaxlfloOFS23DgFMEULXJgy-wNH_cLemLqseter9-Rj2yO3jsruIBj3_DcY_Z87ffxo0SQcYazbJvThTeqb5C1o6hepHZkalgmUvQE6K2CAV_sX4cWrNs5GNqPy7xifM9tlbvhwJvwSvZGlTO_da4e/http%3A%2F%2Fspectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<http://secure-web.cisco.com/1joeovhVrYzawY9kLy658LHgv3dW6Ta7uZzGAlu08q1o21Rw4o615cnGStXQ0NQeAYE2jCpJAzC7DKAzkxD_cgeRLlQkKVRPFm8QFIMXwGcESXOwOmKXVVD5fq8H0gBD6VasiBdUmWrijLwK0QFvEduBRSXIqQ-sWoxrR1onYwyJEPV--XMh-r1O-92mnjtaUNVuSS2KXElVq8rc2O33St8Ga2qd5QSTT2pUQcbr2IJUM4a2gu0ZrtVQgRBRTZU1ZuoPnFZY8AL4tubpqa6vK_x_dt-W6bkV1XKSre2XQMxCaNb9Bm2_bKJgfhamPC-Rf/http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://secure-web.cisco.com/1Jxk0mTZJfEGRuwDAfk6WETmP_z5AYx4ZEYSsRf6Iq7v69ncxGgt70_hIM13VkBESWG6_5J1sazT3n6tfxKKmStUZmOj9j2XvYb9AXirO0aU5Doiod8DV6Exzl1bAO3nro5-AzRdLy12lqtXiY2AecwMn57_JZN9eTrPPB73pyXJrf2Pb0i5PfDRq6o4o2kC5pZ-UgYlYJP2b4Z37vzHDHis8pAUPvc6mKIBdtd_IUbJdvxGfErw3WZIyIdCVNIOyuvVqLDnQcJI9O291mb6-qOXUPiAxrONdm5c0LsU_HkgGbwYXkSRdEUuUtB3n3uM2/http%3A%2F%2Fspectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<http://secure-web.cisco.com/1-tpllNriTqNKP656g4djWNutzuUwZnTeFHXINw7U_4ltbsDdb04bvj1gxe7NEW3YrW-ANqci01QM9ftyYnaBWrgiTCH3PZYu5n3jFwUhSYE8cZh88leH2VaqiXQtN2yH0KW4fHgiO16MYEz35tyoaWGxVcPSk0ExX4ZuXohXCmMt9RrUuBygXw4_v15mAoDPha-mBUTOMtnoWQG7Hq4jJ8CpD6DGaYMV_E23CbJN5fInLFGYJ9b9tP80C_CKMhts_TuB1Z8l6bEYnWj62NqAV-936iq_nLtLoZLKwjZD2-WzENisFTchV_InvVhRgwoL/http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210803/ba397df3/attachment-0002.htm>