[gpfsug-discuss] default value of SIZE in EXTERNAL POOL and EXTERNAL LIST rule

Thu Mar 31 08:53:07 BST 2022

Thanks Alec!

On Thu, Mar 31, 2022 at 7:41 AM Alec <anacreo at gmail.com> wrote:

> I'd like to offer a dissenting opinion...  I believe this is a way to
> limit the number of actual bytes that any execution must work on....  As in
> only send 500gb worth of files to each invocation.  Whereas the -B is the
> actual number of files to be passed to each invocation.  I would think that
> size has no default limit as I've had a single invocation stuck with 30TB
> of data in its 100 file grouping, lol.
>
> I however use a different mechanism to divy up my files because I need to
> worry about my backup streams getting choked up, and I want to take a cross
> section of each directory on different nights...
>
> I call mmapplypolicy with -M shares=<someincrementingnumber>, so let's say
> I increase my number by 1 every day, starting at 0, over the course of a
> week...
>
> In my policy I have this line:
>   WHERE MOD(INODE,6)<=shares
>
> So then I work on 1/7th of the files in my file system and cumulatively
> pick up any INODES until I've hit all of the files... I could pick just the
> matching segment by saying = instead of <=.
>
> This is happening at the file selection phase, so the application of -B is
> after this selection, as my files are quite huge, I tend to use -B 1.
>
> Alec
>
> On Wed, Mar 30, 2022 at 3:44 PM IBM Spectrum Scale <scale at us.ibm.com>
> wrote:
>
>> Hi shao feng,
>>
>> It is controlled by -B option.
>>
>> *-B* *MaxFiles*
>>          Specifies how many files are passed for each invocation
>>          of the EXEC script. The default value is 100.
>>
>>          If the number of files exceeds the value specified for
>>          MaxFiles, *mmapplypolicy* invokes the external
>>          program multiple times.
>>
>>          For more information about file list records, refer to
>>          the IBM Spectrum Scale: Administration Guide.
>>
>> Regards, The Spectrum Scale (GPFS) team
>>
>>
>> ------------------------------------------------------------------------------------------------------------------
>> If your query concerns a potential software error in Spectrum Scale
>> (GPFS) and you have an IBM software maintenance contract please contact
>>  1-800-237-5511 in the United States or your local IBM Service Center in
>> other countries.
>>
>> [image: Inactive hide details for IBM Spectrum Scale---03/30/2022
>> 09:53:51 AM---Hi Diane, Can you please look at the below policy relat]IBM
>> Spectrum Scale---03/30/2022 09:53:51 AM---Hi Diane, Can you please look at
>> the below policy related query.
>>
>> From: IBM Spectrum Scale/Poughkeepsie/IBM at IBMUS
>> To: "gpfsug main discussion list" <gpfsug-discuss at gpfsug.org>, Diane
>> Chung/Poughkeepsie/IBM at IBMMail
>> Cc: "gpfsug-discuss" <gpfsug-discuss-bounces at gpfsug.org>
>> Date: 03/30/2022 09:53 AM
>> Subject: [EXTERNAL] Re: [gpfsug-discuss] default value of SIZE in
>> EXTERNAL POOL and EXTERNAL LIST rule
>> Sent by: "gpfsug-discuss" <gpfsug-discuss-bounces at gpfsug.org>
>> ------------------------------
>>
>>
>>
>> Hi Diane,
>>
>> Can you please look at the below policy related query.
>>
>> Regards, The Spectrum Scale (GPFS) team
>>
>>
>> ------------------------------------------------------------------------------------------------------------------
>> If you feel that your question can benefit other users of  Spectrum Scale
>> (GPFS), then please post it to the public IBM developerWroks Forum at
>> *https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479*
>> <https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479>.
>>
>>
>> If your query concerns a potential software error in Spectrum Scale
>> (GPFS) and you have an IBM software maintenance contract please contact
>>  1-800-237-5511 in the United States or your local IBM Service Center in
>> other countries.
>>
>> The forum is informally monitored as time permits and should not be used
>> for priority messages to the Spectrum Scale (GPFS) team.
>>
>> [image: Inactive hide details for "shao feng" ---28-03-2022 03.21.39
>> PM---Hello, What is the default value of SIZE in EXTERNAL POOL in]"shao
>> feng" ---28-03-2022 03.21.39 PM---Hello, What is the default value of SIZE
>> in EXTERNAL POOL in filesystem policy?
>>
>> From: "shao feng" <shaof777 at gmail.com>
>> To: "gpfsug main discussion list" <gpfsug-discuss at gpfsug.org>
>> Date: 28-03-2022 03.21 PM
>> Subject: [EXTERNAL] [gpfsug-discuss] default value of SIZE in EXTERNAL
>> POOL and EXTERNAL LIST rule
>> Sent by: "gpfsug-discuss" <gpfsug-discuss-bounces at gpfsug.org>
>> ------------------------------
>>
>>
>>
>> Hello,
>>
>> What is the default value of SIZE in EXTERNAL POOL in filesystem policy?
>>
>> The document did not mention that, but I found it is not infinity if not
>> specified.
>>
>> Thank you!
>>
>> *SIZE **sum-number*
>> Is an optional clause of the *EXTERNAL POOL* and *EXTERNAL LIST* rules.
>> *sum-number* limits the total number of bytes in all of the files named
>> in each list of files passed to your EXEC 'script'. If a single file is
>> larger than *sum-number*, it is passed to your EXEC 'script' as the only
>> entry in a “singleton” file list.
>>
>> Specify *sum-number* as a numeric constant or a floating-point value.
>>
>> *Note:* The value of *sum-number* is in kilobytes.
>> *https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=rules-policy-terms*
>> <https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=rules-policy-terms>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> *http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org*
>> <http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>>
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at gpfsug.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20220331/b40dfa92/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20220331/b40dfa92/attachment-0002.gif>