[gpfsug-discuss] Default placement/External Pool

[Marc A Kaplan]

Peter,

1. to best exploit and integrate both Spectrum Scale and Cloud Storage, 
please consider:
https://www.ibm.com/blogs/systems/spectrum-scale-transparent-cloud-tiering/


2. Yes, you can use mmapplypolicy to push copies of files to an "external" 
system.  But you'll probably need a strategy or technique to avoid 
redundantly pushing the "next time" you run the command... 

3. Regarding mmapplypolicy nitty-gritty:   you can use the -N option to 
say exactly which nodes you want to run the command.

And regarding using ... EXTERNAL ... EXEC 'myscript' 

You can further restrict which nodes will act as mmapplypolicy "helpers" 
-- If on a particular node x, 'myscript' does not exist OR
  myscript TEST returns a non-zero exit code then node x will be 
excluded.... You will see a message like this:

[I] Messages tagged with <3> are from node n3.
<3> [E:73] Error on system(/ghome/makaplan/policies/mynodes.sh TEST 
'/foo/bar5'  2>&1)
<3> [W] EXEC '/ghome/makaplan/policies/mynodes.sh' of EXTERNAL POOL or 
LIST 'x' fails TEST with code 73 on this node.

OR

[I] Messages tagged with <5> are from node n4.
<5> sh: /tmp/mynodes.sh: No such file or directory
<5> [E:127] Error on system(/tmp/mynodes.sh TEST '/foo/bar5'  2>&1)
<5> [W] EXEC '/tmp/mynodes.sh' of EXTERNAL POOL or LIST 'x' fails TEST 
with code 127 on this node.



From:   "Chase, Peter" <peter.chase at metoffice.gov.uk>
To:     "'gpfsug-discuss at spectrumscale.org'" 
<gpfsug-discuss at spectrumscale.org>
Date:   11/08/2017 10:51 AM
Subject:        [gpfsug-discuss] Default placement/External Pool
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hello!

A follow up to my previous question about automatically sending files to 
Amazon s3 as they arrive in GPFS.

I have created an interface script to manage Amazon s3 storage as an 
external pool, I have created a migration policy that pre-migrates all 
files to the external pool and I have set that as the default policy for 
the file system.

All good so far, but the problem I'm now facing is: Only some of the 
cluster nodes have access to Amazon due to network constraints.  I read 
the statement "The mmapplypolicy command invokes the external pool script 
on all nodes in the cluster that have installed the script in its 
designated location."[1] and thought, 'Great! I'll only install the script 
on nodes that have access to Amazon' but that appears not to work for a 
placement policy/default policy and instead, the script runs on precisely 
no nodes.

I assumed this happened because running the script on a non-Amazon facing 
node resulted in a horrible error (i.e. file not found), so I edited my 
script to return a non-zero response if being run on a node that isn't in 
my cloudNode class, then installed the script every where. But this 
appears to have had no effect what-so-ever.






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171108/b146f757/attachment-0002.htm>