[gpfsug-discuss] Adding client nodes using a shared NFS root image.

Ruffner, Scott (jpr9c) jpr9c at virginia.edu
Thu Feb 25 15:47:41 GMT 2021


I appreciate the solutions; at the moment, I'm making do with having a key for just this specific copy, and doing our own rsync.  If we had local disk on our nodes, that would be helpful, but /var/mmfs is mounted on tmpfs and gets flushed at every reboot (not just upgrades to hardware or software stack.

I'm testing with invoking the mmsdrrestore from the client only using a key. Yeah, not ideal leaving that laying around.  

-- 
Scott Ruffner
Senior HPC Engineer
UVa Research Computing
(434)924-6778(o)
(434)295-0250(h)
sruffner at virginia.edu
 

On 2/25/21, 5:13 AM, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Jon Diprose" <gpfsug-discuss-bounces at spectrumscale.org on behalf of jon at well.ox.ac.uk> wrote:

    I have written an xCAT plugin such that the client can ask the master to do it, in preparation for some shiny new kit. Not in a public repo yet, so please email me direct if you want further info.
    
    -- 
    Dr. Jonathan Diprose <jon at well.ox.ac.uk>             Tel: 01865 287837
    Research Computing Manager
    Henry Wellcome Building for Genomic Medicine Roosevelt Drive, Headington, Oxford OX3 7BN
    
    -----Original Message-----
    From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Trafford, Tyler
    Sent: 23 February 2021 00:40
    To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
    Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image.
    
    My solution to this is to have the below defined in an XCAT syncfiles config:
    
    /var/mmfs/gen/mmsdrfs -> /var/mmfs/gen/mmsdrfs
    /var/mmfs/ssl/stage/genkey* -> /var/mmfs/ssl/stage/
    
    And then in a postscript this happens on the node being (re)installed:
    
    # Only reconfigure if the host's address is present in the mmsdrfs file # which is synced from the mgt node at install time if for i in $(hostname --all-ip-addresses); do
           grep -q ":20_MEMBER_NODE:.*:${i}:" /var/mmfs/gen/mmsdrfs && break
       done
    then
        /usr/lpp/mmfs/bin/mmsdrrestore
    else
        rm /var/mmfs/gen/mmsdrfs
    fi
    
    
    -Tyler Trafford
    
    ________________________________________
    From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Peter Childs <p.childs at qmul.ac.uk>
    Sent: Monday, February 1, 2021 10:08 AM
    To: gpfsug main discussion list
    Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image.
    
    We used to run
    
    mmsdrestore -p manager -R /usr/bin/scp
    
    in a xcat postscript to re-add our nodes to our Spectrum Scale cluster. however we disliked needing to put the private key for the whole cluster on every host,
    
    We now use
    
    mmsdrestore -N nodename
    
    post-install from a management node to re-add the node to the cluster, so we could stop xcat from distributing the private key for security reasons.
    
    Ideally we would have like the postscript call a manual call back to do this but have not as yet worked out how best to do this in xcat, so currently its a manual task which is fine when our nodes are stateless, but is not possible when your nodes are stateless.
    
    My understanding is that xcat should have a hook to do this like the pre-scripts to run one at the end but I'm yet to find it.
    
    Peter Childs
    
    ________________________________________
    From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Ruffner, Scott (jpr9c) <jpr9c at virginia.edu>
    Sent: Friday, January 29, 2021 8:04 PM
    To: gpfsug main discussion list
    Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image.
    
    Thanks David! Slick solution.
    
    --
    Scott Ruffner
    Senior HPC Engineer
    UVa Research Computing
    (434)924-6778(o)
    (434)295-0250(h)
    sruffner at virginia.edu
    
    
    From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "david_johnson at brown.edu" <david_johnson at brown.edu>
    Reply-To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
    Date: Friday, January 29, 2021 at 2:52 PM
    To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
    Subject: Re: [gpfsug-discuss] Adding client nodes using a shared NFS root image.
    
    We use mmsdrrestore after the node boots. In our case these are diskless nodes provisioned by xCAT.  The post install script takes care of ensuring infiniband is lit up, and does the mmsdrrestore followed by mmstartup.
      -- ddj
    Dave Johnson
    
    
    On Jan 29, 2021, at 2:47 PM, Ruffner, Scott (jpr9c) <jpr9c at virginia.edu> wrote:
    Hi everyone,
    
    We want all of our compute nodes (bare metal) to directly participate in the cluster as client nodes; of course, they are sharing a common root image.
    
    Adding nodes via the regular mmaddnode (with the dsh operation to replicate files to the clients) isn't really viable, but if I short-circuit that, and simply generate the /var/mmfs/gen files and then manually copy those and the keyfiles to the shared root images, is that safe?
    
    Am I going about this the entirely wrong way?
    
    --
    Scott Ruffner
    Senior HPC Engineer
    UVa Research Computing
    (434)924-6778(o)
    (434)295-0250(h)
    sruffner at virginia.edu
    
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=04%7C01%7Ctyler.trafford%40yale.edu%7C6b0df05f560b42fa8d4608d8c6c33934%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C637477889136942435%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yo%2FGq4J3E8vRe4muoiIA1aVqlmuhXN1nsyNcVmlr%2BRg%3D&reserved=0
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    http://gpfsug.org/mailman/listinfo/gpfsug-discuss
    



More information about the gpfsug-discuss mailing list