[gpfsug-discuss] Same file opened by many nodes / processes

[Uwe Falke]

Hi, Peter, 
in theory, the first node opening a file should remain metanode until it 
closes the file, regardless how many other nodes open it in between (if 
all the nodes are within the same cluster). 
MFT is controlling the caching inodes and - AFAIK - also of indirect 
blocks. A 200 GiB file will most likely have indirect blocks, but just a 
few up to some tens, depending on the  block size in the file system. The 
default MFT number is much larger.

However, if you say the metanode is changing, that might cause some 
delays, as all token information has to be passed on to the next metanode 
(not sure how efficient that election is running). Having said that it 
could help if you use a dedicated node having the file open from start and 
all the time - this should prevent new metanodes being elected.

If you do not get told a solution, you might want to run a trace of the 
mmbackup scan (maybe once with jobs accessing the file, once without).

 
Mit freundlichen Grüßen / Kind regards

 
Dr. Uwe Falke
 
IT Specialist
High Performance Computing Services / Integrated Technology Services / 
Data Center Services
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
Rathausstr. 7
09111 Chemnitz
Phone: +49 371 6978 2165
Mobile: +49 175 575 2877
E-Mail: uwefalke at de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland Business & Technology Services GmbH / Geschäftsführung: 
Thomas Wolter, Sven Schooß
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 17122 




From:   Peter Childs <p.childs at qmul.ac.uk>
To:     "gpfsug-discuss at spectrumscale.org" 
<gpfsug-discuss at spectrumscale.org>
Date:   10/07/2018 16:51
Subject:        [gpfsug-discuss] Same file opened by many nodes / 
processes
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



We have an situation where the same file is being read by around 5000
"jobs" this is an array job in uge with a tc set, so the file in
question is being opened by about 100 processes/jobs at the same time.

Its a ~200GB file so copying the file locally first is not an easy
answer, and these jobs are causing issues with mmbackup scanning the
file system, in that the scan is taking 3 hours instead of the normal
40-60 minutes.

This is read only access to the file, I don't know the specifics about
the job.

It looks like the metanode is moving around a fair amount (given what I
can see from mmfsadm saferdump file)

I'm wondering if we there is anything we can do to improve things or
that can be tuned within GPFS, I'm don't think we have an issue with
token management, but would increasing maxFileToCache on our token
manager node help say?

Is there anything else I should look at, to try and attempt to allow
GPFS to share this file better.

Thanks in advance

Peter Childs

-- 
Peter Childs
ITS Research Storage
Queen Mary, University of London
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss