<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" ><em><font size="2" face="Default Monospace,Courier New,Courier,monospace" >[...] We have tried a number of things including Spectrum Scale client<br>version 5.0.5-9[...] in the client code or the server code?</font></em><br> </div>
<div dir="ltr" >there are going multiple improvements in the code.. continuously... Since your version 4.2.3 / 5.0.5 a lot of them are in the area of NSD server/GNR (which is server based) and also a lot of enhancements went into the client part. Some are on both .. such as RoCE, or using multiple TCP/IP sockets per communication pair, etc....</div>
<div dir="ltr" >All this influences your performance..</div>
<div dir="ltr" > </div>
<div dir="ltr" >But I d like to try to give you some answers to your specific Q -</div>
<div dir="ltr" ><em><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Only now do I notice a suggestion:<br> mmchconfig prefetchAggressivenessRead=0 -i<br>I did not use this. Would a performance change be expected?</font></em><br> </div>
<div dir="ltr" > </div>
<div dir="ltr" ><strong>YES</strong>;-) .. this parameter should really help .. from the UG expert talk 2020 we shared some numbers/charts on it</div>
<div dir="ltr" ><a href="https://www.spectrumscaleug.org/event/ssugdigital-spectrum-scale-expert-talks-update-on-performance-enhancements-in-spectrum-scale/">https://www.spectrumscaleug.org/event/ssugdigital-spectrum-scale-expert-talks-update-on-performance-enhancements-in-spectrum-scale/</a></div>
<div dir="ltr" ><font size="2" face="Default Monospace,Courier New,Courier,monospace" >starting ~ 8:30 minutes / just 2 slides ... </font></div>
<div dir="ltr" >let us know, if you need more information</div>
<div dir="ltr" > </div>
<div dir="ltr" >next Q:</div>
<div dir="ltr" ><em><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Would the pagepool size be involved in this?</font></em></div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" ><div>GPFS pagepool can be considered to be pinned in memory. So I doubt - and as you are writing - <em>[...]<font size="2" face="Default Monospace,Courier New,Courier,monospace" >echo 3 > /proc/sys/vm/drop_caches) the slow performance returns[...] -</font></em><font size="2" face="Default Monospace,Courier New,Courier,monospace" > that papgepool is your first choice for further analysis. </font></div>
<div><font size="2" ><font face="Default Monospace,Courier New,Courier,monospace" >however... I can't see/say smth about your pagepool utilization w/o seeing any counters, statistics.. etc..</font></font></div>
<div> </div>
<div>let me know, if this helps you .. or if you need more information ...</div>
<div> </div>
<div> </div>
<div> </div>
<div>Mit freundlichen Grüßen / Kind regards ;-)</div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div><div> <br>Olaf</div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" data-history-expanded="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Ursprüngliche Nachricht -----<br>Von: "Jon Diprose" <jon@well.ox.ac.uk><br>Gesendet von: gpfsug-discuss-bounces@spectrumscale.org<br>An: "gpfsug main discussion list" <gpfsug-discuss@spectrumscale.org><br>CC:<br>Betreff: [EXTERNAL] Re: [gpfsug-discuss] alphafold and mmap performance<br>Datum: Di, 19. Okt 2021 20:12<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Not that it answers Stuart's questions in any way, but we gave up on the same problem on a similar setup, rescued an old fileserver off the scrapheap (RAID6 of 12 x 7.2k rpm SAS on a PERC H710P) and just served the reference data by nfs - good enough to keep the compute busy rather than in cxiWaitEventWait. If there's significant demand for Alphafold then somebody's arm will be twisted for a new server with some NVMe. If I remember right, the reference data is ~2.3TB, ruling out our usual approach of just reading the problematic files into a ramdisk first.<br>We are also interested in hearing how it might be usably served from GPFS.<br>Thanks,<br>Jon<br><br>--<br>Dr. Jonathan Diprose <jon@well.ox.ac.uk> Tel: 01865 287873<br>Research Computing Manager<br>Henry Wellcome Building for Genomic Medicine<br>Roosevelt Drive, Headington, Oxford OX3 7BN<br><br>________________________________________<br>From: gpfsug-discuss-bounces@spectrumscale.org [gpfsug-discuss-bounces@spectrumscale.org] on behalf of Stuart Barkley [stuartb@4gh.net]<br>Sent: 19 October 2021 18:16<br>To: gpfsug-discuss@spectrumscale.org<br>Subject: [gpfsug-discuss] alphafold and mmap performance<br><br>Over the years there have been several discussions about performance<br>problems with mmap() on GPFS/Spectrum Scale.<br><br>We are currently having problems with mmap() performance on our<br>systems with new alphafold <<a href="https://github.com/deepmind/alphafold" target="_blank">https://github.com/deepmind/alphafold</a> ><br>protein folding software. Things look similar to previous times we<br>have had mmap() problems.<br><br>The software component "hhblits" appears to mmap a large file with<br>genomic data and then does random reads throughout the file. GPFS<br>appears to be doing 4K reads for each block limiting the performance.<br><br>The first run takes 20+ hours to run. Subsequent identical runs<br>complete in just 1-2 hours. After clearing the linux system cache<br>(echo 3 > /proc/sys/vm/drop_caches) the slow performance returns for<br>the next run.<br><br>GPFS Server is 4.2.3-5 running on DDN hardware. CentOS 7.3<br>Default GPFS Client is 4.2.3-22. CentOS 7.9<br><br>We have tried a number of things including Spectrum Scale client<br>version 5.0.5-9 which should have Sven's recent mmap performance<br>improvements. Are the recent mmap performance improvements in the<br>client code or the server code?<br><br>Only now do I notice a suggestion:<br> mmchconfig prefetchAggressivenessRead=0 -i<br>I did not use this. Would a performance change be expected?<br><br>Would the pagepool size be involved in this?<br><br>Stuart Barkley<br>--<br>I've never been lost; I was once bewildered for three days, but never lost!<br> -- Daniel Boone<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div><BR>
<BR>