[gpfsug-discuss] Client Latency and High NSD Server Load Average

[Uwe Falke]

Hello, Oluwasijibomi , 
I suppose you are not running ESS (might be wrong on this). 

I'd check the IO history on the NSD servers (high IO times?) and in 
addition the IO traffic at  the block device level , e.g. with iostat or 
the like (still high IO times there? Are the IO sizes ok or too low on the 
NSD servers with high write latencies? ). 

What's the picture on your storage back-end? All caches active? Is the 
storage backend fully loaded or rather idle? How is storage connected? 
SAS? FC? IB?

What is the actual IO pattern when you see these high latencies?

Do you run additional apps on some or all of youre NSD servers?

Mit freundlichen Grüßen / Kind regards

Dr. Uwe Falke
IT Specialist
Global Technology Services / Project Services Delivery / High Performance 
Computing
+49 175 575 2877 Mobile
Rathausstr. 7, 09111 Chemnitz, Germany
uwefalke at de.ibm.com

IBM Services

IBM Data Privacy Statement

IBM Deutschland Business & Technology Services GmbH
Geschäftsführung: Dr. Thomas Wolter, Sven Schooss
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122



From:   "Saula, Oluwasijibomi" <oluwasijibomi.saula at ndsu.edu>
To:     "gpfsug-discuss at spectrumscale.org" 
<gpfsug-discuss at spectrumscale.org>
Date:   03/06/2020 23:45
Subject:        [EXTERNAL] [gpfsug-discuss] Client Latency and High NSD 
Server Load Average
Sent by:        gpfsug-discuss-bounces at spectrumscale.org




Hello,

Anyone faced a situation where a majority of NSDs have a high load average 
and a minority don't?

Also, is 10x NSD server latency for write operations than for read 
operations expected in any circumstance? 

We are seeing client latency between 6 and 9 seconds and are wondering if 
some GPFS configuration or NSD server condition may be triggering this 
poor performance.


Thanks,

Oluwasijibomi (Siji) Saula
HPC Systems Administrator  /  Information Technology
 
Research 2 Building 220B / Fargo ND 58108-6050
p: 701.231.7749 / www.ndsu.edu
 


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=fTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8&m=ql8z1YSfrzUgT8kXQBMEUuA8uyuprz6-fpvC660vG5A&s=JSYPIzNMZFNp17VaqcNWNuwwUE_nQMKu47mOOUonLp0&e=