<font size=2 face="sans-serif">HI </font><font size=2>Doug</font><font size=2 face="sans-serif">,
</font><br><br><font size=2 face="sans-serif">I did some compares with gpfsperf ...
betweend IB and 100GbE .. but we used the 100GbE with ROCE .. so my results
might not be representative for you .. </font><br><font size=2 face="sans-serif">(don't wonder about edited hostnames
.. its from a real customer environment..) </font><br><br><font size=2 face="sans-serif">so with real data workload.. it is nearly
the same... ~ 6 ms</font><br><br><font size=2 face="sans-serif">Infiniband</font><br><font size=1 face="Courier">[root@nnnn ~]# mmdsh -N nnnn,nnnn,nnnn,nnnnn
\</font><br><font size=1 face="Courier"><b>"tsqosperf read seq /gpfs/sase1/testdev/\$(hostname)
-n 100g -r 4m -th 8 -dio" | </b>grep "rate was"</font><br><font size=1 face="Courier">nnnnn Data rate was 5057593.05 Kbytes/sec,
Op Rate was 1205.82 Ops/sec, Avg Latency was 6.557 milliseconds, thread
utilization 0.988, bytesTransferred 107374182400</font><br><font size=1 face="Courier">nnnnn
Data rate was 5046234.10 Kbytes/sec, Op Rate was 1203.12 Ops/sec, Avg Latency
was 6.576 milliseconds, thread utilization 0.989, bytesTransferred 107374182400</font><br><font size=1 face="Courier">nnnnn
Data rate was 4988625.75 Kbytes/sec, Op Rate was 1189.38 Ops/sec, Avg Latency
was 6.557 milliseconds, thread utilization 0.975, bytesTransferred 107374182400</font><br><font size=1 face="Courier">nnnnn Data rate was 4136019.23 Kbytes/sec,
Op Rate was 986.10 Ops/sec, Avg Latency was 7.995 milliseconds, thread
utilization 0.985, bytesTransferred 10737418240</font><br><div><font size=2 face="sans-serif">100GbE RoCE</font><br><font size=1 face="Courier">[root@bb1gssio1 ~]# mmdsh -N c09n1,c09n2,c09n3,c09n4
\</font><br><br><font size=1 face="Courier"><b>"tsqosperf read seq /gpfs/gpfs0/\$(hostname)
-n 100g -r 4m -th 8 -dio" | grep "rate was"</b></font><br><br><font size=1 face="Courier">C09n1 Data
rate was 5350528.27 Kbytes/sec, Op Rate was 1275.67 Ops/sec, Avg Latency
was 6.242 milliseconds, thread utilization 0.995, bytesTransferred 107374182400</font><br><font size=1 face="Courier">C09n2 Data rate was 4964347.14 Kbytes/sec,
Op Rate was 1183.59 Ops/sec, Avg Latency was 6.743 milliseconds, thread
utilization 0.998, bytesTransferred 107374182400</font><br><font size=1 face="Courier">C09n3 Data
rate was 4857516.69 Kbytes/sec, Op Rate was 1158.12 Ops/sec, Avg Latency
was 6.893 milliseconds, thread utilization 0.998, bytesTransferred 107374182400</font><br><font size=1 face="Courier">C09n4 Data
rate was 4829485.95 Kbytes/sec, Op Rate was 1151.44 Ops/sec, Avg Latency
was 6.929 milliseconds, thread utilization 0.997, bytesTransferred 107374182400</font><div><br><br><font size=2 face="sans-serif">with Mellanox ib_read_lat .. the picture
looks different.... ~ 2,5 usec versus 4,6 usec ... </font><br><br><font size=2 face="sans-serif"> </font><br><font size=2 face="sans-serif">Infiniband</font><br><font size=1 face="Courier">[root@nnnnnn ~]# <b>ib_read_lat </b>nnnnnnnnn
</font><br><font size=1 face="Courier">---------------------------------------------------------------------------------------
</font><br><font size=1 face="Courier">
RDMA_Read Latency Test </font><br><font size=1 face="Courier">Dual-port : OFF
Device : mlx4_0
</font><br><font size=1 face="Courier">Number of qps : 1
Transport type : IB </font><br><font size=1 face="Courier">Connection type : RC
Using SRQ : OFF </font><br><font size=1 face="Courier">TX depth : 1
</font><br><font size=1 face="Courier">Mtu
: 2048[B] </font><br><font size=1 face="Courier">Link type : IB </font><br><font size=1 face="Courier">Outstand reads : 16 </font><br><font size=1 face="Courier">rdma_cm QPs : OFF </font><br><font size=1 face="Courier">Data ex. method : Ethernet </font><br><font size=1 face="Courier">---------------------------------------------------------------------------------------
</font><br><font size=1 face="Courier">local address: LID 0x68 QPN 0xd751 PSN
0xadfaf8 OUT 0x10 RKey 0x1001411b VAddr 0x007fca31970000 </font><br><font size=1 face="Courier">remote address: LID 0x58 QPN 0x09d6 PSN
0x6a6c59 OUT 0x10 RKey 0x081392 VAddr 0x003fff879d0000 </font><br><font size=1 face="Courier">---------------------------------------------------------------------------------------
</font><br><font size=1 face="Courier">#bytes #iterations t_min[usec]
t_max[usec] <b>t_typical[usec] t_avg[usec]
</b> t_stdev[usec] 99% percentile[usec] 99.9%
percentile[usec] </font><br><font size=1 face="Courier">2 1000
2.40 18.12
<b>2.46
2.48</b> 0.28
2.57
18.12 </font><br><font size=1 face="Courier">---------------------------------------------------------------------------------------</font><br><br><br><br><br><br><font size=2 face="sans-serif">ROCE</font><br><font size=1 face="Courier">---------------------------------------------------------------------------------------</font><br><font size=1 face="Courier">root@dc18n2:~# <b>ib_read_lat </b>10.10.4.1</font><br><font size=1 face="Courier">---------------------------------------------------------------------------------------</font><br><font size=1 face="Courier">
RDMA_Read Latency Test</font><br><font size=1 face="Courier"> Dual-port :
OFF Device
: mlx5_1</font><br><font size=1 face="Courier"> Number of qps : 1
Transport type : IB</font><br><font size=1 face="Courier"> Connection type : RC
Using SRQ : OFF</font><br><font size=1 face="Courier"> TX depth :
1</font><br><font size=1 face="Courier"> Mtu
: 4096[B]</font><br><font size=1 face="Courier"> Link type :
Ethernet</font><br><font size=1 face="Courier"> GID index :
3</font><br><font size=1 face="Courier"> Outstand reads : 16</font><br><font size=1 face="Courier"> rdma_cm QPs : OFF</font><br><font size=1 face="Courier"> Data ex. method : Ethernet</font><br><font size=1 face="Courier">---------------------------------------------------------------------------------------</font><br><font size=1 face="Courier"> local address: LID 0000 QPN 0x0e7e
PSN 0x51b972 OUT 0x10 RKey 0x089b6f VAddr 0x007fe80cd11000</font><br><font size=1 face="Courier"> GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:19:02:18</font><br><font size=1 face="Courier"> remote address: LID 0000 QPN 0x0d02
PSN 0xdc9761 OUT 0x10 RKey 0x008142 VAddr 0x003fffb24f0000</font><br><font size=1 face="Courier"> GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:10:04:01</font><br><font size=1 face="Courier">---------------------------------------------------------------------------------------</font><br><font size=1 face="Courier"> #bytes #iterations t_min[usec]
t_max[usec] <b> t_typical[usec] t_avg[usec]</b> t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
</font><br><font size=1 face="Courier"> 2 1000
4.55 7.12
<b>4.61
4.62 </b>
0.05 4.82
7.12 </font><br><font size=1 face="Courier">---------------------------------------------------------------------------------------</font><br><br><br><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Douglas Duckworth <dod2014@med.cornell.edu></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif"><gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">03/06/2018 08:00 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
100G RoCEE and Spectrum Scale Performance</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Hi</font><br><br><font size=3>We are currently running Spectrum Scale over FDR Infiniband.
We plan on upgrading to EDR since I have not really encountered documentation
saying to abandon the lower-latency advantage found in </font><font size=2 color=#2f2f2f face="Arial">Infiniband.
Our workloads generally benefit from lower latency.</font><br><br><font size=2 color=#2f2f2f face="Arial">It looks like, ignoring GPFS,
EDR still has higher throughput and lower latency when compared to 100G
RoCEE. </font><br><br><a href="http://sc16.supercomputing.org/sc-archive/tech_poster/poster_files/post149s2-file3.pdf"><font size=3 color=blue><u>http://sc16.supercomputing.org/sc-archive/tech_poster/poster_files/post149s2-file3.pdf</u></font></a><br><br><font size=3>However, I wanted to get feedback on how GPFS performs
with 100G Ethernet instead of FDR.</font><br><br><font size=3>Thanks very much!</font><br><br><font size=3>Doug</font><br><br><font size=2>Thanks,</font><br><font size=2><br>Douglas Duckworth, MSc, LFCS<br>HPC System Administrator<br>Scientific Computing Unit</font><br><font size=2>Physiology and Biophysics</font><br><font size=2>Weill Cornell Medicine</font><br><font size=2>E: </font><a href=mailto:doug@med.cornell.edu target=_blank><font size=2 color=blue><u>doug@med.cornell.edu</u></font></a><font size=2><br>O: 212-746-6305<br>F: 212-746-8690</font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br></div></div><BR>