<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Dear All,<br>
Happy new year to everyone!</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">My goal here at the Paul Scherrer Institut is to merge two different GPFS building
blocks. In partucular, these are not the same technology or not even the same brand:</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">- an IBM ESS-3500 (a NVMe/Performance storage system) consisting of a Power9 confluent
node and two AMD canisters, and 12 NVMe drives</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">- a Lenovo G242 "hybrid" consisting of 4 HDD enclosures, 2 SSD enclosures, 1 Intel support
node and 2 Intel storage nodes.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">The final configuration I would expect is a single building block with 4 IO nodes, 3
declustered array: 1 for HDDs, 1 for SSDs, 1 for NVMe (the last one to be used as a cache pool).</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">First of all, I would like to know if anyone has already tried this solution successfully.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Then, below is the description of what I have done.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I will preface by saying that I was able to configure the two storage clusters separately
without any problem; therefore, I would exclude any inherent problem in each building block (which was installed from scratch). But when I try to have a single cluster, with different node classes, I have problems.</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">The steps I followed (based on documentation I found in IBM pages, https://www.ibm.com/docs/en/ess-p8/5.3.1?topic=command-outline-mmvdisk-use-case)
are as follows:</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">1 access one of the 2 building blocks (that already has a storage cluster configured, with no recoverygroups
defined)</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">2 run "mmaddnode -N <the_two_IO_nodes_to_add_from_the_other_BB>"</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">3 mmchlicense...</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">3 mmvdisk nodeclass create ... to isolate the two "new" IO nodes in a dedicated nodeclass
for the purpose of differentiating configuration parameters, connected drive topology, and then recovery groups</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">4 perform topology discovery with: mmvdisk server list --node-class ess --disk-topology</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">In the following the cluster and node classes:</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
Node Daemon node name IP address Admin node name Designation <br>
---------------------------------------------------------------------- <br>
1 sfdssio1.psi.ch 129.129.241.67 sfdssio1.psi.ch quorum-manager <br>
2 sfdssio2.psi.ch 129.129.241.68 sfdssio2.psi.ch quorum-manager <br>
3 sfessio1.psi.ch 129.129.241.27 sfessio1.psi.ch quorum-manager <br>
4 sfessio2.psi.ch 129.129.241.28 sfessio2.psi.ch quorum-manager <br>
<br>
Node Class Name Members <br>
--------------------- -----------------------------------------------------------
<br>
ess sfessio1.psi.ch,sfessio2.psi.ch <br>
dss sfdssio1.psi.ch,sfdssio2.psi.ch<br>
<br>
</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">The "mmnodeadd" operation was performed while logged into sfdssio1 (which belongs to
the Lenovo G242).<br>
Then:</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[root@sfdssio1 ~]# mmvdisk server list --node-class ess --disk-topology
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
node needs matching <br>
number server attention metric disk topology <br>
------ -------------------------------- --------- -------- ------------- <br>
3 sfessio1.psi.ch yes - unmatched server topology
<br>
4 sfessio2.psi.ch yes - unmatched server topology
<br>
<br>
mmvdisk: To see what needs attention, use the command: <br>
mmvdisk: mmvdisk server list -N sfessio1.psi.ch --disk-topology -L <br>
mmvdisk: mmvdisk server list -N sfessio2.psi.ch --disk-topology -L</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">[root@sfdssio1 ~]# mmvdisk server list -N sfessio1.psi.ch --disk-topology -L
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);">Unable to find a matching topology specification for topology file '/var/mmfs/tmp/cmdTmpDir.mmvdisk.1468913/pdisk-topology.sfessio1.psi.ch'.
<br>
<br>
Topology component identification is using these CST stanza files: <br>
<br>
/usr/lpp/mmfs/data/compSpec-1304.stanza <br>
/usr/lpp/mmfs/data/compSpec-1400.stanza <br>
/usr/lpp/mmfs/data/cst/compSpec-Lenovo.stanza <br>
/usr/lpp/mmfs/data/cst/compSpec-topology.stanza <br>
<br>
Server component: serverType 'ESS3500-5141-FN2' serverArch 'x86_64' serverName 'sfessio1.psi.ch'
<br>
<br>
Enclosure components: 1 found connected to HBAs <br>
Enclosure component: serialNumber '78E4395' enclosureClass 'unknown' <br>
<br>
HBA components: none found connected to enclosures <br>
<br>
Cabling: enclosure '78E4395' controller '' cabled to HBA slot 'UNKNOWN' port 'unknown'
<br>
<br>
Disks: 12 SSDs 0 HDDs <br>
NVRAM: 0 devices/partitions <br>
<br>
Unable to match these components to a serverTopology specification. <br>
mmvdisk: Command failed. Examine previous error messages to determine cause.</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">If I try to do a symmetric operation (I access an IO node of the IBM ESS3500 and try
to add Lenovo nodes, trying to discover their drive topology) I get the same error; but, of course, the topology involved this time is that of the Lenovo hardware.</span></div>
<div class="elementToProof"><span style="font-family: monospace; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Now, I suspect there is a (hidden?) step I would be supposed to know, but unfortunately
I don't (this is my first experience with different and etherogenous building blocks merge). So I'd like to receive from you any suggestions, including a better documentation page (if any) covering this particular use case I have.<br>
<br>
Hope the description of the context is clear enough, in case it is not I apologize and please just ask for any further details required to understand my environment.</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Thank you very much,</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> Alvise Dorigo</span></div>
<div class="elementToProof"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
</body>
</html>