<html><body><p><font size="2">Jonathan, <br></font><font size="2">I suggest you get a formal statement from Lenovo as the DSS-G Platform is no longer an IBM platform.<br></font><font size="2"><br></font><font size="2">But for ESS based platforms the answer would be, it is not supported to run anything on the IO Servers other than GNR and the relevant Scale management services, due to the fact that if you lose an IO Server, or if you in an extended maintenance window the Server needs to host all the work that would be being performed by both IO servers.<br></font><font size="2"><br></font><font size="2">I don't know if Lenovo have different point if view.<br></font><font size="2"><br></font><font size="2">Regards,<br></font><font size="2"><br></font><font size="2">Andrew<br></font><font size="2"><br></font><font size="2">Sent from my iPhone<br></font><font size="2"><br></font><font size="2">> On 3 Oct 2020, at 02:14, Jonathan Buzzard <jonathan.buzzard@strath.ac.uk> wrote:<br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> What if any are the rules around running additional services on DSS/ESS <br></font><font size="2">> nodes with regard to support? Let me outline our scenario<br></font><font size="2">> <br></font><font size="2">> Our main cluster uses 10Gbps ethernet for storage with the DSS-G nodes <br></font><font size="2">> hooked up with redundant 40Gbps ethernet.<br></font><font size="2">> <br></font><font size="2">> However we have an older cluster that is used for undergraduate teaching <br></font><font size="2">> that only has 1Gbps ethernet and QDR Infiniband. With no money to <br></font><font size="2">> upgrade this to 10Gbps ethernet to support this we flipped one of the <br></font><font size="2">> ports on the ConnectX4 cards on each DSS-G node to Infiniband and using <br></font><font size="2">> IPoIB run the teaching nodes in this way.<br></font><font size="2">> <br></font><font size="2">> However it means that we need an Ethernet to Infiniband gateway as the <br></font><font size="2">> ethernet only connected nodes want to talk to the Infiniband connected <br></font><font size="2">> ones on their Infiniband address. Not a problem we grabbed an old spare <br></font><font size="2">> machine installed CentOS and configured it up to act as a bridge, and <br></font><font size="2">> deploy a custom route to all the ethernet only connected nodes. It has <br></font><font size="2">> been working fine for a couple of years now.<br></font><font size="2">> <br></font><font size="2">> The problem is that this becomes firstly a single point of failure, on <br></font><font size="2">> hardware that is six years old now. Secondly to apply updates on the <br></font><font size="2">> gateway machine means all the teaching nodes have to be drained and GPFS <br></font><font size="2">> umounted to reboot the machine after updates have been installed. It is <br></font><font size="2">> currently not getting patched as frequently as I would like (and <br></font><font size="2">> required by the Scottish government).<br></font><font size="2">> <br></font><font size="2">> So thinking about it I have come to the conclusion that the ideal <br></font><font size="2">> situation would be to use the DSS-G nodes as the gateway and run <br></font><font size="2">> keepalived to move the gateway ethernet IP address between the two <br></font><font size="2">> machines. It is idea because as long as one DSS-G node is up then there <br></font><font size="2">> is a functioning gateway and nodes don't get ejected from the cluster. <br></font><font size="2">> If both DSS-G nodes are down then there is no GPFS to mount anyway and <br></font><font size="2">> lack of a gateway is a moot point.<br></font><font size="2">> <br></font><font size="2">> I grabbed a couple of the teaching compute nodes in the summer and <br></font><font size="2">> trialed it out. It works a treat.<br></font><font size="2">> <br></font><font size="2">> I now need to check IBM are not going to throw a wobbler down the line <br></font><font size="2">> if I need to get support before deploying it to the DSS-G nodes :-)<br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> JAB.<br></font><font size="2">> <br></font><font size="2">> -- <br></font><font size="2">> Jonathan A. Buzzard Tel: +44141-5483420<br></font><font size="2">> HPC System Administrator, ARCHIE-WeSt.<br></font><font size="2">> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG<br></font><font size="2">> _______________________________________________<br></font><font size="2">> gpfsug-discuss mailing list<br></font><font size="2">> gpfsug-discuss at spectrumscale.org<br></font><font size="2">> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br></font><font size="2">> <br></font><BR>
</body></html>