<html><body><p>There are two related, but distinctly different issues to consider.<br><br>1) File system format and backward compatibility. The format of a given file system is recorded on disk, and determines the level of code required to mount such a file system. GPFS offers backward compatibility for older file system versions stretching for many releases. The oldest file system format we test with in the lab is 2.2 (we don't believe there are file systems using older versions actually present in the field). So if you have a file system formatted using GPFS V3.5 code, you can mount that file system using GPFS V4.1 or V4.2 without a problem. Of course, you don't get to use the new features that depend on the file system format that came out since V3.5. If you're formatting a new file system on a cluster running newer code, but want that file system to be mountable by older code, you have to use --version with mmcrfs.<br><br>2) RPC format compatibility, aka nodes being able to talk to each other. As the code evolves, the format of some RPCs sent over the network to other nodes naturally has to evolve as well. This of course presents a major problem for code coexistence (running different versions of GPFS on different nodes in the same cluster, or nodes from different clusters mounting the same file system, which effectively means joining a remote cluster), which directly translates into the possibility of a rolling migration (upgrading nodes to newer GPFS level one at a time, without taking all nodes down). Implementing new features while preserving some level of RPC compatibility with older releases is Hard, but this is something GPFS has committed to, long ago. The commitment is not open-ended though, there's a very specific statement of support for what's allowed. GPFS major (meaning 'v' or 'r' is incremented in a v.r.m.f version string) release N stream shall have coexistence with the GPFS major release N - 1 stream. So coexistence of V4.2 with V4.1 is supported, while coexistence of V4.2 with older releases is unsupported (it may or may not work if one tries it, depending on the specific combination of versions, but one would do so entirely on own risk). The reason for limiting the extent of RPC compatibility is prosaic: in order to support something, we have to be able to test this something. We have the resources to test the N / N - 1 combination, for every major release N. If we had to extend this to N, N - 1, N - 2, N - 3, you can do the math on how many combinations to test that would create. That would bust the test budget.<br><br>So if you want to cross-mount a file system from a home cluster running V4.2, you have to run at least V4.1.x on client nodes, and the file system would have to be formatted using the lowest version used on any node mounting the file system.<br><br>Hope this clarifies things a bit.<br><br>yuri<br><br><img width="16" height="16" src="cid:1__=07BBF5EBDFFB2DCA8f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for "Uwe Falke" ---03/16/2016 11:52:28 AM---Hi, Damir, I have not done that, but a rolling upgrade from "><font color="#424282">"Uwe Falke" ---03/16/2016 11:52:28 AM---Hi, Damir, I have not done that, but a rolling upgrade from 3.5.x to 4.1.x (maybe</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Uwe Falke" <UWEFALKE@de.ibm.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>, </font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">03/16/2016 11:52 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] cross-cluster mounting different versions ofgpfs</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt>Hi, Damir, <br><br>I have not done that, but a rolling upgrade from 3.5.x to 4.1.x (maybe <br>even to 4.2) is supported. <br>So, as long as you do not need all 500 nodes of your compute cluster <br>permanently active, you might upgrade them in batches without fully-blown <br>downtime. Nicely orchestrated by some scripts it could be done quite <br>smoothly (depending on the percentage of compute nodes which can go down <br>at once and on the run time / wall clocks of your jobs this will take <br>between few hours and many days ...).<br><br><br> <br>Mit freundlichen Grüßen / Kind regards<br><br> <br>Dr. Uwe Falke<br> <br>IT Specialist<br>High Performance Computing Services / Integrated Technology Services / <br>Data Center Services<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland<br>Rathausstr. 7<br>09111 Chemnitz<br>Phone: +49 371 6978 2165<br>Mobile: +49 175 575 2877<br>E-Mail: uwefalke@de.ibm.com<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland Business & Technology Services GmbH / Geschäftsführung: <br>Frank Hammer, Thorsten Moehring<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, <br>HRB 17122 <br><br><br><br><br>From: Damir Krstic <damir.krstic@gmail.com><br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Date: 03/16/2016 07:08 PM<br>Subject: Re: [gpfsug-discuss] cross-cluster mounting different <br>versions of gpfs<br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br><br><br><br>Sven,<br><br>For us, at least, at this point in time, we have to create new filesystem <br>with version flag. The reason is we can't take downtime to upgrade all of <br>our 500+ compute nodes that will cross-cluster mount this new storage. We <br>can take downtime in June and get all of the nodes up to 4.2 gpfs version <br>but we have users today that need to start using the filesystem. <br><br>So at this point in time, we either have ESS built with 4.1 version and <br>cross mount its filesystem (also built with --version flag I assume) to <br>our 3.5 compute cluster, or...we proceed with 4.2 ESS and build <br>filesystems with --version flag and then in June when we get all of our <br>clients upgrade we run =latest gpfs command and then mmchfs -V to get <br>filesystem back up to 4.2 features. <br><br>It's unfortunate that we are in this bind with the downtime of the compute <br>cluster. If we were allowed to upgrade our compute nodes before June, we <br>could proceed with 4.2 build without having to worry about filesystem <br>versions. <br><br>Thanks for your reply.<br><br>Damir<br><br>On Wed, Mar 16, 2016 at 12:18 PM Sven Oehme <oehmes@gmail.com> wrote:<br>while this is all correct people should think twice about doing this.<br>if you create a filesystem with older versions, it might prevent you from <br>using some features like data-in-inode, encryption, adding 4k disks to <br>existing filesystem, etc even if you will eventually upgrade to the latest <br>code. <br><br>for some customers its a good point in time to also migrate to larger <br>blocksizes compared to what they run right now and migrate the data. i <br>have seen customer systems gaining factors of performance improvements <br>even on existing HW by creating new filesystems with larger blocksize and <br>latest filesystem layout (that they couldn't before due to small file <br>waste which is now partly solved by data-in-inode). while this is heavily <br>dependent on workload and environment its at least worth thinking about. <br><br>sven <br><br><br><br>On Wed, Mar 16, 2016 at 4:20 PM, Marc A Kaplan <makaplan@us.ibm.com> <br>wrote:<br>The key point is that you must create the file system so that is "looks" <br>like a 3.5 file system. See mmcrfs ... --version. Tip: create or find a <br>test filesystem back on the 3.5 cluster and look at the version string. <br> mmslfs xxx -V. Then go to the 4.x system and try to create a file system <br>with the same version string....<br><br><br><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt>[attachment <br>"atthrpb5.gif" deleted by Uwe Falke/Germany/IBM] <br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br><br><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br><br></tt><br><BR>
</body></html>