[gpfsug-discuss] kernel updates and GPFS modules: manual, DKMS, cron, other?

[Jonathan Buzzard]

On 03/03/2023 19:29, David Magda wrote:

> 
> Hello,
> 
> I am a new user of GPFS (Spectrum Scale) and would like to know if
> there is a ‘best practice’ on handling kernel updates on HPC clients.
> We are running Ubuntu 18.04 and 20.04 clients with 5.1.x, talking to
> RHEL storage servers, and would like to know how to handle
> re-compiling the client-side kernel modules.
> 
> There is of course the “mmbuildgpl” utility:
> 
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdocs%2Fen%2Fspectrum-scale%2F5.1.3%3Ftopic%3Dreference-mmbuildgpl-command&data=05%7C01%7Cjonathan.buzzard%40strath.ac.uk%7C441bff633acd491f1d5d08db1c1e178d%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C638134687790209306%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZQS%2BCAgmIhByy6mrDx9UudY%2BjkRGsaudBE8LUI%2Ft9Mg%3D&reserved=0
>
>  but how do folks invoke it? Manually, via cron at night on or
> reboot, via some kind of apt (dpkg-trigger(1)) / RPM hook?
> 
> We have the “unattended-upgrades” package enabled, which only
> installs security-tagged updates by default, but sometimes this does
> include kernel updates, which may become active on the next reboot:
> 
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpackages.ubuntu.com%2Fsearch%3Fkeywords%3Dunattended-upgrades&data=05%7C01%7Cjonathan.buzzard%40strath.ac.uk%7C441bff633acd491f1d5d08db1c1e178d%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C638134687790209306%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tfzFmrS5zdNlFz75newnxLxDPj5AzUWnF%2BX1NK6GoHE%3D&reserved=0

I would suggest that you disable any automatic upgrading of the kernel. 
Kernel upgrades should *only* be done *after* you have verified that it 
will work.

If you don't it is only a matter of time before a security update breaks 
GPFS. There was at least one instance of that happing in the last five 
years.

>
>  So is there a best practice? Has someone invented this wheel that I
> could leverage, or will I have to invent it myself?
> 

I use a RPM package called gpfs-helper that I created. It installs a 
local "helper" to the systemd gpfs unit file

/etc/systemd/system/gpfs.service.d/install-module.conf

[Service]
ExecStartPre=-/usr/bin/dnf --assumeyes --enablerepo dssg install 
gpfs.gplbin-%v

This causes the correct gpfs.gplbin RPM to be installed when starting 
GPFS. If the correct one is already installed it does nothing, otherwise 
it downloads the correct RPM and installs it before attempting to start 
GPFS. I am pretty sure you could do the same with apt. The %v is the 
magic bit which basically matches the running kernel.

So after testing that GPFS works on the new kernel, I build the RPM, put 
it in the local repo and winner winner chicken dinner.


JAB.

--
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG